Validation of ICD-10 codes for studying foreign body airway obstructions: A health administrative data cohort study

Abstract

Aim

To validate a case definition for foreign body airway obstructions (FBAO) using International Classification of Diseases version 10 (ICD-10) codes to accurately identify patients in administrative health databases and improve reporting on this injury.

Methods

We identified prehospital patient encounters in Alberta, Canada between Jan 1, 2018 and Dec 31, 2021 by querying the provincial emergency medical services’ (EMS) patient care records for FBAO-related presentations, EMS protocols, or treatments. We deterministically linked EMS patient encounters to data on emergency department visits and hospital admissions, which included ICD-10 codes. Two physicians independently reviewed encounters to determine true FBAO cases. We then calculated diagnostic accuracy measures (sensitivity, specificity, likelihood ratios) of various algorithms.

Results

We identified 3677 EMS patient encounters, 2121 were linked to hospital administrative databases. Of these encounters, 825 (38.9%) were true FBAO. The combination of two ICD-10 codes (T17 = foreign body in the respiratory tract or T18.0 = foreign body in the mouth) was the most specific algorithm (96.9% [95%CI 95.8–97.8%]), while the combination of all FBAO-related ICD-10 codes and R06.8 (other breathing abnormalities) was the most sensitive (75.0% [95%CI 71.9–78.0]). We identified an additional 453 (35.4%) FBAO cases not transported by EMS (due to death or transport refusal), and therefore not linked to the hospital administrative databases. Of these unlinked encounters, 44 (9.7%) cases resulted in the patient’s death.

Conclusions

FBAO can be identified with reasonable accuracy using health administrative data and ICD-10 codes. All algorithms had a trade-off between sensitivity and specificity, and failed to identify a third of FBAO cases, of which 10% resulted in death.

Introduction

Foreign body airway obstructions (FBAO or choking) result in significant mortality and morbidity globally however contemporary epidemiologic data is scarce.1, 2 Without updated data we are unable to determine the magnitude of the problem and develop strategies to prevent and treat FBAO. One reason for the limited data is the challenge of identifying FBAO; prospective data collection or detailed chart review is usually required which is resource intensive. Using International Classification of Diseases codes (ICD) is one method that researchers can use to overcome this limitation.3, 4, 5, 6 However, an administrative data case definition of FBAO has not been validated. Evaluating the accuracy of ICD-based algorithms for identifying FBAO cases is important to minimize the risk of over- or under-estimating the true FBAO burden. By validating a case definition using ICD codes, researchers can apply it to large, population-based databases globally, where reference standard methods may not be feasible due to resource constraints. We aimed to validate an ICD-based case definition to identify patients who experienced a FBAO.

Methods

Study design and setting

This observational cohort study using administrative health data validated ICD-10 based case definitions of FBAO. The study adhered to the modified Standards for Reporting of Diagnostic Accuracy (STARD) guidelines.⁷ The Conjoint Health Research Ethics Board of the University of Calgary (REB21-0690) approved the study on July 8, 2021.

The study occurred in Alberta, Canada between January 1 2018, and December 31 2021. In Alberta, a single health authority (Alberta Health Services) delivers healthcare services, including prehospital care, to the 4.4 million people in Alberta.⁸

Data sources and cohort identification

Consecutive FBAO patient encounters were identified using Alberta Health Services’ Emergency Medical Services (EMS) Patient Care Record (Siren ePCR Version 4.6.26, Medusa Medical Technologies, Halifax, NS). Each encounter represents the electronic medical record created when EMS attends to a patient for assessment and treatment. The ePCR is the sole paramedic prehospital electronic charting system used in Alberta. It includes demographics, vital signs, and standardized reporting of incident, patient, and treatment details. It also includes a free-text narrative synopsis of the event, including information provided by the patient or bystanders. Importantly, this data is not available for hospital-based diagnostic coders limiting their knowledge of prehospital events to secondhand information from the in-hospital clinicians.⁹

An EMS research specialist developed an algorithm to identify patients in the cohort by querying the ePCR for patients with a relevant primary impression (ePCR field: “PRIMARY_IMPRESSION”; criteria: “Airway Obstruction”), EMS protocol name (ePCR field: “PROTOCOL_NAME”; criteria: “Foreign Body Airway Obstruction”) or treatment (ePCR field: “TREATMENT_NAME”; criteria: “Clear Airway”).

We deterministically linked data on emergency department visits from the National Ambulatory Care Reporting System (NACRS) and hospital admissions from the Discharge Abstract Database (DAD) to ePCR using a unique provincial healthcare number to obtain ICD codes for each identified patient encounter. The province assigns a healthcare number to individuals either at birth (if born in Alberta) or when they register (if they move into the province and must be completed within 3 months of arrival). Every healthcare encounter for an individual is linked using this number. Prehospital EMS uses an individual’s name and date of birth to identify their healthcare number (if it is not readily available) and link the encounter. If a patient remains unidentified prehospital, personnel work to identify and link to health records upon arrival to the healthcare facility.

NACRS captures administrative, clinical, and demographic information for hospital and community-based ambulatory care visits including emergency departments. DAD captures similar data for all hospital admissions. NACRS and DAD use the International Classification of Diseases–Tenth Revisions, Canada (ICD-10-CA) to record up to 10 (NACRS) or 25 (DAD) diagnostic codes per encounter.¹⁰ Both NACRS and DAD undergo regular quality testing and validation. Data collection methods for the NACRS and DAD are standardized and federally mandated.11, 12 We included all ages in the analysis and excluded patients without a provincial healthcare number.

EMS patient encounters where the patient was not transported to the hospital (due to refusal or death) would not have an associated NACRS or DAD entry [as they did not become an ED or hospital patient] and could not be linked. We therefore excluded these encounters from the analysis. Descriptive statistics of these unlinked cases are presented to identify the proportion of FBAO (and deaths) missed by only using ICD-10 codes for patient identification. We conducted a manual chart review of each unlinked case to determine mortality. Age, sex, and ICD-10 codes were unique variables in the database already and did not require extraction.

Foreign body airway obstruction definition

We used a pragmatic definition of FBAO as the reference standard. Individuals with the injury were identified if there was a history of object or substance not native to the airway introduced followed by clinical signs of obstruction (e.g., stridor, cyanosis, hypoxemia, inability to move air). Two physicians trained in emergency medicine and prehospital care (CD, JC) independently reviewed the ePCR synopsis, along with vital signs and treatment details, to determine whether the patient encounter represented a true FBAO. Initially, adjudicators were trained on the research protocol and FBAO definition using sample cases as examples. Next, they independently reviewed a randomly selected 20 cases for training purposes and had perfect agreement. Subsequently, a subset of encounters was reviewed by both and interrater agreement was measured using Cohen’s Kappa. Adjudicators were aware of the study’s purpose but blinded to ICD-10 code assignment during the review phase.

Validation of ICD-10 codes

We identified six ICD-10 codes, in both NACRS and DAD, related to FBAO (Table 1). Starting with T17.X (foreign body in the respiratory tract), we then built more complex combinations of ICD-10 codes and evaluated the performance of each.

Table 1.

International Classification of Disease (ICD) Codes [Version 10] related to foreign body airway obstructions.

ICD-10 Code	Definition
T17	Foreign body in the respiratory tract
T18.0	Foreign body in mouth
T18.1	Foreign body in esophagus
W78	Inhalation of gastric contents
W79	Inhalation and ingestion of food causing respiratory obstruction
W80	Inhalation and ingestion of non-food substances causing respiratory obstruction

Analysis

Descriptive statistics were calculated for patient demographic characteristics of the cohort.

Our primary outcome was diagnostic accuracy (sensitivity, specificity, and likelihood ratios). We calculated the sensitivity, specificity, and likelihood ratios (with associated 95% confidence intervals) for each combination of ICD-10 codes. Sensitivity refers to the proportion of FBAO cases that were correctly identified by the ICD-10 code combination. Specificity refers to the proportion of controls that were correctly not assigned a FBAO ICD-10 code. Positive likelihood ratios (+LR) were calculated as sensitivity/(1 − specificity) and negative likelihood ratios (-LR) were calculated as (1 − sensitivity)/specificity.13, 14 Positive and negative predictive values were also calculated and presented in Supplemental Material One. For each algorithm, Youden’s Index was calculated to summarize their performance. A Receiver Operator Curve (ROC) was constructed for each algorithm (Microsoft Excel, Microsoft Corporation [2018]).

All analyses were performed using STATA 16 and the “diagt” function was used for performance testing calculations.¹⁵ Diagnostic accuracy was explored for several subgroups including age categories, sex, cases where an intervention was performed (i.e., greater severity of FBAO), and cases involving hospital admissions (DAD database) where coders would have more information available.

Results

The search of the ePCR identified 3677 unique patient encounters during the study period (Fig. 1). Eighty-two cases (2.2%) were excluded due to a missing Alberta healthcare number. Of the remaining patient encounters, 2121 (58.3%) were subsequently treated at an emergency department or admitted to hospital following EMS assessment and assigned ICD-10 codes in NACRS or DAD, respectively.

Fig. 1.

Flow diagram of study participants.

After review by the two adjudicators, 825 cases were true FBAO. Agreement for classification of true FBAO was high (Cohen’s k = 0.802 [95%CI 0.688–0.910]). Table 2 describes the patient demographic characteristics.

Table 2.

Comparison of foreign body airway obstruction cases and controls.

Characteristic	Total	Foreign body airway obstructions	Controls	p-value
Age (Median [IQR])	50 (23–72)	58 (4–78)	46 (67–65)	0.004
Female	967 (45.7)	417 (50.7)	550 (42.4)	<0.001

IQR = Interquartile Range; SD = Standard Deviation.

Performance of ICD-10 classification

Table 3 describes the sensitivity, specificity, and likelihood ratios for combinations of FBAO-related ICD-10 codes. The combination of T17 (foreign body in the respiratory tract) and T18.0 (foreign body in the mouth) produced the greatest specificity (96.9% [95%CI 95.8–97.8%]), with the highest positive likelihood ratio (14.1 [95%CI 10.3–19.3]). Conversely, the combination including all related ICD-10 codes produced the greatest sensitivity (73.5% [95%CI 69.9–76.1%]) and lowest negative likelihood ratio (0.333 [95%CI 0.296–0.375]). Accuracy was not affected when only considering cases where an intervention was required to relieve the obstruction (i.e, greater severity of obstruction) or when stratified by age or sex (Supplemental Material Two). Diagnostic accuracy was also unchanged when only ED visits (NACRS database) were evaluated. When only hospital admissions (DAD database) were considered, there were improved specificity for the combination of T17 or T18.0 codes (98.7% [95%CI 97.2–99.5%]) and improved sensitivity for the combination of all FBAO-related codes (88.8% [95%CI 82.2–93.2%] (Supplemental Material Three). However, both improvements came with impaired sensitivity or specificity, respectively.

Table 3.

Performance characteristics of ICD-10 codes classifying foreign body airway obstructions.

ICD Code	Sensitivity	Specificity	+LR	−LR	Youden’s Index
T17	42.5 (39.1–46.0)	96.9 (95.8–97.8)	13.8 (10.1–18.9)	0.593 (0.559–0.629)	0.394
T17 or W78	43.2 (39.7–46.6)	95.3 (94.0–96.4)	9.17 (7.09–11.9)	0.597 (0.561–0.634)	0.385
T17 or W79	49.6 (46.1–53.0)	95.7 (94.4–96.7)	11.5 (8.8–15.0)	0.527 (0.492–0.564)	0.453
T17 or W80	47.2 (43.7–50.6)	92.4 (90.8–83.7)	6.17 (5.04–7.56)	0.572 (0.535–0.611)	0.396
W79 or W80	27.4 (24.4–30.6)	93.4 (91.9–94.7)	4.13 (3.27–5.21)	0.778 (0.744–0.813)	0.208
T17 or W79 or W80	53.1 (49.6–56.5)	91.3 (89.6–92.8)	6.09 (5.05–7.34)	0.514 (0.477–0.554)	0.444
T17 or W79 or W80 or W78	53.3 (49.9–56.8)	90.0 (88.3–91.6)	5.36 (4.49–6.39)	0.518 (0.481–0.559)	0.433
T17 or W79 or W80 or T181	66.8 (63.5–70.0)	84.1 (82.0–86.1)	4.2 (3.67–4.81)	0.395 (0.357–0.436)	0.509
T17 or W79 or W80 or T18.0	53.9 (50.5–57.4)	91.3 (89.6–92.8)	6.19 (5.13–7.46)	0.505 (0.468–0.544)	0.452
T17 or W79 or W80 or T18.0 or T18.1	67.5 (64.2–70.7)	84.1 (82–86.1)	4.25 (3.72–4.86)	0.386 (0.349–0.427)	0.516
T17 or W79 or W80 or T18.0 or T18.1 or W78	67.8 (64.4–70.9)	82.9 (80.7–84.9)	3.96 (3.48–4.50)	0.389 (0.351–0.431)	0.507
T17 or T18.0	43.4 (40.0–46.9)	96.9 (95.8–97.8)	14.1 (10.3–19.3)	0.584 (0.550–0.621)	0.403
T17 or T18.1	57.7 (54.2–61.1)	89.6 (87.8–91.2)	5.54 (4.67–6.57)	0.472 (0.435–0.512)	0.473
T17 or T18.0 or T18.1	58.4 (55.0–61.8)	89.6 (87.8–91.2)	5.61 (4.73–6.65)	0.464 (0.427–0.504)	0.480
T17 or W80 or T18.0	48.0 (44.5–51.5)	92.4 (90.8–93.7)	6.28 (5.13–7.69)	0.563 (0.526–0.602)	0.404
T17 or W79 or W80 or T18.0 or T18.1 or W78 or R06.8*	75.0 (71.9–78.0)	81.4 (79.2–83.5)	4.03 (3.58–4.55)	0.307 (0.272–0.346)	0.564

+LR = Positive Likelihood Ratio; −LR = Negative Likelihood Ratio; ICD = International Classification of Disease (Version 10) * Post-hoc analysis.

The most frequent principal diagnostic code used for FBAO cases not coded accurately were: R06.8 (other abnormalities of breathing = 20.9%), T18.8/9 (foreign body in alimentary tract, part unspecified or multiple parts = 14.1%), R13 (dysphagia = 12.5%), I46 (cardiac arrest, unspecified = 5.3%), and Z04.3 (examination and observation following other accident = 5.3%). Post-hoc analysis using these additional codes demonstrated improved accuracy only when R06.8 was added to the algorithm (Table 3).

Unlinked cases

From the provincial EMS database, there were 453 FBAO that were assessed by EMS and not transported (35.4% of total EMS patient encounters for FBAO). Of these, 253 (55.8%) required at least one intervention by bystanders or EMS to resolve the obstruction. Forty-four unlinked patients (9.7%) died prior to transport to hospital.

Discussion

This study found that the most specific case definition of FBAO was the combination of T17 (foreign body in the respiratory tract) or W18.0 (foreign body in the mouth). This produced a false positive rate of only 3%, however missed over 55% of FBAO cases. Conversely, the most sensitive case definition included every FBAO-related ICD-10 code and R06.8 (other abnormalities of breathing). This combination detected 670 (31.6%) more cases but included 328 (15.5%) more false positives.

By presenting multiple algorithms, users can select which balance of diagnostic accuracy best suits the aim of their research. If researchers are conducting a surveillance study where they are estimating the burden of injury or resource planning, then selecting a highly sensitive strategy which avoids underestimating the impact is appropriate. Conversely, if researchers desire a cohort which minimizes non-FBAO cases (such as in an intervention study), then a highly specific strategy should be selected. If a balance of both characteristics is desire, Youden’s Index suggests the optimal strategy is a combination of five codes (T17, W79, W80, T180.0 and T181).

Researchers have previously used ICD codes to study FBAO; however, different groups have used different algorithms.16, 17, 18, 19, 20, 21 In a recent study conducted in Italy, researchers used all subgroupings of the ICD-9 codes 933.x (foreign body in pharynx and larynx) and 934.x (foreign body in trachea, bronchi, and lung) to identify cases.¹⁶ In contrast, a study in the United States of America used a similar algorithm but without 933.0 (foreign body in the pharynx).¹⁷ Even a slight change to the inclusion criteria can result in dramatic differences to case estimates. From our study, if we compare the algorithm using only T17 versus T17 and 180.1, it results in a 15.2% increase in sensitivity (322 more cases included) and 7.3% decrease in specificity (155 more false positives). This demonstrates the importance of a validated, standardized case definition to enable improved comparison of FBAO burden between health regions.

ICD code search algorithms have been developed to study the epidemiology of many medical conditions, such as epilepsy and sepsis, as well as other injuries like drowning.22, 23, 24 We were unable to identify any ICD algorithm validation studies for FBAO, however, this approach has been performed in cardiac arrest research.25, 26, 27 Coppler et al. evaluated a single algorithm of ICD-9 codes and found a remarkably poor sensitivity of 4%. They noted that their gold standard registry had better outcomes than the cases that the ICD algorithm identified, and when only considering cases with an improved outcome, the ICD algorithm had a worse sensitivity (100% for those who never achieved return of spontaneous circulation versus 19% for those who survived to ED disposition). Coppler hypothesized that coders better identify cardiac arrests in those with active resuscitation upon arrival or die in the emergency department compared to cases where return of spontaneous circulation was achieved prehospital and then the patient stabilized.²⁵ If this effect occurs for FBAO also, it could impact the accuracy of FBAO algorithms as the choking occurs and is treated prehospital in most cases. Patients are often transported to the ED post-FBAO due to a complication such as respiratory distress from aspiration. This likely explains the significantly improved sensitivity in our study for identifying FBAO when R06.8 (other abnormalities of breathing) was added to the algorithm.

Another Canadian study investigated ICD codes for pediatric cardiac arrest.²⁷ They evaluated different algorithms for identifying cases, starting with just the cardiac arrest code (I46) and adding codes for specific etiologies or whether they were admitted to hospital. This study found their tested algorithms had excellent specificity (99.9–100%) and variable sensitivity (62.7–87.3%). They concluded that using the cardiac arrest code had too low sensitivity (62.7%) to use to identify cases, however, by adding further codes to algorithm they were able to obtain adequate accuracy.²⁷ This parallels our finding that using T17 (foreign body in the respiratory tract) alone produced insufficient performance characteristics to use as a case definition.

Regardless of case definition, over 35% of FBAO that require EMS attendance were not transported to hospital, and therefore not identified in DAD or NACRS. Over half of these cases required a choking intervention, with 10% of patients dying. These cases must be considered when assessing the population-level burden of FBAO. Utilizing a separate database which also identifies all out-of-hospital deaths and assigns ICD-10 codes based on cause (such as Vital Statistics in Canada) could improve estimates of incidence and mortality, but a study is needed to determine the accuracy of this approach.

Limitations

Readers should view our results within the context of several limitations. We obtained data from a single Canadian province, with its present prehospital system and documentation processes, therefore it may not be representative of other countries or regions. Although we utilized prehospital, narrative data which was recorded at the time of the incident to inform our gold standard, diagnosis of FBAO is entirely reliant on the interpretation of signs, symptoms and preceding events by the layperson responders and EMS. As a result, misclassification of cases is possible. Finally, we analyzed ICD-10 codes for our case definition so it may not be transferable to other iterations. Given the launch of ICD-11 codes in 2022, most countries will be transitioning in the coming years.²⁸ We have provided a suggested conversion table for our included codes in Supplemental Material Four, however, a future study to confirm equivalency will be needed.

Conclusion

Accurate identification of foreign body airway obstruction is possible using routinely collected, administrative health data with ICD-10 codes, which can facilitate the surveillance of FBAO moving forward. However there exists a trade off in sensitivity versus specificity depending on the case definition used. Over one-third of EMS foreign body airway obstruction encounters, 10% of which are fatal, do not result in hospital transport and are not captured by ICD codes in health administrate databases without prehospital data.

Funding

The work was supported by the Canadian Association of Emergency Physicians’ Junior Investigator Grant 2022.

CRediT authorship contribution statement

Cody L Dunne: Conceptualization, Methodology, Formal analysis, Data curation, Writing – original draft, Writing – review & editing, Funding acquisition. Julia Cirone: Data curation, Writing – original draft, Writing – review & editing. Andrew D McRae: Conceptualization, Methodology, Formal analysis, Supervision, Writing – review & editing, Funding acquisition. Ian Blanchard: Conceptualization, Methodology, Data acquisition, Writing – review & editing. Jayna Holroyd-Leduc: Conceptualization, Methodology, Writing – review & editing; Khara Sauro: Conceptualization, Methodology, Formal analysis, Supervision, Writing – review & editing, Funding acquisition.

Declaration of competing interest

The authors declare the following financial interests/personal relationships which may be considered as potential competing interests: Cody Dunne reports financial support was provided by Canadian Association of Emergency Physicians.

Appendix A. Supplementary material

The following are the Supplementary material to this article: