Measuring accuracy of International Classification of Diseases codes in identification of patients with non-cystic fibrosis bronchiectasis

Abstract

Introduction

Non-cystic fibrosis bronchiectasis is a disease which is increasing in incidence and prevalence worldwide. The incidence of the disease is frequently estimated using databases that rely on International Classification of Diseases, ninth and tenth revisions, clinical modification (ICD-9-CM/ICD-10-CM) discharge diagnoses. Code accuracy has proved to be a major issue for other diagnoses using ICD codes. This study aims to investigate the accuracy of the ICD codes for the diagnosis of non-cystic fibrosis bronchiectasis.

Methods

This is a retrospective diagnostic accuracy study which compares the radiologist's diagnosis of bronchiectasis with the ICD code reflection of that diagnosis at discharge.

Results

Sensitivities were 34% (same for both ICD-9-CM and ICD-10-CM windows) and specificities ranged from 69% for the ICD-9-CM window to 81% for ICD-10-CM window.

Conclusion

We observed that ICD codes are an insufficient method to identify patients with a radiologist diagnosis of bronchiectasis.

Shareable abstract

ICD discharge diagnosis is insufficient to identify patients with a radiological diagnosis of non-cystic fibrosis bronchiectasis. The disease is probably more prevalent and clinically important than current data suggest. https://bit.ly/483iBkG

Introduction

In observational studies, identification of a representative sample of subjects is paramount to proper inference. In this setting, International Classification of Diseases (ICD) codes are frequently used to conduct this sampling; however, poor accuracy of these codes can be problematic. In particular, non-cystic fibrosis bronchiectasis (NCFB) can be difficult to diagnose and prior to designing additional studies based on ICD codes, we wished to evaluate the accuracy of these codes.

NCFB is characterised clinically by the presence of chronic cough and sputum production, with the presence on chest computed tomography (CT) imaging primarily of an increased bronchial:arterial ratio and smooth dilation of bronchi to the periphery. There are ancillary radiological findings, such as the presence of mucus plugs, and strings or clusters of cysts, which help to make the diagnosis more certain [1]. Prior to CT criteria being published, the diagnosis was routinely made by chest radiography findings as described by Gudbjerg [2], but the gold standard remained bronchography with iodinated airway contrast such as iotrolan and propyliodone. In the modern era, high-resolution computed tomography (HRCT), with its ease of acquisition compared to bronchograms, the improved sensitivity and specificity compared to chest radiography, and ability to identify coexistent pulmonary pathology, has become the primary imaging tool utilised in making the diagnosis [3, 4].

The ICD ninth revision, clinical modification (ICD-9-CM) coding system was instituted in 1977 and continued to be used exclusively until 2009 when the new ICD-10-CM was introduced. In 2015, use of the ICD-9-CM was discontinued [5]. The ICD coding system provided a convenient way for physicians to unify diagnoses and quickly became a vital starting point for research and reimbursement. However, code accuracy has been a major issue. In some studies looking at a wide range of diagnostic codes, the error rates range from 20% to 80% and may represent errors in the patient trajectory; from clinician assignment of disease status to the radiologist identification of subtle disease, in the case of NCFB [6]. Errors in the paper trail from physicians to coders may also result in inappropriate imputation of disease states [7]. Studies that have compared the ICD-9-CM compared to ICD-10-CM have largely found similar validity between the coding versions. It is worthwhile noting that NCFB was not one of the included diagnoses [8]. To date, no study has looked at the accuracy of the ICD codes for NCFB or compared the ICD-9-CM to the ICD-10-CM in this regard. The overall goal of this study was to investigate the diagnostic accuracy of the ICD-9-CM and ICD-10-CM coding system in identifying patients with NCFB in a cohort of patients who have had a chest CT for an indication, or a final diagnosis, of bronchiectasis.

Methods

Study population

Subjects were sampled from a single health system which includes five hospitals (including the area's only tertiary-care medical centre) and an extensive outpatient network in Western Massachusetts (USA). The system has a central radiology department which manages the acquisition and interpretation of CT images across several health systems in the region. Eligible patients included a cohort of adults receiving a chest CT scan that was either positive for bronchiectasis or was performed for a clinical indication to rule out bronchiectasis from 1 June 2012 through 30 June 2019. With the goal of limiting the sample to patients within our catchment area (i.e. they would receive follow-up care within our system), we also limited the sample to patients who had a visit within our main hospital or hospital clinics within ±365 days from the CT scan. We excluded patients with a coexistent diagnosis of cystic fibrosis or pulmonary fibrosis as a discharge diagnosis (ICD-9-CM codes 277.0, 516.31/515 or ICD-10-CM codes E84/E84.9, J84.10/J84.112), or if the diagnosis was identified on review of the medical records, but not coded at discharge. The study was reviewed and approved by the institutional review board.

Data collection

Patients were identified using our radiology database (FUJIFILM Healthcare Americas Corporation Synapse Radiology PACS version 5.7.245US), using keywords from the final report. First, we included all reports with the term “bronchiectasis” while simultaneously excluding reports with the term “no bronchiectasis” within five words of the initial search term. A second search was conducted using the Boolean term “? bronchiectasis” and results from both queries were combined. If multiple CT scans were conducted in the study period, the earliest instance was chosen. Among these eligible patients, we randomly sampled 750 subjects whose billed visit occurred prior to 1 October 2015 (the ICD-9-CM window) and 750 subjects whose billed visit occurred on or after 1 October 2015 (the ICD-10-CM window). Study data was collected and managed using REDCap [9]. Initially sampled records were manually reviewed by three attending physicians (two pulmonologists and one radiologist) for documentation of the presence or absence of a radiological diagnosis of bronchiectasis in the final radiologists’ report.

Following record review, we paired the CT scan with the closest billed visit in our inpatient or outpatient hospital system with an ICD-9-CM discharge code (494.0, 494.1, 011.50, 748.61) or ICD-10-CM discharge code (J47.0, J47.1, J47.9) for bronchiectasis. This pairing was conducted twice: once using a caliper of ±365 days from the CT scan (to capture pre-existing or prevalent ICD codes) and once using a caliper of 0–30 days from the CT scan (to capture codes directly related to the diagnosis). Subjects without a relevant ICD code were paired with descriptive characteristics from the billed visit closest to the CT scan. Additional data collected included patient age, sex and race and the characteristics of the CT scan (inpatient versus outpatient, indication).

Statistical analysis

Our primary analysis focused on the accuracy of ICD codes compared to the documented radiological diagnosis in the medical record (gold standard). Accuracy was estimated using sensitivity and specificity with 95% confidence intervals. Sensitivity analyses were also conducted by stratifying results by ICD type (ICD-9-CM versus ICD-10-CM) and indication for the CT scan (clinical suspicion, incidental finding versus follow-up). Sample size estimates were based on our ability to detect whether the minimum sensitivity and specificity were ≥80%. This was chosen as we felt that either false positive or false negative rates of ≥0.20 would be sufficient to suggest the lack of accuracy in using ICD codes to identify patients with NCFB. Given this, we estimated that with a sample size of 750 subjects (∼85% of whom had radiologically diagnosed bronchiectasis), we would need to observe sensitivity and specificity values of ≥0.86 to have 80% power (α=0.05) to detect this minimum threshold. Therefore, 1500 records were randomly sampled (750 for each ICD window). In addition, due to the diagnostic challenges related to NCFB, we wished to confirm the validity of the original radiologists reading of the CT exam. Therefore, we embedded a substudy in which we randomly sampled 80 patients and verified the original radiologist read using a consensus panel of three physicians (two pulmonologists with special interest and experience in bronchiectasis, and a chest radiologist). The reviewers were mid-career physicians with an average of 23 years’ practice post their basic medical degree. For this portion of our study, we used the following definition of bronchiectasis: a bronchoarterial ratio of ≥1, a lack of tapering of the bronchi toward the periphery, and the presence of peribronchial thickening with or without subsegmental collapse/consolidation. Each physician independently reviewed CT scans while being blinded to the original radiologist reading and to the other reviewers’ opinions. Sensitivity and specificity were calculated comparing the original radiologist diagnosis to the consensus panel as the gold standard.

Results

Patient selection and characteristics

4364 subjects were initially identified from our radiology database, with 2279 eligible for sampling (figure 1). Following our initial random sample of 1500 records, when pairing with billing data using our 365-day caliper, we excluded 162 following identification of a coexistent diagnosis of either cystic fibrosis or pulmonary fibrosis on manual review. An additional 19 patients were not included in the final population sample, as they did not have a chest CT available in our PACS system. Two were discovered to be duplicate patients and three had no billed visits within the larger system, resulting in a final sample size of 1314 individuals; 661 (50.3%) within the ICD-9 window and 653 (49.7%) within the ICD-10 window. When repeating the pairing using the 0–30-day caliper, an additional 13 subjects had no billed visits within the system, resulting in a final sample size of 1301.

FIGURE 1.

Flow diagram demonstrating patient selection. CT: computed tomography; ICD-9/10: International Classification of Diseases, ninth/tenth revision; CF: cystic fibrosis; PF: pulmonary fibrosis.

Our sample (n=1314) had a mean±sd age of 70±14 years, which was similar across both ICD versions (table 1). Patients were more likely to be female (54%), while most identified as white race (86%) and non-Hispanic (84%). CT scans were primarily conducted on outpatients (70%) and most CT indications (83%) were for indications other than to investigate a diagnosis of bronchiectasis. Interestingly, this was more common in the ICD-10-CM window (89%) compared to ICD-9 window (78%). In addition, very few prevalent (follow-up) CT scans were identified (1%) among those with a CT scan during the ICD-10 window. The median (interquartile range) for time from the CT scan to the billed visit was 0 (−0.1–0.5) days.

TABLE 1.

Sample characteristics by International Classification of Diseases, ninth and tenth revisions, clinical modification (ICD-9-CM/ICD-10-CM) window

	Overall	ICD-9-CM window	ICD-10-CM window
Subjects	1314	661	653
Age at CT scan years	69.9±14.1	69.4±14.7	70.4±13.5
Patient gender
Female	709 (54.0)	364 (55.1)	345 (52.8)
Male	605 (46.0)	297 (44.9)	308 (47.2)
Race
Asian	44 (3.3)	25 (3.8)	19 (2.9)
Black or African American	106 (8.1)	67 (10.1)	39 (6.0)
Native American or Alaska Native	1 (0.1)	0 (0.0)	1 (0.2)
Native Hawaiian or Pacific Islander	1 (0.1)	0 (0.0)	1 (0.2)
White	1124 (85.5)	547 (82.8)	577 (88.4)
Missing	38 (2.9)	22 (3.3)	16 (2.5)
Hispanic
Non-Hispanic	1102 (83.9)	543 (82.1)	559 (85.6)
Hispanic	114 (8.7)	69 (10.4)	45 (6.9)
Missing	98 (7.5)	49 (7.4)	49 (7.5)
Location of CT scan
Outpatient	923 (70.2)	462 (69.9)	461 (70.6)
Inpatient	391 (29.8)	199 (30.1)	192 (29.4)
CT indication
Incident (clinical suspicion)	137 (10.4)	73 (11.0)	64 (9.8)
Incident (not primary reason for CT)	1094 (83.3)	513 (77.6)	581 (89.0)
Prevalent (follow-up CT)	83 (6.3)	75 (11.3)	8 (1.2)
Location of billed ICD visit
Hospital	1167 (88.8)	574 (86.8)	593 (90.8)
Clinic	6 (0.5)	4 (0.6)	2 (0.3)
Other outpatient	137 (10.4)	80 (12.1)	57 (8.7)
Missing	4 (0.3)	3 (0.5)	1 (0.2)
Days from CT scan to visit
Median (interquartile range)	−0.0 (−0.1–0.5)	−0.0 (−0.1–0.5)	−0.0 (−0.0–0.5)
Range	−364.0–364.0	−364.0–364.0	−307.6–362.8

Data are presented as n, mean±sd or n (%), unless otherwise stated. CT: computed tomography.

Accuracy of ICD codes for diagnosing non-cystic fibrosis bronchiectasis

With regards to validity, our primary analysis utilised the window pairing CT scans to billed visits within 0–30 days (n=1301) and we observed that ICD codes are an insufficient method to identify patients with a radiologist diagnosis of bronchiectasis (table 2, figure 2). Specifically, sensitivities were 34% (for both ICD-9-CM and ICD-10-CM windows) and specificities ranged from 69% for the ICD-9-CM window to 81% for the ICD-10-CM window. After further stratification by CT indication, when the CT scan was conducted for clinical suspicion or for follow-up of a prevalent bronchiectasis, we observed the sensitivities to improve to 73% and 70%, respectively. However, specificities were worse (∼50% for both, although confidence intervals were very wide). Most indications (n=1218; 94%) for CT scans resulted in incidental findings of bronchiectasis where the sensitivity was 28% and the specificity was 96%. After expanding the window for the billed visit from 0–30 days following the CT scan to ±365 days, we observed a minor improvement in the sensitivities (∼42%) and slight decreases in the specificities (∼70%) (supplementary table S1). In terms of our substudy assessing validity of the original radiologists reading compared to a consensus panel, 75 patients had CT scans available for review. Among these subjects we observed that the sensitivity of the original radiologist's diagnosis was 95.1% (95% CI 86.3–99.0%) and the specificity was 28.6% (95% CI 8.4–58.1%).

TABLE 2.

Bronchiectasis accuracy statistics

	Radiological diagnosis			Sensitivity % (95% CI)	Specificity % (95% CI)
	No	Yes	Total
ICD code overall				34.0 (31.2–36.8)	74.4 (67.1–80.8)
No	125	748	873
Yes	43	385	428
Total	168	1133	1301
ICD-9 only				34.3 (30.4–38.3)	69.1 (58.8–78.3)
No	65	374	439
Yes	29	195	224
Total	94	569	663
ICD-10 only				33.7 (29.8–37.8)	81.1 (70.3–89.3)
No	60	374	434
Yes	14	190	204
Total	74	564	638
Incident: clinical suspicion for bronchiectasis				73.4 (60.9–83.7)	47.2 (35.3–59.3)
No	34	17	51
Yes	38	47	85
Total	72	64	136
Incident: nonbronchiectasis indication				28.4 (25.6–31.4)	95.7 (89.5–98.8)
No	90	707	797
Yes	4	281	285
Total	94	988	1082
Prevalent: bronchiectasis follow-up CT				70.4 (59.2–80.0)	50 (1.3–98.7)
No	1	24	25
Yes	1	57	58
Total	2	81	83

Data are presented as n, unless otherwise stated. International Classification of Diseases (ICD) window ranges from 0 to 30 days following the computed tomography (CT) scan.

FIGURE 2.

Sensitivity and specificity of International Classification of Diseases, ninth and tenth revisions, clinical modification (ICD-9-CM, ICD-10-CM), and combined ICD codes in detecting non-cystic fibrosis bronchiectasis when compared to the initial radiologist's diagnosis. Discharge diagnosis sampled at ±30 days and ±365 days window from chest computed tomography (CT). Error bars represent 95% CI.

Discussion

In this single-centre study, we observed extremely poor diagnostic accuracy of ICD codes for identifying patients with NCFB. This suggests that alternative sampling strategies, such as clinical data repositories, may be required to conduct observational studies with representative samples. The accuracy of the ICD codes may be reflective of the relative weight for reimbursement attached to particular codes, the rank order of disease importance to a clinical encounter, and physicians’ awareness of the importance of NCFB as a disease entity. In a retrospective analysis of the diagnostic accuracy of ICD-10-CM for appendicitis compared to chart adjudication, there was a positive predictive value of 95.3% for acute appendicitis and 67.8% for unspecified appendicitis. The positive predictive value for unspecified appendicitis increased to 99% when combined with a surgical code according to the Nordic Medico-Statistical Committee [10]. This suggests that alternative strategies for abstracting data for research may be helpful in identifying patient groups of interest. That ICD coding from the medical record may also miss events or illnesses which, although important, were not explicitly included in the physician documentation is demonstrated in a review of the accuracy of ICD-10-CM to identify self-harm events in a series looking at data from patients reporting suicidal ideation. It was found on chart review that 28.2% of those coded as undetermined, 7.9% of those coded as accidental and 11% of those without coding of intent qualified for being coded as self-harm [11]. Given that the importance of NCFB is just gradually emerging in the medical literature, many physicians might not consider this disease sufficiently important to rank in the coding [12, 13].

One might sense that physicians with an interest in the disease, or who may be investigating primarily to diagnose bronchiectasis would subsequently code for disease if found on radiography. As the prevalence increases, by as much as 8.7% annually, we may expect this to be considered a priority diagnosis for coding [14]. The real-world situation is that much of coding is driven by concerns regarding reimbursement and rank order of diagnoses. However, our understanding of disease prevalence, and the public health measures that this drives in response to disease, are affected by coding. This is what makes this study so important.

At our institution, most patients carry either a Medicare or Medicare-managed plan; this is not surprising, considering the age range of our cohort, and is consistent with previously published epidemiological studies [14–17]. Very few patients are uninsured reflecting the near universal coverage mandated by the Commonwealth of Massachusetts initially and subsequently the implementation of the Affordable Care Act during the study period.

The implications of this study are that the actual incidence and prevalence of NCFB may be much higher than reported in studies based on ICD sampling. An important question might be, why is it important to accurately identify and codify patients with NCFB who are not directly referred by a physician for chest CT. The clinical symptoms of bronchiectasis are protean and often share homology with other diseases such as chronic bronchitis and severe obstructive asthma. The European Bronchiectasis registry reports that 25.5% and 31% have COPD and asthma, respectively [18]. Failure to recognise disease can lead to poor recognition of a subset of patients who many benefit from skilled care directed towards their bronchiectasis with increased morbidity and hospitalisations, progressive failure to thrive and death [19]. This is further complicated by the fact that the clinical symptoms and signs of bronchiectasis are variable and require high clinical suspicion for the HRCT to be ordered. Identifying and addressing this disease could improve quality of life and decrease health expenditure. We do wish to note several limitations to our study. First, these findings are from a single health system in the United States and may reflect coding practices of that system (or country) which are not generalisable to other health systems. However, as outlined earlier, due to rank ordering and reimbursement issues, we would not expect major differences within the United States that would change our overall inference. Second, we present data suggesting high sensitivity (95%), but low specificity (29%) of the original radiologist reading compared to a consensus review. In this setting, we are mostly concerned with a high sensitivity (e.g. minimising false negative rate) to cast the widest net to capture patients with “true” NCFB. Our high sensitivity supports this; however, given our low specificity, we would also anticipate a high false positive rate or a potential for overdiagnosis by a radiologist. While this is an important question for future research (perhaps this is influenced by the severity of disease), we are not particularly concerned about this finding regarding our primary research goal, since the overall accuracy of ICD codes was so poor, the low specificity is irrelevant. Third, our sample had a smaller proportion of females than other published populations; 54% in our sample versus 79% in the US Bronchiectasis Registry and 60.9% in a European registry [18, 20]. Post-menopausal females tend to have more severe disease [21]. Milder disease with minimal symptoms might affect the rank order of bronchiectasis as a diagnosis for clinicians and coders. Even with our limitations, we believe that researchers should be aware of the potential accuracy issues in using discharge diagnoses, so that, when possible, research methods are developed to capture patients with bronchiectasis that is separate from use of ICD discharge diagnoses. However, additional confirmatory studies would be required to generalise our findings.

Conclusion

Currently, ICD diagnostic codes should not be used to identify patients with NCFB. The currently stated incidence and prevalence is most likely an underestimate of the disease. Using radiology databases could be one way of arriving at a more concise estimate of the incidence and prevalence of this disease. To improve generalisability of our findings, a larger multicentre study is needed.

Supplementary material

Please note: supplementary material is not edited by the Editorial Office, and is uploaded as it has been supplied by the author.

Supplementary material 00715-2023.SUPPLEMENT ^{(582.4KB, pdf)}