Accuracy of identifying pediatric acute bacterial sinusitis diagnoses in outpatient claims data

Abstract

Purpose

Acute bacterial sinusitis is among the most frequent outpatient infections in children and adolescents and is well suited to study in large healthcare utilization databases, but the validity of International Classification of Diseases, 10^th Revision (ICD-10) codes together with antibiotic prescriptions to identify cases of acute bacterial sinusitis has not been established. We aimed to evaluate the validity of ICD-10 codes combined with antibiotic prescriptions to identify new diagnoses of acute bacterial sinusitis among pediatric patients evaluated in the outpatient setting.

Methods

Children and adolescents aged 17 years and younger with an outpatient diagnosis of acute sinusitis along with an antibiotic prescription from an ambulatory facility affiliated with the Mass General Brigham health system were identified via a clinical data warehouse. Patients were stratified by age (0-5 years, 6-11 years, and 12-17 years), and fifty cases per age group were randomly sampled. Medical records were independently reviewed by two pediatric infectious diseases physicians to assess for the documentation of a clinician-defined diagnosis of acute bacterial sinusitis. Positive predictive values (PPVs) and 95% confidence intervals (CIs) were calculated.

Results

150 patients were included in the final cohort. Frontal, maxillary, and “unspecified” sinuses accounted for 88% of the diagnoses. The positive predictive value of the algorithm to identify clinician-defined diagnoses of acute bacterial sinusitis was 92% (95% CI 87%,95%). The PPVs were consistent across age strata.

Conclusions

ICD-10 codes for acute sinusitis, when paired with a same-day antibiotic prescription, have a high positive predictive value among a cohort of pediatric patients, suggesting that they can be used to study new acute bacterial sinusitis diagnoses with claims.

Introduction

Pediatric patients account for a disproportionately large percentage of antibiotic prescriptions, with acute sinusitis having the highest rate of antibiotic prescribing among outpatient diagnoses in children and adolescents.¹ Acute bacterial sinusitis is an acute upper respiratory tract infection characterized by nasal drainage or cough, with or without fever.^2,3 It may be characterized by initial improvement in symptoms followed by worsening (suggestive of a primary viral and secondary bacterial infection), severe symptoms (high fever with nasal discharge for three or more days), or persistent symptoms (10 or more days).³ Inappropriate antibiotic prescribing is common in this setting and results in substantial increases in adverse events when unnecessary or excessively broad antibiotics are prescribed.^1,4 In addition to treatment-related adverse events, excessive antibiotic use is also a driver of antibiotic resistance: 1.27 million infections are attributed to multidrug resistant organisms each year, a number that is predicted to grow substantially in coming years.⁵ Meta-analyses have identified a benefit to antibiotic treatment of acute bacterial sinusitis when compared to placebo, and evidence is needed to identify which antibiotic is the most effective initial treatment for pediatric patients.^6–11

While several small RCTs have evaluated the benefit of antibiotics compared to placebo for the treatment of acute bacterial sinusitis, there are few published studies evaluating the prescribing patterns or the comparative safety and effectiveness of narrow and broad-spectrum antibiotics for acute sinusitis in children and adolescents.^8–12 Healthcare utilization databases that collect administrative and billing information as a part of the routine practice of care are well suited to address these research needs through robust prescribing information and International Classification of Diseases (ICD) codes that capture diagnostic information. As such, real-world evidence studies using these data sources can be helpful to identify the most narrow-spectrum effective treatments for acute sinusitis and subsequently help to drive down antibiotic overuse.

To ensure that diagnoses are accurately identified in these data sources, and thus potential outcome misclassification is minimized, a validated algorithm to identify acute sinusitis is needed. This study aimed to assess the validity of ICD version 10 (ICD-10) codes paired with same-day antibiotic prescriptions to accurately identify new diagnoses of acute bacterial sinusitis among children and adolescents in the outpatient setting.

Methods

Cohort selection

The Research Patient Data Registry (RPDR) is a clinical data warehouse for patients receiving care in the Mass General Brigham (MGB) Healthcare system in Massachusetts, USA. We queried the RPDR to identify subjects who had an outpatient encounter at a practice affiliated with Brigham and Women’s Hospital, Massachusetts General Hospital, or Newton-Wellesley Hospital with (1) age < 18 years, (2) an ICD-10 code for acute sinusitis in any position (Table 1), and (3) a prescription written for one of the following antibiotics: amoxicillin, amoxicillin-clavulanate, azithromycin, cefaclor, cefadroxil, cefdinir, cefditoren, cefixime, cefpodoxime, cefprozil, ceftibuten, cefuroxime, cephalexin, ciprofloxacin, clarithromycin, clindamycin, delafloxacin, dicloxacillin, doxycycline, erythromycin, levofloxacin, linezolid, moxifloxacin, penicillin V, tedizolid, telithromycin, tetracycline, or trimethoprim-sulfamethoxazole. These antibiotics were selected as they comprise a comprehensive list of the oral antibiotics routinely prescribed to children and adolescents in the United States. Antibiotics were included as a component of the case definition to increase the likelihood that the cases identified were due to bacterial, rather than viral, infections, recognizing that acute sinusitis is likely overtreated with antibiotics.¹³ We chose ICD-10 codes for validation to support studies using recent data (ICD-10 was implemented in the United States in October 2015). To permit application to earlier years, we also identified corresponding ICD-9 codes using the Centers for Medicare & Medicaid Services General Equivalence Mappings (Table 1).¹⁴

Table 1:

International Classification of Diseases, 9^th Revision and 10^th Revision codes for acute sinusitis

Outcome	ICD-9 code	ICD-10 code
Acute maxillary sinusitis	461.0	J01.00
Acute frontal sinusitis	461.1	J01.10
Acute ethmoidal sinusitis	461.2	J01.20
Acute sphenoidal sinusitis	461.3	J01.30
Acute pansinusitis	461.8	J01.40
Other acute sinusitis	461.8	J01.80
Acute sinusitis, unspecified	461.9	J01.90

After identification of encounters meeting the inclusion criteria, we stratified patients by age to ensure representation across the pediatric age range: 0-5 years, 6-11 years, and 12-17 years. Within each age strata, random numbers between 0 and 1, to 15 digits, were generated for each record, and records were subsequently ordered by random number, from largest to smallest. We then sampled the first 50 patients in each age stratum with complete clinical documentation. The final study population included 150 patients treated between October 2015 and March 2022.

Medical Record Review

The gold standard was defined a priori as documentation of a diagnosis of acute bacterial sinusitis by the treating clinician in the assessment and plan section of the encounter note. While there are diagnostic criteria from the American Academy of Pediatrics for acute bacterial sinusitis, we anticipated that the majority of medical records would not have detailed enough documentation to permit evaluation of these criteria, which informed our choice of gold standard.³ Medical records of selected patients were independently reviewed by two pediatric infectious diseases physicians (T.J.S. and H.W.) for the presence of the clinician-documented diagnosis of acute sinusitis. If there was initial disagreement in assessment of the outcome, the two reviewers discussed the case to reach consensus.

Statistical analysis

Using the clinician-documented diagnosis of acute bacterial sinusitis as the gold standard, positive predictive values (PPVs) with 95% confidence intervals were calculated. PPVs represented the proportion of patients identified via the algorithms with a medical record confirmed diagnosis. For all false positive cases, we further explored why these patients were wrongly classified as having the outcome. The Brigham and Women’s Hospital and Boston Children’s Hospital Institutional Review Boards determined this study to be exempt and waived the need for patient informed consent.

Results

1781 encounters meeting the claims-based definition of acute sinusitis were identified in the RDPR at the time of the query (April 27, 2022). The first qualifying encounter for each patient was retained, resulting in 1341 patients/encounters (Figure 1). After stratification, there were 389, 369, and 583 patients for ages 0-5 years, 6-11 years, and 12-17 years, respectively, prior to random selection to identify the final cohort of 150 patients.

Figure 1:

Cohort development

Among the final cohort, patients were treated at 30 clinics, including pediatric, family medicine, medicine/pediatrics, and urgent care settings. The minimum patient age was 5 months (Table 2). Based upon ICD coding, among patients 6 years and older, frontal and maxillary sinuses accounted for at least 50% of the sites of infection, while for children under 6 years old, “unspecified” sinus location accounted for 66% of infections.

Table 2:

Demographic and clinical characteristics of final cohort

Characteristic, n (%)	Total	0-5 years	6-11 years	12-17 years
Female	75 (50)	17 (34)	28 (56)	30 (60)
Race
Asian	5 (3)	2 (4)	3 (6)	0 (0)
Black	15 (10)	6 (12)	2 (4)	7 (14)
Other	17 (11)	7 (14)	4 (8)	6 (12)
Two or more races	7 (5)	3 (6)	2 (4)	2 (4)
Unknown	20 (13)	11 (22)	6 (12)	3 (6)
White	86 (57)	21 (42)	33 (66)	32 (64)
Ethnicity
Hispanic	14 (9)	5 (10)	3 (6)	6 (12)
Not Hispanic	111 (74)	35 (70)	42 (84)	34 (68)
Unknown	25 (17)	10 (20)	5 (10)	10 (20)
Sinus
Ethmoid	6 (4)	3 (6)	1 (2)	2 (4)
Frontal	20 (13)	1 (2)	9 (18)	10 (20)
Maxillary	42 (28)	9 (18)	16 (32)	17 (34)
Other	8 (5)	4 (8)	3 (6)	1 (2)
Pansinusitis	2 (1)	0 (0)	0 (0)	2 (4)
Sphenoidal	1 (1)	0 (0)	1 (2)	0 (0)
Unspecified	71 (47)	33 (66)	20 (40)	18 (36)
Antibiotic prescribed
Amoxicillin	57 (38)	29 (58)	19 (38)	9 (18)
Amoxicillin-clavulanate	53 (35)	15 (30)	17 (34)	21 (42)
Azithromycin	24 (16)	4 (8)	9 (18)	11 (22)
Cefdinir	7 (5)	1 (2)	3 (6)	3 (6)
Cefixime	1 (1)	0 (0)	1 (2)	0 (0)
Cefprozil	4 (3)	1 (2)	1 (2)	2 (4)
Cefuroxime	1 (1)	0 (0)	0 (0)	1 (2)
Doxycycline	2 (1)	0 (0)	0 (0)	2 (4)
Levofloxacin	1 (1)	0 (0)	0 (0)	1 (1)

The positive predictive value was 92% (95% CI, 87%, 95%) overall. The PPV was identical across age strata: 92% (95% CI, 81%,97%) for all strata (Table 3). After initial review, the two reviewers’ assessments agreed for 142/150 patients (94.7%). Among the 8 patients for whom there were discrepant initial reviews, two were ultimately assessed to have acute sinusitis (true positives) and six were not (false positives). Overall, there were 12 false positive cases. For these patients, clinical documentation indicated that the primary diagnoses were allergic rhinitis (3 cases), persistent or recurrent sinusitis (2 cases), sore throat (2 cases), unrelated diagnoses (4 cases), and viral upper respiratory tract infection (1 case).

Table 3:

Positive predictive values of ICD-10 codes for acute sinusitis

Ages (y)	Cases identified, n	Cases reviewed, n	True positives, n	False positives, n	PPV (95% CI)	Reasons for false positives
Total	1781	150	138	12	92% (87%, 95%)
0-5	497	50	46	4	92% (81%, 97%)	Recurrent sinusitis (1), viral infection (1), allergic (1), unrelated diagnosis (1)
6-11	467	50	46	4	92% (81%, 97%)	Unrelated diagnoses (2), allergic rhinitis (1), purulent nasal discharge (1)
12-17	817	50	46	4	92% (81%, 97%)	Sore throat (2), allergic (1), recurrent sinusitis (1)

Discussion

In this study to validate the accuracy of ICD-10 codes together with antibiotic prescriptions to identify new diagnoses of pediatric acute bacterial sinusitis, the PPV was high at 92%. PPVs were consistent across all three age strata, suggesting low rates of misclassification regardless of pediatric age. To our knowledge, previous studies have not validated ICD-10 codes to identify a frequently encountered outpatient infection among children and adolescents, despite these diagnoses accounting for a large fraction of antibiotic use in pediatric patients.¹⁵ A study by Hwee and colleagues evaluated the validity of ICD-8 and ICD-9 codes truncated to three digits for several outpatient infections in children and found a lower PPV for sinusitis when using ICD codes alone (67%, 95% CI: 35%, 90%), although the small number of subjects resulted in low precision. In contrast, the findings herein suggest that when paired with an antibiotic prescription, ICD-10 codes for acute sinusitis can accurately identify cases of acute bacterial sinusitis ensuring this outcome can validly be evaluated in comparative safety and effectiveness studies.¹⁶

Other prior studies to validate infectious diseases diagnoses in children have largely focused upon inpatient infections.^17–19 This study employed a large sample of pediatric patients from outpatient clinics affiliated with a large, academic health system and included patients from 0-17 years old. Cases were independently adjudicated by two pediatric infectious diseases physicians. While we did not validate ICD-9 codes in the identification of cases of pediatric acute sinusitis, we did find good correspondence between ICD-9 and ICD-10 codes for acute sinusitis, and thus expect similar performance of our algorithm using ICD-9 codes.

This study has several limitations. We did not have data on children or adolescents with acute bacterial sinusitis who did not meet our algorithmic definition, and thus cannot report sensitivity or specificity. Estimating these test characteristics of the outcome defining algorithm would require the evaluation of (a random sample of) thousands of pediatric charts to assess whether any new diagnoses of acute sinusitis were missed with the algorithm and to report on the sensitivity with any level of confidence. This is practically not feasible. Importantly, we deliberately opted for an outcome definition expected to have high specificity, even at the cost of sensitivity. Under these conditions, the relative risk estimates from causal inference studies using this outcome definition should be unbiased, assuming the sensitivity is non-differential. If sensitivity is low, using this approach to identify new diagnoses of acute bacterial sinusitis would result in an underestimation of absolute risks and risk differences. However, we believe new cases of acute bacterial sinusitis should be well captured using the proposed algorithm. Study subjects were included from outpatient practices affiliated with a large, academic healthcare system in Massachusetts, which may limit the generalizability of results to other care settings with different coding and clinical practices. While this study aimed to focus on identifying cases of acute bacterial sinusitis, a substantial portion of diagnoses of acute sinusitis are viral, and microbiologic studies to attempt to discern the causative organism are not routinely recommended, limiting the ability to identify only bacterial infections.^2,3 We restricted our study cohort to patients who received both an ICD-10 code for acute sinusitis and an antibiotic prescription in an attempt to increase the likelihood of identifying clinician-diagnosed bacterial infections, as this would be most relevant to clinical practice and make the approach straightforward for others to reproduce.

This study suggests that pediatric patients with new diagnoses of acute bacterial sinusitis can be accurately identified in claims databases by using selected ICD-10 codes and same-day antibiotic prescriptions. Future studies can use this validated approach to evaluate the quality of care as it relates to clinical practice guidelines for acute sinusitis (e.g., not ordering x-rays or CT scans), and to study the comparative safety and effectiveness of antibiotics used to treat acute sinusitis in children and adolescents.

Five key points:

• Large healthcare utilization databases have the potential to be well suited to studying acute bacterial sinusitis

• A validated algorithm to use claims to identify new cases of acute bacterial sinusitis is essential for conducting high quality research

• The accuracy of claims-based algorithms to identify new diagnoses of pediatric acute bacterial sinusitis has not been previously assessed

• In a cohort of 150 pediatric patients, an ICD-10 code for acute sinusitis and an antibiotic prescription had a PPV of 92% (95% CI 87%,95%)

• New diagnoses of acute bacterial sinusitis can be accurately identified in claims databases

Plain language summary:

Acute bacterial sinusitis is associated with high rates of antibiotic prescribing. Despite this, there have not been many studies characterizing antibiotic prescribing or comparing the effectiveness and safety of various antibiotics for acute bacterial sinusitis in children and adolescents. Large healthcare utilization databases, with diagnostic coding and prescription dispensation information, could be high-quality data sources to study these research areas. To ensure that cases are accurately identified, validation of ICD-10 diagnostic codes is necessary. In a cohort of 150 children and adolescents, new cases of acute bacterial sinusitis were able to be accurately identified when using an ICD-10 code for sinusitis paired with a same-day antibiotic prescription. This suggests that large healthcare utilization databases with diagnostic and medication information are well suited to study the treatment of acute bacterial sinusitis in children and adolescents.

Data sharing:

No additional data available