Identifying Pediatric Community-acquired Pneumonia Hospitalizations: Accuracy of Administrative Billing Codes

Abstract

Importance

Community-acquired pneumonia (CAP) remains one of the most common indications for pediatric hospitalization in the United States, and it is frequently the focus of research and quality studies. Use of administrative data is increasingly common for these purposes, although proper validation is required to ensure valid study conclusions.

Objective

To validate administrative billing data for childhood community-acquired pneumonia (CAP) hospitalizations.

Design

Case-control.

Setting

Four freestanding children’s hospitals in the United States.

Participants

Medical records of a 25% random sample of 3,646 children (n=998) discharged in 2010 with at least one ICD-9-CM code representing possible pneumonia were reviewed. Discharges (matched on date of admission) without a pneumonia-related discharge code were also reviewed to identify potential missed pneumonia cases. Two reference standards, based on provider diagnosis alone (provider-confirmed) or in combination with clinical and radiographic evidence of pneumonia (definite), were used to identify CAP.

Main Exposure

Twelve ICD-9-CM based coding strategies, each using a combination of primary or secondary codes representing pneumonia or pneumonia-related complications. Six algorithms excluded children with complex chronic conditions.

Main Outcome Measures

Sensitivity, specificity, negative and positive predictive values (NPV, PPV) of the twelve identification strategies.

Results

For provider-confirmed CAP (n=680), sensitivity ranged from 60.7–99.7%; specificity 75.7–96.4%; PPV 67.9–89.6%; and NPV 82.6–99.8%. For definite CAP (n=547), sensitivity ranged from 65.6–99.6%; specificity 68.7–93.0%; PPV 54.6–77.9%; and NPV 87.8–99.8%. Unrestricted use of the pneumonia-related codes was inaccurate, although several strategies improved specificity to >90% with variable impact on sensitivity. Excluding children with complex chronic conditions demonstrated the most favorable performance characteristics. Performance of the algorithms was similar across institutions.

Conclusions and Relevance

Administrative data are valuable for studying pediatric CAP hospitalizations. The strategies presented here will aid in the accurate identification of relevant and comparable patient populations for both research and performance improvement studies.

INTRODUCTION

Community-acquired pneumonia (CAP) is one of the most common causes of childhood mortality worldwide, especially among those under 5 years of age.¹ Although mortality is lower in developed countries, pediatric CAP is still associated with substantial morbidity and remains the most common indication for pediatric hospitalization outside the newborn period in the United States.² It is also one of the most costly with hospital costs alone approaching $1 billion annually. As a result, pediatric CAP is often the focus of epidemiologic studies and outcomes research and serves as an ideal target for quality benchmarking and improvement efforts.

Administrative data is often used for such assessments due to the convenience and ready access to large study populations without the substantially higher costs of prospective studies. However, studies using administrative data often rely on discharge diagnosis codes, such as the International Classification of Diseases, 9th Revision, Clinical Modification (ICD-9-CM) codes, to identify study populations, and the assignment of these codes may be influenced by reimbursement patterns and local coding practices, or subject to error.^3,4 In the case of pneumonia, there are many ICD-9-CM discharge codes which represent its many microbiological causes (e.g. influenza versus Streptococcus pneumoniae) and potential complications (e.g. empyema). Furthermore, a discharge code indicating pneumonia does not readily differentiate CAP from pneumonia that is the result of a hospital acquired infection, such as ventilator-associated pneumonia. As a result, use of administrative data to identify CAP hospitalizations without appropriate validation is not advisable due to the heterogeneous nature of the disease and the potential for misclassification and erroneous conclusions.^5,6,7,8

Studies in adults have assessed the validity of ICD-9-CM codes to identify pneumonia hospitalizations, although with varying results depending on the population studied and the codes selected for identification (PPV 57–96%; sensitivity, 48–98%).^9–13 The findings from these studies may be limited in their application since they relied on small sample sizes,^9–12 were conducted at a single site,^9,10,12 or in healthcare systems outside of the United States.^11–13 Most importantly, none of these studies included children, and thus the results may not be readily generalizable to pediatric populations. Consequently, the primary objective of our study was to assess the performance of a variety of ICD-9-CM coding strategies to identify pediatric CAP hospitalizations using data from four tertiary care, freestanding children’s hospitals in the United States.

PATIENTS AND METHODS

Design and Setting

For this multicenter retrospective study, the Pediatric Health Information System (PHIS); Children’s Hospital Association (CHA), (Overland Park, KS) was used to identify children from 4 freestanding pediatric hospitals (Monroe Carell, Jr. Children’s Hospital at Vanderbilt, Nashville, TN; Children’s Mercy Hospitals & Clinics, Kansas City, MO; Seattle Children’s Hospital, Seattle, WA; and Cincinnati Children’s Hospital Medical Center, Cincinnati, OH). The institutional review board at each hospital approved the study.

Study Population

Children 60 days–18 years of age that were discharged from one of the participating hospitals between January 1, 2010 and December 31, 2010, with at least one ICD-9-CM discharge diagnosis code (primary or any secondary) indicating pneumonia, pleural effusion, or empyema (n=5023) were considered for inclusion. [Figure 1.] Two investigators independently reviewed concomitant ICD-9-CM codes for each hospitalization (up to 21 codes), excluding patients with diagnoses that precluded CAP (e.g. cystic fibrosis or immunodeficiency) (n=1,377). A 25% random sample of the remaining 3,646 discharges was then selected for medical record review (n=998). To identify potential missed pneumonia cases, discharges without an ICD-9-CM pneumonia-related diagnosis code (matched by date of admission to account for seasonal trends in pneumonia admissions) were also selected for medical record review (n=1000).

Figure 1.

Study population

Data Sources

Two data sources were used for this study, the PHIS database and medical record review. The PHIS database contains clinical and billing data from 42 freestanding children’s hospitals and was used to identify all potential subjects and define patient demographics. Data quality are assured through a joint effort between CHA and participating hospitals, as described previously.¹⁴ Medical record data were extracted by trained investigators at each site using a central, web-based data collection system.¹⁵

Reference Standards and ICD-9-CM CAP Identification Strategies

Two reference standards were used to represent CAP, one highly sensitive (provider-diagnosis) and the other highly specific (provider-diagnosis plus consistent clinical and radiographic evidence).^16,17 We assessed each medical record for the presence of the following criteria: 1) provider diagnosis of pneumonia within the first 48 hours of hospitalization (mention of suspected CAP along with consistent management strategy); 2) abnormal temperature (≥ 38.0 C) or white blood cell count (<5,000 or >15,000 cells/mL); 3) evidence of respiratory illness (e.g. cough or increased work of breathing); and 4) chest radiograph indicating pneumonia (e.g. infiltrate or consolidation). Any child with a condition precluding CAP (e.g. cystic fibrosis or effusions following admission for cardiothoracic surgery) was not classified as CAP regardless of the above mentioned criteria. After applying this exclusion, any child with at least a provider diagnosis of pneumonia was considered provider-confirmed CAP, and any child with all four criteria was considered definite CAP.

For each reference standard, we assessed the performance of twelve ICD-9-CM coding algorithms to identify CAP hospitalizations among our study population. These strategies incorporate a number of previously described CAP algorithms as well as the classification of complex chronic conditions (CCCs) described by Feudtner et al.^{9,10,13,18–20} Strategies that included codes representative of pneumonia symptoms (e.g. cough) were also considered;¹⁰ however, these algorithms did not identify additional CAP cases and were excluded from further analyses.

Analysis

Characteristics of the study population were summarized using proportions for categorical variables and median and interquartile range for continuous variables. Characteristics of children classified as definite or provider-confirmed CAP were compared using chi-square tests for categorical variables and Wilcoxon rank-sum for continuous variables.

To determine the performance of each identification strategy for CAP, we calculated sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) for each reference standard.²¹ We randomly sampled from all discharges with a pneumonia-related discharge code; thus the PPV estimates are assumed to reflect the entire population under study (including disease prevalence within the population). To maximize the possibility of identifying false negatives (algorithm negative, reference standard positive), we selected a random sample of hospitalizations without a pneumonia-related discharge code matched on admission date to those hospitalizations with a pneumonia-related discharge code.

To describe inter-hospital variation in coding practices for CAP, the performance of each algorithm for provider-confirmed CAP were characterized for each hospital. To assess differences in patient characteristics and outcomes across identification strategies for children with provider-confirmed CAP, we summarized several patient level variables, including concurrent respiratory diagnoses, care utilization measures, and severe outcomes (intensive care admission, mechanical ventilation, or empyema).

RESULTS

Study Population

We reviewed records for 998 out of a possible 3,646 discharges with a pneumonia-related ICD-9-CM discharge code and 1000 discharges without a pneumonia-related discharge code. [Figure 1.] Among discharges with a pneumonia-related code, 677 (67.8%) were classified as provider-confirmed CAP; 545 (80.5%) were further classified as definite CAP. Children with definite CAP were less likely to have a concurrent discharge diagnosis of asthma but were more likely to have a parapneumonic effusion or empyema compared to those without definite CAP. [Table 1] Although hospital LOS was slightly longer, children with definite CAP were less likely to have a complex chronic condition or require admission to intensive care; Average hospital charges were also less. Of the 1,000 discharges without a pneumonia-related discharge code, three additional cases of provider-confirmed CAP, including 2 cases of definite CAP, were identified.

Table 1.

Characteristics of the Study Population with Community-acquired Pneumonia: Comparison of Two Reference Standards

Characteristics	Reference Standard			p-value1
	All Provider-confirmed CAP (n=680)	Definite CAP (n=547)	Provider-confirmed CAP only (n=133)
Age, years
<1	116 (17.1)	91 (16.6)	25 (18.8)	0.308
1–5	365 (53.7)	302 (55.2)	63 (47.4)
6–10	135 (19.9)	102 (18.6)	33 (24.8)
≥11	64 (9.4)	52 (9.5)	12 (9)
Male sex	369 (54.5)	301 (55.2)	68 (51.5)	0.442
Race
Non-Hispanic White	377 (55.7)	315 (57.8)	62 (47)	0.001
Non-Hispanic Black	117 (17.3)	78 (14.3)	39 (29.5)
Hispanic	65 (9.6)	55 (10.1)	10 (7.6)
Asian	19 (2.8)	17 (3.1)	2 (1.5)
Other	99 (14.6)	80 (14.7)	19 (14.4)
Insurance
Public	361 (53.3)	281 (51.6)	80 (60.6)	0.160
Private	315 (46.5)	263 (48.3)	52 (39.4)
Other	1 (0.1)	1 (0.2)	0 (0.0)
Complex comorbid condition2	62 (9.1)	41 (7.5)	21 (15.8)	0.003
Intensive care admission	75 (11.1)	51 (9.4)	24 (18)	0.004
Mechanical ventilation	28 (4.1)	24 (4.4)	4 (3)	0.468
Empyema	102 (15)	94 (17.2)	8 (6)	0.001
Concurrent diagnosis
Asthma	219 (32.2)	159 (29.1)	60 (45.1)	<.001
Bronchiolitis	53 (7.8)	46 (8.4)	7 (5.3)	0.225
Hospital length of stay, days	2 (1–4)	2 (2–4)	2 (1–3)	<.001
Hospital charges, $	11,755 (8,000–19,840)	11,645 (8,000–19,647)	12,700 (8,024–20,026)	<.001

Provider-confirmed CAP includes discharges with a provider diagnosis of pneumonia within 48 hours of hospitalization and no exclusionary conditions (n=680). Definite CAP includes discharges with a provider diagnosis of pneumonia within 48 hours of hospitalization, abnormal temperature or white blood cell count, evidence of respiratory illness, radiographic evidence of pneumonia, and no exclusionary conditions (n=547).

¹Statistical comparisons are displayed between those classified as provider-confirmed CAP only (n=133) and those classified as definite CAP (n=547).

²See reference 20.

Abbreviations: CAP, Community-acquired pneumonia.

Algorithm Performance, Provider-confirmed CAP

Using the reference standard of provider-confirmed CAP(n=680), sensitivity ranged from 60.7–99.7% (NPV 82.6–99.8%). Specificity ranged from 75.7–96.4% (PPV 67.9–89.6%). A pneumonia diagnosis code in any position (algorithm 1) identified nearly all children with provider-confirmed CAP (sensitivity 99.7%), although this strategy had the lowest specificity (75.7%). [eTable 1, Figure 2a] An identification strategy that only included children with a primary diagnosis of pneumonia (algorithm 2) improved specificity to nearly 90% but reduced sensitivity to 71%. Results were similar for the additional inclusion of children with a primary diagnosis of pneumonia and/or a primary diagnosis of a pneumonia-related complication plus a secondary diagnosis of pneumonia (algorithm 3). We also assessed the performance of algorithms 1–3 after excluding children with at least one CCC discharge code.²⁰ These algorithms (1b, 2b, 3b) improved specificity (89.7–95.7%) compared to the analogous algorithms that included CCCs, although at the expense of decreased sensitivity.

Figure 2.

Sensitivity and specificity of twelve ICD-9-CM code identification algorithms for community-acquired pneumonia (CAP) using two reference standards: (a) Provider-confirmed CAP (n=680); (b) Definite CAP (n=547). Horizontal and vertical bars represent calculated 95% confidence limits.¹ Algorithm definitions, 1: 1° or any 2° diagnosis of pneumonia or effusion/empyema (1b excludes complex chronic conditions [CCCs]—see reference 20); 2: 1° diagnosis of pneumonia or effusion/empyema (2b excludes CCCs); 3: 1° diagnosis of pneumonia or effusion/empyema OR 1° diagnosis of pneumonia-related complication PLUS any 2° diagnosis of pneumonia or effusion/empyema (3b excludes CCCs); 4: 1° or any 2° diagnosis of pneumonia (4b excludes CCCs); 5: 1° diagnosis of pneumonia (5b excludes CCCs); 6: 1° diagnosis of pneumonia OR 1° diagnosis of pneumonia-related complication or effusion/empyema PLUS any 2° diagnosis of pneumonia (6b excludes CCCs); ICD-9-CM codes used in the study are as follows, pneumonia: 480.0-2, 480.8-9, 481, 482.0, 482.30-2, 482.41-2, 482.83, 482.89-90, 483.8, 484.3, 485, 486, 487.0; effusion/empyema: 510.0, 510.9, 511.0-1, 511.8-9, 513; pneumonia-related complication: 38.9, 458.9, 518.81, 790.7, 799.1, 995.91-2, 997.3.

Since diagnosis codes indicating pleural effusion or empyema are not exclusive to CAP, we assessed the performance of identification strategies that included effusion or empyema codes only when coupled with more explicit pneumonia codes. [eTable 1, Figure 2a] Algorithms 4–6 demonstrated improved specificity with a minimal impact on sensitivity compared to each of the analogous algorithms 1–3. The most substantial improvement in specificity was noted for algorithm 4 (84.2%, 95% CI [82.2, 86.1]) compared to algorithm 1 (75.7%, 95% CI [73.3, 78]). Excluding discharges that had one or more CCC codes from algorithms 4–6 (4b, 5b, 6b) further improved specificity (91.5–96.1%).

Algorithm Performance, Definite CAP

Using the reference standard of definite CAP (n=547), sensitivity ranged from 65.6–99.6% (NPV 87.8–99.8%). Specificity ranged from 68.7–93% (PPV 54.6–77.9%). [eTable 1, Figure 2b] The performance characteristics of the algorithms relative to one another remained similar.

Hospital and Patient Level Characteristics

We also explored the performance of identification strategies by hospital. [eFigures 1a and 1b] We noted few differences when comparing the individual algorithms across hospitals. Small differences in sensitivity for several of the algorithms occurred at one institution; otherwise no differences were noted.

Finally, we described patient characteristics, including other respiratory diagnoses, complications, and outcomes for CAP, for each algorithm to assess for potential differences. [Table 2] Several differences were noted according to the algorithm selected, including the proportion of children with a concurrent diagnosis of asthma or bronchiolitis, and the proportion of children experiencing a severe outcome (intensive care admission, mechanical ventilation, or empyema).

Table 2.

Outcomes and Concurrent Diagnoses for Provider-Confirmed Community-acquired Pneumonia According to Identification Algorithm

Algorithm1	Asthma	Bronchiolitis	Intensive care admission	Mechanical ventilation	Empyema	Hospital length of stay, days	Hospital charges, $
1	212 (31.9)	53 (8)	71 (10.7)	28 (4.2)	102 (15.3)	2 (1–4)	11,662 (8,000–19,858)
1b	200 (33.1)	50 (8.3)	57 (9.4)	21 (3.5)	93 (15.4)	2 (1–3)	11,373 (7,900–18,990)
2	120 (25.1)	12 (2.5)	26 (5.4)	9 (1.9)	88 (18.4)	2 (1–3)	10,608 (7,648–17,169)
2b	111 (25.8)	11 (2.6)	19 (4.4)	6 (1.4)	81 (18.8)	2 (1–3)	10,246 (7,583–15,885)
3	122 (24.9)	13 (2.7)	35 (7.2)	18 (3.7)	91 (18.6)	2 (1–3)	10,784 (7,679–17,782)
3b	113 (25.9)	12 (2.7)	26 (5.9)	13 (3)	83 (19)	2 (1–3)	10,324 (7,612–16,346)
4	212 (32.3)	53 (8.1)	70 (10.7)	28 (4.3)	93 (14.2)	2 (1–4)	11,523 (7931–19551)
4b	200 (33.6)	50 (8.4)	56 (9.4)	21 (3.5)	85 (14.3)	2 (1–3)	11,217 (7,870–18,413)
5	118 (25.9)	12 (2.6)	23 (5)	8 (1.8)	65 (14.3)	2 (1–3)	10,247 (7,538–15,713)
5b	109 (26.7)	11 (2.7)	16 (3.9)	5 (1.2)	60 (14.7)	2 (1–3)	10,073 (7522–15,081)
6	122 (25.3)	13 (2.7)	35 (7.3)	18 (3.7)	84 (17.4)	2 (1–3)	10,611 (7,653–17,338)
6b	113 (26.2)	12 (2.8)	26 (6)	13 (3)	77 (17.9)	2 (1–3)	10,276 (7,583–16,100)

Data presented as No. (%) and median (interquartile range).

¹Algorithm definitions, 1: 1° or any 2° diagnosis of pneumonia or effusion/empyema (1b excludes complex chronic conditions [CCCs]—see reference 20); 2: 1° diagnosis of pneumonia or effusion/empyema (2b excludes CCCs); 3: 1° diagnosis of pneumonia or effusion/empyema OR 1° diagnosis of pneumonia-related complication PLUS any 2° diagnosis of pneumonia or effusion/empyema (3b excludes CCCs); 4: 1° or any 2° diagnosis of pneumonia (4b excludes CCCs); 5: 1° diagnosis of pneumonia (5b excludes CCCs); 6: 1° diagnosis of pneumonia OR 1° diagnosis of pneumonia-related complication or effusion/empyema PLUS any 2° diagnosis of pneumonia (6b excludes CCCs); ICD-9-CM codes used in the study are as follows, pneumonia: 480.0-2, 480.8-9, 481, 482.0, 482.30-2, 482.41-2, 482.83, 482.89-90, 483.8, 484.3, 485, 486, 487.0; effusion/empyema: 510.0, 510.9, 511.0-1, 511.8-9, 513; pneumonia-related complication: 38.9, 458.9, 518.81, 790.7, 799.1, 995.91-2, 997.3.

DISCUSSION

This study determined the performance of twelve ICD-9-CM coding algorithms in identifying pediatric CAP hospitalizations at four tertiary care pediatric hospitals in the United States. Unrestricted application of the pneumonia-related codes does not accurately identify CAP in children, although several coding strategies offer improvements in specificity while also retaining good sensitivity. Administrative data can be used to study pediatric CAP hospitalizations, although the study population must be carefully defined and the specificity of the diagnosis understood in order to ensure valid study conclusions.

Use of the pneumonia and effusion or empyema codes in any position (algorithm 1) was not accurate, although this strategy identified nearly all children with CAP. This is an important attribute for studies aimed at identifying all episodes of CAP, such as assessments of disease frequency or monitoring for adverse drug events. To avoid misclassification, application of this algorithm requires a multi-step identification strategy, such as ICD-9-CM based screening with medical record review to confirm the diagnosis.

In contrast to algorithms 1–3 which allowed the effusion or empyema codes to represent pneumonia without the requirement of an additional pneumonia code, algorithms 4–6 required a more explicit pneumonia code and only algorithm 6 including the effusion or empyema codes (as a pneumonia-related complication). This resulted in an 11% increase in specificity for pneumonia codes in any position (algorithm 4) and identified nearly all CAP cases. Predictive performance was further improved when restricting the population to those with a primary diagnosis of pneumonia (algorithm 5) or primary diagnosis of a pneumonia-related complication plus a secondary diagnosis of pneumonia (algorithm 6), similar to validation studies in other populations.^22,23 However, these algorithms only identified approximately 70% of provider-confirmed CAP and specificity was only marginally improved over the analogous algorithms 2 and 3. Nonetheless, these identification strategies may be a better choice for assessments that seek to maximize specificity over sensitivity and are unable to include confirmation using medical records. For example, these algorithms could be used to improve the validity of outcomes data in a study comparing the effectiveness of antimicrobial treatment choices for CAP.

The algorithms with the best performance characteristics (i.e. both highly sensitive and specific) were 1b and 4b. These identification strategies, which do not restrict use of the pneumonia codes, excluded children with CCCs identified using a previously reported classification scheme.²⁰ The sensitivity and specificity of both algorithms was approximately 90% for provider-confirmed CAP. Overall, application of the CCC restriction yielded meaningful improvements in specificity for all of the algorithms studied without large reductions in sensitivity. Likely, restriction of these codes excludes children with a high probability of complicating factors which may alter the true risk of CAP (e.g. frequent hospitalizations or technology-dependence) while minimally affecting children without these risk factors. As a result, these strategies are ideal for identifying and studying pediatric CAP hospitalizations, especially if the presence of complex comorbid conditions is thought to confound hypothesized relationships.

Requiring the provider diagnosis of pneumonia to be coupled with objective clinical evidence of a lower respiratory tract infection with radiographic confirmation (i.e. definite CAP) resulted in expected reductions in specificity for all of the algorithms studied. Several differences were noted between those classified as definite CAP and those with provider-confirmed CAP only, underscoring the clinical challenge of diagnosing CAP in children. For instance, nearly half of those with provider-confirmed CAP who were not also classified as definite CAP had a concurrent discharge diagnosis of asthma. Although asthma is a risk factor for CAP, in the clinical setting it is often difficult to distinguish a superimposed pneumonia from atelectasis in the acutely wheezing asthmatic child when a chest radiograph reveals an opacity.^24,25 In contrast, the presence of a parapneumonic effusion in a child with presumed pneumonia offers additional certainty of the CAP diagnosis. Indeed, the presence of effusion or empyema was more common among those diagnosed with definite CAP. Thus, the clinical uncertainty inherent in diagnosing CAP is reflected in the differences in predictive performance for the two reference standards presented here. This helps to more objectively assess the degree of uncertainty likely to exist in a population of children with CAP identified using administrative data. To this end, it is important to note that 80% of the children with provider-confirmed CAP were also classified as definite CAP. Coding variation across institutions may limit the validity of administrative data.²⁶ In our study, specificity across all four institutions and sensitivity at three of the four institutions studied were similar. Sensitivity was lower for several algorithms at one institution, although differences were small and likely reflect local variation in coding practices. Nonetheless, our findings demonstrate that coding practices for pediatric CAP are generally consistent across hospitals, suggesting that these identification strategies can be used to accurately benchmark performance and compare patients across institutions.^27,28

We also observed that identification strategies which considered either a primary or secondary diagnosis of pneumonia (algorithms 1/1b and 4/4b) identified a higher proportion of children with a concurrent diagnosis of asthma or bronchiolitis. In contrast, algorithms restricted to a primary diagnosis of pneumonia (algorithms 5/5b) included fewer children with these diagnoses, but also identified the lowest proportion of children requiring ICU admission or mechanical ventilation. This is an important consideration when designing studies to examine severe pneumonia outcomes.^10,29 Moreover, strategies which considered a secondary diagnosis of pneumonia only if it was coupled with a primary diagnosis of a pneumonia-related complication (algorithms 6/6b) identified a high proportion of children with severe outcomes while minimizing the proportion of children with concurrent diagnoses of asthma or bronchiolitis. This suggests that these latter algorithms may be particularly relevant for identifying severe CAP while also enriching the study population for bacterial pneumonia.

Strengths of our study include the multicenter design and large sample of children, the use of standardized data collection methods and CAP reference standards, a wide range of CAP identification algorithms, and the exclusive focus on pediatric patients hospitalized with CAP, a population that has not been previously validated using administrative data. There are also several limitations. Discharges with and without a pneumonia-related discharge code were reviewed separately, although reviewers were blinded to individual billing codes. A comparison of rater agreement was also not performed. There is no universally accepted gold standard for defining CAP and thus, as with all studies of pneumonia, there is at least some level of uncertainty in classifying the disease.³⁰ We attempted to minimize misclassification by using two reference standards which rely on both provider assessment and objectively measured criteria, investigator training and piloting of data collection procedures, and use of a common, web-based data collection system. With over 40,000 discharges in 2010 among the four study hospitals, it was not feasible to review all hospitalizations during the study period. However, we selected a random sample of discharges using a wide range of possible pneumonia-related diagnosis codes and also reviewed a similar number of discharges without a pneumonia-related diagnosis code. Our matching strategy also maximizes the possibility of identifying missed pneumonia cases, resulting in an accurate estimation of PPV and a most conservative estimate of NPV. However, it is likely that this strategy overestimates sensitivity and underestimates specificity. As an example, applying the PPV and NPV estimates from algorithm 1 for provider-confirmed CAP to the entire study population (using actual numbers of algorithm positive and total hospital discharges in 2010) results in an estimated sensitivity of 96.1% (reported 99.7%) and specificity of 97.7% (reported 75.7%). Although this exercise is an oversimplification, the true sensitivity and specificity can be expected to fall within this range. Finally, since this study was conducted at tertiary care children’s hospitals, our results may not be generalizable to community hospitals. Nevertheless, CHA affiliated hospitals account for nearly 20% of pediatric hospitalizations each year, making PHIS one of the largest and most frequently used administrative data sources for quality and research studies of pediatric hospitalizations in the United States. Moreover, given the high degree of agreement between hospitals, it is likely that our results would be applicable to other pediatric administrative datasets.

CONCLUSION

This is the first study to validate ICD-9-CM discharge diagnosis codes for children hospitalized with CAP. We have demonstrated that administrative data is a valuable tool for studying pediatric CAP and provide several strategies that reliably identify this population. Application of these validated algorithms allows for more accurate identification of relevant and comparable patient populations for both research and performance improvement purposes. Understanding the strengths and limitations of these data, as well as the uncertainty occasionally associated with diagnosing CAP, will help ensure valid study conclusions.