Accuracy of Administrative Billing Codes to Detect Urinary Tract Infection Hospitalizations

Abstract

BACKGROUND:

Hospital billing data are frequently used for quality measures and research, but the accuracy of the use of discharge codes to identify urinary tract infections (UTIs) is unknown.

OBJECTIVE:

To determine the accuracy of International Classification of Diseases, 9th revision (ICD-9) discharge codes to identify children hospitalized with UTIs.

METHODS:

This multicenter study conducted in 5 children's hospitals included children aged 3 days to 18 years who had been admitted to the hospital, undergone a urinalysis or urine culture, and discharged from the hospital. Data were obtained from the pediatric health information system database and medical record review. With the use of 2 gold-standard methods, the positive predictive value (PPV) was calculated for individual and combined UTI codes and for common UTI identification strategies. PPV was measured for all groupings for which the UTI code was the principal discharge diagnosis.

RESULTS:

There were 833 patients in the study. The PPV was 50.3% with the use of the gold standard of laboratory-confirmed UTIs but increased to 85% with provider confirmation. Restriction of the study cohort to patients with a principle diagnosis of UTI improved the PPV for laboratory-confirmed UTI (61.2%) and provider-confirmed UTI (93.2%), as well as the ability to benchmark performance. Other common identification strategies did not markedly affect the PPV.

CONCLUSIONS:

ICD-9 codes can be used to identify patients with UTIs but are most accurate when UTI is the principal discharge diagnosis. The identification strategies reported in this study can be used to improve the accuracy and applicability of benchmarking measures.

WHAT'S KNOWN ON THIS SUBJECT:

Hospital billing data are frequently used for quality measures and research, but the accuracy of identification of urinary tract infections on the basis of International Classification of Diseases, 9th revision, discharge codes is unknown.

WHAT THIS STUDY ADDS:

The accuracy of the International Classification of Diseases, 9th revision, discharge codes as a basis for identification of children hospitalized with a urinary tract infections was assessed. The results can be used by investigators to identify study patients and monitor their outcomes.

Urinary tract infections (UTIs) are among the most common reasons for hospitalization of children.¹ Unwarranted variability in the management, treatment, and outcomes of children hospitalized with UTI (eg, hospitalization rates, hospital-acquired UTIs, use of imaging) contributes to unnecessary risks for patients and costs for the health care system.^2–6 Validated health care measures that identify hospital or provider factors that contribute to variability or underperformance can improve the quality of care for hospitalized children.

Many organizations such as the Agency for Healthcare Research and Quality (AHRQ) have programs to develop meaningful, reliable, and actionable metrics.^3,4 To drive quality improvement efforts through benchmarking hospital performance, however, these metrics must accurately identify the patient population (eg, case ascertainment) in which the care improvement efforts are to be implemented. Definition of a patient population for comparison can be especially challenging for hospitals that provide care for populations with disparate levels of comorbidities and medical complexity.

Researchers, hospitals, federal agencies, and payers frequently use the International Classification of Diseases, Ninth Revision (ICD-9) coding system for benchmarking physician and hospital performance, although this system was designed to capture billing information.^5,6 In many studies of common inpatient conditions, investigators have revealed variability across hospital systems, which has raised concern for quality of care, but these conclusions were based on ICD-9 case definitions.^1,6,7,8 One of the major controversies in medical outcomes research is whether the codes accurately represent the clinical condition from the provider or laboratory perspective or if observed variability is attributable to inconsistent coding practices. In fact, several ICD-9 codes have been shown to be highly accurate, whereas others fail to reliably capture the disease, differences that demonstrate the need to assess the accuracy of ICD-9 codes for a specific illness before embarkation on any project that relies on these codes to identify a cohort of patients.^9–12

The objectives of this study were to determine: (1) the accuracy of individual and combined ICD-9 discharge diagnosis codes for children with UTI confirmed by laboratory tests results; (2) whether previously reported identification strategies do indeed improve the identification of patients with UTI^3,6; (3) how hospital performance (based on length of stay [LOS]) varied with different UTI identification strategies; and (4) how hospital performance rank may change depending on which UTI codes were used for case ascertainment.

METHODS

Design and Setting

For this multicenter, retrospective study we used the pediatric hospital information system (PHIS) to identify patients from 5 freestanding pediatric hospitals (Seattle Children's Hospital, Seattle, WA; Children's Hospital at Vanderbilt, Nashville, TN; Cincinnati Children's Hospital Medical Center, Children's Mercy Hospital, Kansas City, MO; Children's Hospital of Philadelphia, Philadelphia, PA). The institutional review board of each hospital approved the study with a waiver of informed consent.

Subjects

Patients identified from the PHIS database were aged 3 days to 18 years, discharged from 1 of the participating hospitals from July 1, 2008, to June 30, 2009, and had a charge for a urinalysis (UA) or urine culture. We based our sample size on inference regarding the positive predictive value for the principal discharge diagnosis of UTI.¹³ If we assumed the sensitivity and specificity of the code is 95%, and desire to achieve a lower 99% confidence boundary for positive predictive value that exceeds a limit of 95% with 95% power, we needed 834 patients with a 2:1 allocation. Therefore, of 2729 patients with a charge for a UA and urine culture, we initially included 864 patients; 544 randomly sampled patients with an ICD-9 discharge diagnosis code of UTI or pyelonephritis (590.1, 590.2, 590.8, 599.0, and 771.82) and 320 matched patients with a code for UA or urine culture but no discharge diagnosis code for UTI. The patients were matched on age, gender, LOS, and hospital to ensure a robust sensitivity estimate. Patients were excluded if they were admitted to an inpatient psychiatric or rehabilitation unit (n = 31; 18 with a UTI code and 13 without a UTI code).

To determine the discharge ICD-9 codes for UTI, we searched for the terms “urinary tract infection” and “pyelonephritis” as indexed in the ICD-9-CM, 5th Edition.⁷ These codes were compared with the AHRQ PDI 18 definition and a report of a study by Conway et al.³ Codes for cystitis (595.0 and 595.9; included in the AHRQ PDI but not in Conway et al) were evaluated and excluded because they were not frequently used as a principal diagnosis (n = 16).

Data Sources

There were 2 data sources, the PHIS database and medical record review. The PHIS database contained clinical and billing data from 42 freestanding children's hospitals. Data quality and coding reliability are assured through a joint effort between the Child Health Corporation of America (Shawnee Mission, KS) and participating hospitals, as described previously.^14,15 The PHIS database was used to identify all participants, patient demographics, and utilization of hospital resources.

Data from medical records, reviewed by investigators blinded to the assigned ICD-9 codes, were entered into a Web-based data collection system. To ensure consistency, all investigators responsible for chart abstraction underwent training. In addition, we performed 2 pilot medical record reviews and held group discussions of results to ensure a common understanding of questions, preselected answers, and interpretation of chart data. Data collected included age, gender, presenting symptoms, reason for admission, prophylactic antibiotic treatment, fever pattern, presence of emesis during the first 14 days of admission, and history of previous UTI or anatomic abnormality. Results for laboratory studies (eg, UA and complete blood count), blood and urine culture, antibiotic sensitivity, and renal ultrasound were also abstracted.

Data Analysis

We defined 2 gold standards for UTI, laboratory confirmed and provider confirmed. A laboratory-confirmed UTI was defined as either a positive urine culture result of >100 000 colony forming units (CFUs) or a urine culture with >50 000 CFUs and an abnormal UA.¹⁸ A provider-confirmed UTI was defined as provider documentation (progress note, discharge summary, discharge orders) and treatment plan for a UTI at the time of discharge with and without laboratory confirmation of UTI. Both gold standards were used to determine the sensitivity, specificity, negative predictive value (NPV), and positive predictive value (PPV) for all ICD-9 codes, combined with associated exact 95% confidence intervals (CIs). Because PPV is the proportion of patients correctly identified, it was used for both gold standards to compare accuracy of individual ICD-9 discharge codes, combined ICD-9 codes, and for the following commonly used UTI identification strategies: (1) timing of UA or urine culture within 48 hours of admission; (2) exclusion of comorbid complex medical conditions (eg, immunosuppression, cancer, paralysis, renal transplantation, trauma, cardiac and respiratory disease); (3) restriction of population to ages with the highest likelihood of admission for a UTI (>59 days and <12 years); (4) AHRQ quality indicator for UTI; and (5) a previously published case definition from a study in which administrative data were used.^3,7 The denominator in each instance was the number of patients identified by a specific set of ICD-9 codes, and the numerator was the number of patients from that denominator with UTI. The PPV was then measured for all groupings for which the UTI ICD-9 code was listed as the principal discharge diagnosis, rather than any of the secondary diagnoses.

To illustrate the impact of different UTI identification strategies on quality benchmarking measures, we computed each PHIS hospital's average LOS (ALOS) with 95% CIs for cases identified. Hospitals with CIs above the overall mean were considered statistically high outliers, and those with CIs below the overall mean were considered statistically low outliers. Hospitals were considered inliers if their 95% CI included the overall mean. Across 2 identification strategies, we compared the hospitals' ALOS by using a signed-rank test and the hospital-to-hospital variation in the ALOS by using Bartlett's test. To illustrate the impact of coding accuracy on benchmarking for individual and combined ICD-9 codes, we reported the change in hospital rank based on median LOS after adjusting for race, gender, and insurance. The LOS was risk adjusted by using linear mixed-effects models according to demographic characteristics (age, gender, race, and payer) and the APR-DRG (All-Patient Refined–Diagnosis-Related Group) severity of illness index.

Comparisons were made across hospitals by using the χ² test for categorical variables and the Kruskal-Wallis tests for continuous variables. All analyses were clustered according to hospital. Analyses were performed by using SAS version 9.2 (SAS Institute, Cary, NC). A 2-tailed P < .05 was considered statistically significant.

RESULTS

Overall and Hospital Level Demographics

From the 5 study hospitals there were 3063 patients with a UTI ICD-9 discharge code; 2729 patients qualified for a chart review because they were discharged with a UA or urine culture. A total of 864 charts (526 patients with a UTI discharge code and 307 matched patients without a UTI discharge code) were randomly selected for review after exclusion of 31 charts for patients hospitalized in rehabilitation or psychiatric units (Table 1). There were 265 true UTIs based on laboratory confirmation and 447 based on provider confirmation. There were no significant differences between patients with a discharge code of UTI and their matches with respect to insurance type, mortality, or median adjusted charges (Table 1). Male gender varied across the hospitals, as did patient age (Table 2). Median LOS and hospitalization charges varied across the hospitals, with higher costs being associated with longer stays (Table 2). There were few false negatives in the study population for both gold standards; 4 matches with a laboratory-confirmed UTI (NPV: 98.7%) and 5 with a provider-confirmed UTI (NPV: 98.4%).

TABLE 1.

Description of the Study Population

	Study Population With ICD-9 code for UA or Urine Culture	No Discharge Diagnosis of UTI	Discharge Diagnosis of UTI	P
n	833	307 (36.9)	526 (63.1)
Male	248 (29.8)	90 (29.3)	158 (30)	.826
Medicaid	362 (43.5)	137 (44.6)	225 (42.8)	.603
Race
Non-Hispanic White	517 (63.1)	184 (61.1)	333 (64.3)	.171
Non-Hispanic Black	125 (15.3)	58 (19.3)	67 (12.9)
Hispanic	70 (8.5)	22 (7.3)	48 (9.3)
Asian	28 (3.4)	10 (3.3)	18 (3.5)
Other	79 (9.6)	27 (9)	52 (10)
Disposition
Home	769 (92.3)	287 (93.5)	482 (91.6)	.609
Died	14 (1.7)	4 (1.3)	10 (1.9)
Other	50 (6)	16 (5.2)	34 (6.5)
Median Age, y	3.4 (0.4–11.0)	3.7 (0.4–11.6)	3.2 (0.3–10.3)	.699
Median LOS, d	3 (2–9)	4 (2–9)	3 (2–9)	.822
Median charges, $a	20 083 (10 759–53 681)	23 120 (11 199–68 794)	18 632 (10 510–49 107)	.084

Values are listed as n (%) or median (interquartile range).

^aAdjusted for Center for Medicare and Medicaid Services Wage Index.

TABLE 2.

Description of Study Population and Outcomes According to Hospital

Study Populationa	Total	Hospital 1	Hospital 2	Hospital 3	Hospital 4	Hospital 5
n	833	164 (19.7)	169 (20.3)	173 (20.8)	156 (18.7)	171 (20.5)
Male	248 (29.8)	62 (37.8)	31 (18.3)	66 (38.2)	39 (25.0)	50 (29.2)
Median age, y	3.4 (0.4, 11)	2.9 (0.4, 9.6)	4.6 (0.6, 10.6)	4.7 (0.5, 12.2)	3.4 (0.2, 11.2)	1.7 (0.2, 8.9)
Medicaid	362 (43.5)	72 (43.9)	86 (50.9)	87 (50.3)	19 (12.2)	98 (57.3)
Outcomesa
Median LOS, d	3 (2–9)	3 (2–9.5)	3 (2–6)	5 (3–14)	4 (2–6)	3 (2–9)
Median adjusted total charges, $b	20 083 (10 759–53 681)	19 559 (10 276–50 815)	18 256 (9707–38 776)	27 454 (13 248–123 073)	20 209 (10 639–43 315)	17 377 (10 518–62 143)

Values are listed as n (%) or median (interquartile range).

^aP < .05 for across hospital comparison according to Kruskal-Wallis test.

^bAdjusted for Center for Medicare and Medicaid Services Wage Index.

Individual ICD-9 Codes

Individual ICD-9 codes that indicated a diagnosis of UTI were evaluated to determine their PPV for use as any of the secondary discharge diagnoses compared with their use as a principal discharge diagnosis. In general, the accuracy was better for provider-confirmed diagnosis compared with laboratory-confirmed diagnosis alone and best when the discharge codes were used for the principal diagnosis.

The discharge diagnoses “acute pyelonephritis” and “other pyelonephritis” were the most accurate individual ICD-9 codes. When listed as the principal diagnosis, the PPV for acute pyelonephritis was 100% (95% CI: 83.1–100.0) for provider-confirmed UTI and 65.0% (95% CI: 40.7–84.6) for laboratory-confirmed UTI (Table 3); PPVs werev only minimally lower when “acute pyelonephritis” was listed as a secondary diagnosis. In comparison, the PPV for the discharge diagnosis codes for “UTI, site not specified” and “UTI of the newborn” were not as good but still relatively accurate, especially for provider-confirmed UTI when listed either as the principal diagnosis (90.8% [95% CI: 84.0–95.3] and 90.5% [95% CI: 69.6–98.8], respectively) or as a secondary diagnosis (80.7% [95% CI: 76.0–84.9] and 89.2% [95% CI: 74.5–97.0], respectively) (Table 3). Individual ICD-9 codes for “renal or perinephric abscess,” were the least accurate; the PPV for provider-confirmed was 66.7% (95% CI: 34.8–90.1) and for laboratory-confirmed was 25.0% (95% CI: 5.5–57.2) (Table 3).

TABLE 3.

Positive Predictive Values According to Algorithms and Principal Diagnosis for Patients Aged 3 Days to 18 Years From July 2008 to June 2009

Description	Age	ICD-9 code	Laboratory-Confirmed PPV Result		Provider-Confirmed PPV Result
			Any Diagnosis	Principal Diagnosis	Any Diagnosis	Principal Diagnosis
Individual ICD-9 codes	3 d to 18 y
Acute pyelonephritis		590.1	62.5	65.0	95.8	100.0
Renal and perinephric abscess		590.2	25.0	16.7	66.7	66.7
Other pyelonnephritis not specified as acute/chronic		590.8	58.9	59.4	95.2	96.9
UTI, site not specified		599.0	47.4	66.4	80.7	90.8
UTI of newborn		771.82	40.5	47.6	89.2	90.5
Combined ICD-9 codes	3 d to 18 y
		590.1, 590.2, 590.8	58.8	58.5	93.5	95.9
		590.1, 590.2, 590.8, 599.0, 771.82	50.3	61.2	85.0	93.2
Identification algorithm
UA or urine culture done within 48 h of admission	3 d to 18 y	590, 599.0, 771.83, Lab or Procedure code for UA or Urine Culture	49.3	59.9	85.0	93.1
IV or oral antibiotic within 48 h of admission	3 d to 18 y	590, 599.0, 771.82, Medication Code for Antibiotic	51.0	61.5	85.3	93.1
Combined ICD-9 codes without comorbidities	3 d to 18 y	590, 599.0, 771.83	54.0	61.6	87.7	93.8
Combined ICD-9 codes with age restrictions	60 d to 12 y	590, 599.0, 771.83	49.8	61.9	85.7	92.9
AHRQ pediatric quality indicator inpatient UTI4	90 d to 17 y	590.10 590.11 590.2590.80 590.81 590.9595.0595.9599.0	NA	59.4	NA	92.2
Published definition3	30 d to 12 y	590.10, 590.11, 590.80, 599.0, “first UTI”	58.2	NA	87.9	NA

Any Diagnosis, ICD-9 code for UTI was either the principal or 1 of any secondary discharge diagnoses; Principal Diagnosis, at least 1 ICD-9 code for UTI was listed as the principal discharge diagnosis.

Combined ICD-9 Codes

When we combined ICD-9 diagnosis codes for pyelonephritis (590.1 ×, 590.2 ×) and renal or perinephric abscess (590.8 ×), the PPV was >90% for provider-confirmed UTI, regardless of whether UTI was listed as the principal or a secondary diagnosis. In contrast, the PPV for laboratory-confirmed UTI was only 58% for both principal and secondary diagnoses. Combining all of the individual ICD-9 diagnosis codes for UTI resulted in high PPV for provider-confirmed infection in the setting of secondary diagnosis (PPV: 85.0%; sensitivity: 99.1%; specificity: 80.9%) and principal diagnosis (PPV: 93.2%; sensitivity: 99.6%; specificity: 90.6%), but the PPV for laboratory-confirmed diagnosis remained low (PPV: 50.3%; sensitivity: 98.9%; specificity: 56.1%) and only marginally increased in the setting of UTI as the principal diagnosis (PPV: 61.2%; sensitivity: 99.4%; specificity: 62.9%) (Table 3).

Identification Algorithms

To evaluate whether 4 commonly used identification strategies enhanced the ability to identify a UTI, we calculated the PPV for patients who had at least 1 of the UTI ICD-9 discharge codes and who (1) had a UA or urine culture within 48 hours of admission, (2) received intravenous antibiotics within the 48 hours of admission, (3) did not have comorbid conditions, or (4) were aged between 60 days and12 years (Table 3). The use of these identification strategies did not improve the PPV compared with that for the combined ICD-9 codes alone.

We also tested the PPV for the AHRQ area-level quality indicator (“PDI 18”) for inpatient UTI, which restricts the age to 90 days to 17 years and the discharge diagnosis code of UTI to the principal diagnosis.⁶ The AHRQ identification strategy was accurate but no better than the PPV for (Table 3).

Lastly, we used ICD-9–based inclusion and exclusion criteria from a recent publication in which outcomes were reported (LOS, admission rates, and imaging) for patients with “first-time UTI.” Again, the PPV was similar to that for both laboratory-confirmed and provider-confirmed UTI (Table 3).

Effect on Quality Measurement

Figure 1 displays the change in the ALOS according to hospital when UTI is used as a principal diagnosis compared with any 1 of the secondary diagnoses. When UTI was the principal discharge diagnosis, ALOS was significantly lower in each hospital (P < .001) than when UTI was listed as a secondary diagnosis, and variation across hospitals in ALOS was also reduced significantly (P < .001). In addition, only 24 (55.8%) of hospitals had the same result under both strategies (ie, low outlier, high outlier, or inlier). Furthermore, 4 (9.3%) had opposite results (ie, high outlier with 1 approach and low outliers with the other approach), 9 (21.4%) were outliers as any diagnosis and became inliers with principal diagnosis, and 6 (13.9%) were inliers in the setting of any diagnosis and became outliers with principal diagnosis. Figure 2 shows a change in nearly every hospitals' rank (according to median LOS) for each UTI ICD-9 code used. Hospital ranks also changed substantially when individual ICD-9 codes were considered (Fig 2).

FIGURE 1.

Effect of principal diagnosis on hospital performance according to length of stay. The plot shows each hospital's mean length of stay with 95% CI for urinary tract infection listed as the principal diagnosis (dark circle) or as any diagnosis (open circle).

FIGURE 2.

Change in hospital rank for individual and combined urinary tract infection discharge diagnosis codes as principal discharge diagnosis according to adjusted median LOS. The reference for comparison was the hospital rank when any of the discharge diagnosis codes were listed as the principal diagnosis. Values were adjusted for age, gender, race, and insurance type. The ICD-9 discharge diagnosis codes were identified as follows: acute pyelonephritis, 590.1; renal and perinephric abscess, 590.2; other pyelonephritis or pyonephrosis, not specified as acute or chronic, 590.8; UTI, site not specified, 599.0; and UTI of the newborn, 771.82.

DISCUSSION

The results of this multicenter study support the use of administrative data to identify children hospitalized with a provider-confirmed UTI; however, they also demonstrate the need to assess the accuracy of individual codes and identification algorithms before using them to compare hospital performance. Overall, the PPV for provider-confirmed UTIs was best when UTI was listed as the principal discharge code; accuracy was reduced with the inclusion of secondary discharge codes. Despite a high NPV, the PPV for laboratory-confirmed UTIs was poor, regardless of whether discharge diagnosis codes were listed as a primary or secondary diagnosis. Finally, incorporating additional data elements from UTI identification algorithms did not substantially alter the PPV for provider or laboratory-confirmed UTIs.

This study has several implications. First, we demonstrated that accurate identification of patients is a critical step to developing a quality measure. Even with few false negatives, the inclusion of patients without a condition under study (ie, false positives) can unpredictably affect hospital-level outcomes (eg, LOS). This level of misclassification can lead to erroneous conclusions about performance. Researchers and quality improvement specialists can use the predictive values from this study to best define their patient population, and thus improve the reliability and applicability of their outcomes data. For example, a community hospital wishing to improve care for “first-time UTI” should consider limiting the patient population to those patients with select principal UTI diagnosis codes (eg, pyelonephritis with PPV 100%), whereas hospitals conducting a project focused on decreasing nosocomial UTIs, for which UTI is unlikely to be the principal diagnosis, will require additional refinement of the population definition to improve the predictive value and subsequent applicability of the outcome measure.

ICD-9 code–based identification algorithms that include additional elements, such as age restrictions, are often used to increase the predictive value to best define patient populations involved in a care process. Yet, application of such identification algorithms did not substantially alter the PPV. In addition, applying previously published identification strategies, including the AHRQ pediatric quality indicator for UTI admission rate in children as well as that described by Conway et al yielded similar results.^3,7 Limiting the population to children with a principal diagnosis of UTI, however, greatly improved the validity of the case definition, regardless of the additional criteria included. Thus, other than considering only principal diagnosis in the identification of UTI hospitalizations, the use of increasingly complex algorithms does not necessarily result in improved case validity, and may actually reduce generalizability. The observed effect of identification strategies on LOS illustrates the importance of accurately definition the target population before the establishment of a benchmark for best practice.

This study had several limitations. First, this study did not capture patients who may have had a UTI but were not assigned a charge code for a UA or urine culture (ie, false negative). However, inclusion of every patient with a UTI was not critical to the quality of the measurement and monitoring of UTIs because we have no reason to suspect that such exclusions occurred systematically. Accordingly, in this study we used the PPV (the proportion of patients with a UTI that is correctly diagnosed), rather than sensitivity, to compare accuracy between different identification strategies.¹⁵ Also, for study inclusion children were required to have a UA or urine culture performed at the study hospital. This criterion led to exclusion of patients transferred from an outside institution and those whose urine testing was performed in an ambulatory-care setting, factors that should be considered when these results are applied to databases that do not include procedure or laboratory testing codes. A second limitation was that our findings are based on administrative data from 5 children's hospitals that contribute data to the PHIS database. As a result, it is not clear whether these results can be applied to other administrative data sets or to non–children's hospitals. Third, the reliability of each site's data abstractors was not assessed; however, abstraction procedures were reviewed and tested at each site and discussed in a collaborative fashion before study implementation. Finally, there was substantial discrepancy between the laboratory and provider-confirmed gold standards, with provider-confirmed UTIs resulting in significantly improved PPV. Although it is logical that provider documentation would enhance detection of UTIs, especially in cases in which UTI laboratory studies were performed at another facility, some providers still may overdiagnose UTIs by not adhering to the national standards for laboratory diagnosis. The degree to which underreporting of urine studies lowered the PPV of the laboratory-confirmed gold standard is unknown, although it is likely small because it would be limited to patients with a repeat UA or urine culture after transfer.