The impact of measurement changes on evaluating hospital performance: The case of catheter-associated urinary tract infections

Abstract

Catheter-associated urinary tract infections in 592 hospitals immediately declined after federal value-based incentive program implementation, but this was fully attributable to a concurrent surveillance case definition revision. Post revision, more hospitals had favorable standardized infection ratios, likely leading to artificial inflation of their performance scores unrelated to changes in patient safety.

Centers for Medicare and Medicaid Services (CMS) value-based incentive programs link financial incentives or penalties to quality metrics. Hospital Value-Based Purchasing (HVBP) and the Hospital-Acquired Conditions Reduction Program (HACRP) are CMS programs that incorporate National Healthcare Safety Network (NHSN) healthcare-associated infection (HAI) metrics into assessments of hospital performance. These programs determine whether hospitals receive financial rewards or penalties using NHSN standardized infection ratios (SIRs), the HAI summary measures that compare current “observed” HAI rates to historical, risk-adjusted “predicted” rates. As such, ensuring the validity and reliability of NHSN measures over time and their relevance to quality is critical.¹

In January 2015, NHSN revised surveillance protocols for multiple HAIs,^2,3 including catheter-associated urinary tract infection (CAUTI). The new CAUTI criteria excluded urinalysis findings and urine cultures growing nonbacterial organisms or <100,000 colony-forming units (CFU) per milliliter of bacteria.^2,4 These revisions occurred within months of initiation of HACRP in October 2014 and HVBP in October 2015, but neither program accounted for case definition changes in determining hospital performance. We recently demonstrated that HACRP and HVBP implementation was not associated with any changes in CAUTI rates when evaluated using a consistent 2015 case definition.⁵ However, the impact of NHSN revisions on CAUTI rates using contemporaneous case definitions and the potential for the revisions to affect the validity of payment program hospital performance scoring have not been described.

Methods

We used an interrupted time-series design to examine the association of the 2015 NHSN CAUTI case definition revisions with changes in observed device-associated CAUTI rates (CAUTI per 1,000 urinary catheter days). We obtained NHSN CAUTI surveillance data from non-federal acute-care hospitals subject to the Inpatient Prospective Payment System (IPPS) and enrolled in the Preventing Avoidable Infectious Complications by Adjusting Payment study.⁶ We included all enrolled adult intensive care units (ICUs) reporting CAUTIs to the NHSN from January 1, 2013, through December 31, 2017. Hospital characteristics were obtained from the 2015 American Hospital Association survey.

We fit negative binomial models for quarterly rates of device-associated CAUTI to assess for changes in either level or trend after the 2015 NHSN revision. We based inferences on generalized estimating equations with robust sandwich variances, accounting for hospital-level clustering. Models included time, a post-2015 NHSN revision indicator, and a 2-way interaction term. We performed sensitivity analyses that included only ICUs contributing data in both the first and last study years and adjusted for hospital size, ownership, and teaching status. We additionally identified the proportion of CAUTI in each quarter prior to 2015 caused by different organisms, including yeast, low-growth bacteria (<100,000 CFU/mL), high-growth bacteria (≥100,000 CFU/mL), and “other” organisms (eg, nonyeast fungi, viruses, or commensal skin organisms also detected in blood culture). For a subset of ICUs with available annual SIR data, we assessed the proportion with performance that was significantly better than expected (SIR< 1.0), worse than expected (SIR > 1.0), or in the expected range (SIR not significantly different from 1.0) in 2014 versus 2015 (ie, pre- vs post-NHSN revisions). To reflect the data reported by hospitals to CMS, all SIRs compared observed CAUTI rates with risk-adjusted predicted rates from 2009. We considered P < .05 to be significant, and we conducted our analyses using SAS version 9.4 software (SAS Institute, Cary, NC). The Harvard Pilgrim Health Care Institute Institutional Review Board approved this study.

Results

From 2013–2017, 1,185 ICUs from 592 eligible study hospitals in 49 states and Washington, DC, reported CAUTIs to the NHSN. Hospitals logged 22,572,494 patient days and 13,607,240 indwelling urinary catheter days and reported 24,898 CAUTI using contemporaneous case definitions. Most were teaching hospitals (n = 344, 58%), not-for-profit hospitals (n = 360, 61%), and medium-sized hospitals (100–399 beds; n = 358, 60%). The CAUTI SIRs were available for 684 ICUs (58%) in both 2014 and 2015. Most hospital ICUs (n = 895, 76%) contributed NHSN data in the first and last study years.

Prior to the 2015 NHSN revisions, 4,655 of 14,351 CAUTIs (32%) reported were due to yeast, and 1,235 (9%) were due to low-growth bacteria (Fig. 1). The NHSN revision was associated with an immediate 42% decline (ie, level change) in reported CAUTI (incidence rate ratio [IRR], 0.58; 95% CI, 0.54–0.63; P < .0001) but no significant change in trend (IRR, 1.00; 95% CI, 0.99–1.02; P = .94) (Fig. 2). Quarterly CAUTI rates were stable in the pre- and postrevision periods (prerevision: IRR, 0.99; 95% CI, 0.98–1.00; P = .19 and postrevision IRR, 0.99; 95% CI, 0.98–1.00; P = .11). Sensitivity analyses were consistent with the primary analysis, indicating a significant, immediate impact of the NHSN revision on CAUTI rates. Between 2014 and 2015, the proportion of ICUs with better than expected SIRs increased from 6% to 16%, while those with worse than expected SIRs decreased from 18% to 2%. Most ICUs had SIRs in the expected range in both 2014 (76%) and 2015 (82%).

Fig. 1.

Aggregated device-associated CAUTI rates for study hospitals by urine culture organism type over time. Each series depicts observed quarterly device-associated CAUTI rates (CAUTIs per 1,000 indwelling urinary catheter days) aggregated for all study hospitals by category of organism associated with the urine culture. The vertical dark-grey double line indicates the timing of the January 2015 NHSN CAUTI surveillance case definition change. The light-grey shaded area depicts the period between implementation of the Hospital-Acquired Condition Reduction Program in October 2014 and Hospital Value-Based Purchasing implementation in October 2015. Note: CAUTI, catheter-associated urinary tract infection; NHSN, National Healthcare Safety Network; Q, quarter.

Fig. 2.

Association of the 2015 NHSN definition revision with observed and model-predicted device-associated CAUTI rates. Closed circles depict the observed quarterly rate of CAUTIs per 1,000 indwelling urinary catheter days derived using contemporaneous NHSN surveillance case definitions. The solid line represents the model-predicted CAUTI rate and the dotted lines represent the upper and lower limits of the 95% confidence intervals of the model-predicted rate. The vertical dark-grey double line indicates the timing of the NHSN CAUTI surveillance case definition change in January 2015. The light-grey shaded area depicts the period between implementation of the Hospital-Acquired Condition Reduction Program in October 2014 and Hospital Value-Based Purchasing implementation in October 2015. Note: Q, quarter.

Discussion

In this analysis of prospective surveillance data from ICUs in 592 acute-care hospitals, the 2015 NHSN CAUTI surveillance definition revision was associated with an immediate 42% reduction in otherwise stable device-associated CAUTI rates. As anticipated, this reduction was attributed to exclusion of nonbacterial organisms and low-growth bacteria from CAUTI criteria.⁷ In contrast, a prior analysis across 2013–2017 showed no change in CAUTI rates associated with HACRP and HVBP implementation when using consistent 2015 NHSN case criteria.⁵

Adoption of NHSN metrics for the HACRP and HVBP represents a shift by CMS toward performance measures that aim to reflect hospital quality more accurately than claims-based infection rates. However, a disconnect remains between NHSN efforts to improve the validity of case ascertainment and implementation of hospital performance score calculations for CMS programs. With the revision-associated reduction in CAUTI rates, more hospitals reported “better than expected” CAUTI SIRs to CMS. This change generated a misleading picture of improvement for CAUTI and may have led to inflation of hospital performance scores in the absence of any true change in hospital quality.

In addition to CAUTI, the 2015 NHSN protocol revisions affected other HAIs targeted by the HACRP and HVBP, including central-line–associated bloodstream infections and surgical site infections.⁸ To correct the problem of postrevision versus prerevision comparisons within SIR calculations, NHSN updated the risk-adjusted predicted rates using 2015 data and began reporting “2015 re-baseline” SIRs starting in January 2017.³ However, CMS programs did not immediately adopt the 2015 re-baseline in their own hospital performance score determinations and continued to use SIRs calculated using pre-2015 predicted infection rates until 2019.

The main limitation of this study was its sole focus on CAUTI, one of several CMS-targeted HAIs affected by NHSN protocol revisions.^2,3 This was done because the NHSN did not routinely collect the data needed to assess how the 2015 NHSN revisions impacted other CMS-targeted outcomes. However, studies from single centers indicate that the 2015 NHSN revisions impacted rates of all payment program-targeted HAIs either directly or indirectly.^9,10

In conclusion, this study highlights the potential pitfalls of value-based incentive models that do not account for changes in measurement criteria, including inappropriate assessment of financial rewards or penalties that do not reflect meaningful differences in patient safety. To ensure the fairness and transparency of these programs, greater coordination between CMS and NHSN will be critical to properly account for revisions in NHSN definitions when they occur.