A Multi-center longitudinal Study of Hospital-Onset Bacteremia: time for a New Quality Outcome Measure?

Abstract

Background

Central line-associated bloodstream infection (CLABSI) rate is an important quality measure, but suffers from subjectivity and inter-rater variability. Decreasing national CLABSI rates may compromise its power to discriminate between hospitals. This study evaluates hospital-onset bacteremia (HOB), (any positive blood culture obtained 48 hours post-admission), as a healthcare-associated infection related outcome measure by assessing association between HOB and CLABSI rates, and comparing the power of each to discriminate quality among ICUs.

Methods

In this multi-center study, ICUs provided monthly CLABSI and HOB rates for 2012 and 2013. A Poisson regression model was used to assess the association between the two rates. We compared the power of each measure to discriminate between ICUs using Standardized Infection Ratios (SIRs) with 95% CIs. A measure was defined as having greater power to discriminate if more of the SIRs (with surrounding CIs) were different from 1.

Results

80 ICUs from 16 hospitals in the US and Canada reported a total of 663 CLABSIs, 475,420 central line days, 11,280 HOBs, and 966,757 patient-days. An absolute change in HOB of 1 per 1,000 patient-days was associated with a 2.5% change in CLABSI rate (P<0.001). Among the 80 ICUs, 20 (25%) had a CLABSI SIR and 60 (75%) a HOB SIR that was different from 1 (P<0.001).

Conclusion

Change in HOB rate is strongly associated with change in CLABSI rate and has greater power to discriminate between ICU performances. Consideration should be given to using HOB to replace CLABSI as an outcome measure in infection prevention quality.

Background

Outcome measures in healthcare play a pivotal role in quantifying the ability of an organization to provide high quality healthcare. Healthcare-associated infection (HAI) measures, in particular National Healthcare Safety Network (NHSN)-defined central line-associated bloodstream infection (CLABSI) rates, are becoming increasingly important as the Centers for Medicare and Medicaid Services (CMS) and private insurers use these measures in pay-for-performance programs such as the Hospital-acquired Conditions Reduction program and the 2015 Value-Based Performance program.(1) The majority of US states mandate public reporting of CLABSI data and publish these data in hospital report cards available to consumers, healthcare providers, and hospital administrators to compare hospital performances in quality of care (2) However, for an outcome measure to adequately serve these purposes, it needs to reflect the truth, be feasible, and have the power to discriminate between facilities. (3) Several studies have shown that the NHSN CLABSI rates do not necessarily reflect the truth, are subjective and resource-intensive, and therefore a questionable choice for such a highly weighted outcome measure. (3-7) Another potential major limitation of NHSN CLABSI as a quality measure is that uniformly low CLABSI rates nationally – including frequent “zeros” – may no longer allow meaningful comparisons between hospitals i.e., this outcome measure may lack the power to truly discriminate between hospitals.

In this study, we investigate a new HAI outcome measure, hospital-onset bacteremia (HOB), defined as a positive blood culture obtained > 48 hours of admission to hospital. In comparison to CLABSI, HOB is objective, simple to understand, easily automated, and easier to collect, thus time-saving. In addition, HOB is a more global or inclusive measure of HAI-related quality as it incorporates bacteremia as a result of any healthcare associated infection (e.g. urinary tract infection or pneumonia) and not just CLABSI.

The first study hypothesis is that changes in HOB rates will be associated with changes in CLABSI rates, meaning that changes in HOB would reflect changes in CLABSI allowing HOB to be used as a CLABSI surrogate while being a more inclusive measure than CLABSI. The second study hypothesis is that HOB is a more frequent event than CLABSI and thus will have greater power to discriminate between (i.e., “rank”) hospitals.

Methods

In this multicenter ecological study, hospitals were recruited through the SHEA research network. The SHEA Research Network is a consortium of more than 200 hospitals conducting multicenter research projects in healthcare epidemiology. (8, 9) Facilities within the US and Canada, with adult, pediatric, or neonatal ICUs were invited to participate. Each center obtained approval from its respective institutional review board.

Study variables were defined as follows:

CLABSI was defined as a primary bloodstream infection in a patient with one or more central lines within the 48-hour period prior to the onset of the bloodstream infection, and the bloodstream infection was not related to any infection at other foci as per CDC definitions. (10)

HOB was defined as a positive blood culture for any organism from any cause (including contaminants and repeat positive blood cultures) sent from the ICU and taken at least 48 hours after admission to hospital. HOB rate was the number of HOBs divided by the number of ICU-patient days.

Total number of blood cultures obtained included all blood cultures, positive and negative, sent from the ICU for each study month.

Data collection

Each participating hospital contributed monthly-aggregate data for each ICU for the number of CLABSIs, central line days, HOBs, ICU patient-days, and total number of blood cultures obtained from January 2012 to December 2013. CLABSI determination was performed by each hospital's infection prevention program independent of this study, by conducting chart review using standard CDC NHSN definitions and reporting methods. The components of the HOB outcome measure were retrieved in an automated fashion directly from hospital microbiology and admission-transfer-discharge databases without medical record review. The ICU-type was also collected using CDC-NHSN classification.(10)

Survey

Each participating hospital completed an on-line survey to assess hospital and ICU level factors (See appendix). Questions included the number of infection preventionists at the hospital and the estimated time spent by infection preventionists on CLABSI surveillance.

Statistical Methods

Association between HOB and CLABSI

We tested the association between HOB and CLABSI using a mixed effect Poisson regression model. Candidate predictors include HOB rate, time period (month and year), hospital, ICU type, and total number of blood cultures obtained. A backward selection for best fit model, using the deviance information criterion, combined with clinical judgment, with CLABSI rates as an outcome was used. The total number of blood cultures obtained was expressed as a rate per 1000 ICU patient days and was included as it was considered an important potential confounder. The ICU was included as a random effect, to account for correlation of observations within the ICU. HOB rate and total number of blood cultures per ICU patient day were included as fixed effects. The over-dispersed distribution of CLABSIs was adjusted by using additive over-dispersion.(11) These analyses were performed in the R programming language using MCMCglmm package.(12)

Discrimination between ICUs

We assessed the ability of HOB and CLABSI to discriminate between different ICUs of the same type using two methods: 1) standardized infection ratios (SIRs) and 2) proportion of ICU-months with zero CLABI and zero HOB.

For method 1 we used indirect standardization methods similar to what is used by CMS on the Hospital Compare website and benchmarked each ICU against similar types of ICU within the cohort (2). For each ICU type (e.g., MICU, SICU etc.) we summed the total number of patient-days and the total number of positive blood cultures for all of those ICUs, and divided the total number of positive blood cultures by the total number of patient-days to get the “benchmark” HOB rate for that type of ICU. For each ICU in the study, the number of expected HOB was calculated using the benchmark rate and observed patient-days. This observed number of HOB was then divided by the expected number to calculate a HOB SIR. This allows for the comparison of ICUs with different number of patient days, e.g. a MICU with a higher number of patient days would be expected to have a higher number of HOBs than another MICU with fewer patient days. The same procedure was used to calculate CLABSI SIRs. Poisson 95% confidence intervals (CI) around each SIR were calculated and interpreted as follows: an SIR 95% CI that includes 1 means that the ICU rate is the same as expected for that type of ICU, greater than 1 means that ICU has a higher than expected rate, and less than 1 means a lower than expected rate. The proportion of ICUs whose SIR and 95% CIs included 1 were calculated and compared for CLABSI and HOB using chi-square (or Fisher's exact) test. We also calculated SIRs for each of the hospitals for overall CLABSI and HOB rates using similar methods to above.

In the second method, we assessed for a “ceiling effect” by calculating the percentage of the total ICU-months with the minimum possible number of CLABSIs and HOBs (e.g. zero). The term ceiling effect is used when the performance of a large proportion of subjects for a given measure is as “good” as possible.(13) The presence of the ceiling effect implies that power to discriminate is compromised and further improvement in performance cannot be captured. We compared the ceiling effect between CLABSI and HOB by comparing the proportion of ICU-months with zero CLABSI to the proportion that had zero HOB, using a chi-square test. These analyses were performed using SAS 9.3 (SAS Institute, Cary NC).

Results

Eighty ICUs from 16 hospitals in the United States and Canada participated in the study. Thirteen of the 16 hospitals were academic hospitals. The number of beds was greater than 500 in 10 hospitals, 300-500 in 4 hospitals, and 100-300 in 2 hospitals. The average number of beds in adult and neonatal ICUs was 17.4, and 37.3 respectively. The average number of infection preventionists was 5.1 per hospital and 1.1 per ICU. Infection preventionists spent an average of 16.6 hours per week on CLABSI surveillance (an average of 2.9 hours per ICU per week). Over the 2-year study period there were 982,609 ICU patient days, 475,420 central line days, and 157,383 total blood cultures obtained, with 11,280 HOBs and 663 CLABSIs reported; CLABSIs represented approximately 6% of the overall HOB. Table 1 shows ICU type and number, and CLABSI and HOB rates and ranges for the participating ICUs.

Table 1.

ICU Types, frequency and rates of CLABSI and HOB.

ICU Type	# ICU	Total # CLABSI	Total # CLD*	CLABSI rate†	# CLABSI range (min, max)	CLABSI rate range† (min, max)	Total # HOB	Total # ICU‡	HOB rate§	# HOB range (min, max)	HOB rate range§ (min, max)
Medical	12	104	85858	1.21	1, 19	0.29, 3	2735	152404	17.95	73, 402	9.41, 39.89
Cardiac	10	53	43234	1.23	1, 13	0.21, 3.77	1254	78869	15.90	35, 216	3.54, 38
Surgical	10	77	69100	1.11	2, 23	0.19, 2.36	1621	127936	12.67	46, 251	5.42, 24.84
Neonatal	9	99	76139	1.30	2, 15	0.45, 2.33	776	238921	3.25	37, 156	1.12, 9.27
Pediatric: Medical/Surgical	9	78	40300	1.94	0, 20	0, 4	880	88601	9.93	7, 203	2.59, 18.3
Cardiothoracic	7	64	57919	1.10	0, 17	0, 1.7	972	76604	12.69	14, 327	4.07, 28.67
Trauma		57	28867	1.97	2, 17	0.8, 2.68	888	56133	15.82	120, 171	8.25, 22.05
Neurosurgical	5	29	26369	1.10	1, 11	0.14, 2.57	460	66469	6.92	65, 136	4.77, 10.1
Burn	4	38	7426	5.12	1, 24	0.86, 11.23	346	24454	14.15	38, 145	6.88, 40.41
Medical/Surgical	4	35	19471	1.80	0, 23	0, 2	710	32082	22.13	17, 414	7.65, 27.16
Neurologic	2	4	7864	0.51	0, 4	0, 0.74	269	22037	12.21	119, 150	9.51, 18.96
Pediatric: Cardiothoracic	1	13	7266	1.79	13, 13	1.79, 1.79	87	8162	10.66	87, 87	10.67, 10.67
Pediatric: Mixed Acuity Unit	1	12	5607	2.14	12, 12	2.14, 2.14	282	9934	28.39	282, 282	28.39, 28.39
Total for all ICUs	80	663	475420				11280	982609

^*CLD= Central Line Day

^†CLABSI rate is expressed per 1000 Central Line days

^‡PD=Patient day

^§HOB rate is expressed per 1000 ICU Patient days

Association between CLABSI and HOB rates

The best, most parsimonious model had HOB rate and rate of total number of blood cultures obtained as independent variables. Adjusting for rate of blood cultures obtained, HOB was associated with CLABSI; an increase in the absolute rate of 1 HOB per 1,000 ICU patient days was associated with a relative increase of 2.5% in CLABSI rate (P<0.001). The regression equation is as follows: log (CLABSI rate / (1-CLABSI rate)) = − 1.639 + (0.024* HOB per 1000 patient days) + (−0.47991* total blood culture per 1000 patient days). For example, an ICU with an increase in their HOB rate from 20 HOB/1000 ICU patient days to 30 HOB/1000 ICU patient days would expect to see an associated 27.7% increase in their CLABSI rate e.g., from 2 CLABSI per 1000 central line days to 2.54 CLABSI per 1000 central line days.

Discrimination between ICUs

Twenty of 80 (25%) of ICUs had CLABSI rates that could be distinguished from the same type ICUs (SIR with 95% CI that does not include 1) while 60 of 80 (75%) of ICUs had HOB rates that were different from the rate for same type of ICU (P < 0.001). Figure 1 shows the SIRs for the 2 most common ICUs: the MICU and NICU.

Figure 1.

“Figure 1 shows the SIRs for CLABSI and HOB for each of the MICUs and NICUs. The vertical line at 1 represents the reference or null value: where the expected rate (study benchmark) of CLABSI or HOB for each MICU or NICU lies (SIR=1). The filled in square represents the HOB rate and the filled in circle represents the CLABSI rate. The horizontal line though each symbol represents the 95% confidence interval around the parameter. Those that lie to the right of the SIR 1 reference line have greater than the expected number of CLABSI or HOB (colored in red; worse than the study benchmark), conversely those that lie to the left have less than expected number of CLABSI or HOB (colored in green; better than the study benchmark). Those that include the expected number of CLABSI or HOB include the SIR reference line and are colored in orange”

Pooling the CLABSI and HOB data from all ICUs per hospital, 9 of 16 (56.3%) hospitals had CLABSI SIRs that included 1, and 2 of 16 (12.5%) hospitals had HOB SIRs that included 1 (P=0.02). CLABSI rates were zero (i.e., achieved the ceiling effect) for 71.7% of individual ICU-months (1,376/1920) compared to HOB rates, which were zero 11.5% (221/1920) of ICU-months (P<0.0001).

Discussion

We collected CLABSI rates and calculated HOB rates for 80 ICUs in 16 hospitals within the US and Canada. We found that changes in HOB rates and CLABSI rates were significantly associated, demonstrating that HOB has merit and should be explored as a surrogate marker for CLABSI. We also showed that HOB is much better at discriminating between ICU performances than CLABSI.

HOB is a more global measure of preventable HAIs than CLABSI; it is inclusive of CLABSI but also bacteremias from other causes such as urinary tract infection and pneumonia. Potential advantages of HOB over CLABSI include the objectivity of the definition as it does not require chart review of the bacteremia, and can be easily obtained in an automated manner from hospital databases. We found HOB rates to be significantly associated with CLABSI rates over time; for every change of 1 HOB per 1,000 ICU-patient days there was a 2.5% change in CLABSI rate. This association has also been demonstrated in a previous single-center study, where a 5.1% decrease in CLABSI post intervention to prevent CLABSI was associated with a 2.7% decrease in HOB. (14) This finding provides support for the premise that HOB could be used in place of CLABSI as a more inclusive HAI outcome measure but still reflect changes in CLABSI rates.

The second major finding of this study was that HOB is much better at discriminating between ICU performances than CLABSI. To demonstrate this point, SIRs were calculated and interpreted in a similar way to the one CMS uses on the Hospital Compare website. (2) ICUs were assigned red, yellow, or green “traffic light” colors meaning they were worse, the same or better than other same type ICUs. The majority of CLABSI SIR confidence intervals (60/80; 75%) were found to include 1, meaning that the majority of CLABSI SIRs cannot be discriminated from an SIR of 1 or from the average performance of same type ICUs. For HOB SIRs only 20/80 (25%) include 1, meaning that the majority can be discriminated from the average performance of same type ICUs. The power to discriminate between SIRs increases with increases in the expected outcome frequency (“sample size”) - i.e., the greater the expected numbers of outcomes (CLABSIs or HOBs), the narrower and more precise the confidence interval, with greater likelihood of discriminating between two SIRs. (15,16) This is represented visually in Figure 1—with so many ICUs receiving the same “traffic light” color for CLABSI and the large overlapping confidence intervals showing little discrimination it is difficult to truly discern poor and good quality and thus CLABSI may not be an ideal measure for public reporting. Of note, some ICUs had differing color SIRs for CLABSI and for HOB, e.g., MICU E. Possible reasons for this may be the inherent lack of objectivity of the CLABSI measure and the potential for over or under reporting. Also because the expected number of CLABSIs is so few, even 1 or 2 additional CLABSIs could result in an SIR going from green to red. The NHSN CLABSI SIR has a strong weight in Domain 2 of the CMS Hospital-acquired Conditions Reduction program and is also included in the Value-Based Purchasing program .(17) Given the lack of power of CLABSI SIR to truly discriminate between ICUs in our study, CLABSI appears to be a poor choice for an HAI outcome measure used in external benchmarking and hospitals could be unfairly financially penalized.

One reason that the CLABSI outcome measure has lack of power to discriminate is due to the infrequency of the occurrence of CLABSI. In fact, there were 71.7% of individual ICU-months with zero CLABSIs, conversely only 11.5% ICU-months had zero HOB. This study shows that CLABSI is subject to ceiling effects; meaning that it is often at the lowest or “best performance” level. There are likely numerous reasons for this infrequent number of CLABSIs seen in our study and in the real world (18). First, there is likely a true decrease in CLABSI resulting from improvements in infection prevention in the last decade. In addition it is possible that due to the significant financial and reputational repercussions associated with higher than expected CLABSI rates, and lack of objectivity in the application of the definition, there may be some intentional or unintentional under-reporting of CLABSI rates.(3,4,6,18) Also, smaller hospitals with fewer ICU beds and fewer central line days may also have frequent zeros. In our study CLABSI represented only approximately 6% of HOB. A hospital may have preventable bacteremia but by only measuring those that are associated with central lines i.e., CLABSI, further improvement cannot be measured due to ceiling effect, and may lead to a false sense that no opportunity for improvement exists. (20)

An additional important benefit of HOB over CLABSI is that it is less resource-intensive than CLABSI. In our study, infection preventionists spent an average of 16.6 hours per week on CLABSI surveillance, which is predominantly manual chart review. For HOB surveillance there is no manual chart review required as all bloodstream infections are included, regardless of etiology. This move toward laboratory based surveillance is supported by recent introduction of laboratory based surveillance for Clostridium difficile infection and Methicillin resistant Staphylococcus aureus bacteremia by the NSHN.

Limitations of this study include the retrospective nature of the data. However, inclusion of bloodstream infections already classified as CLABSIs was in an effort to make the study as “real world” as possible, as these same CLABSIs would be reported to NHSN. Further, the retrospective nature also ensured that CLABSI determination was not affected by the study itself. We used internal sample-derived benchmarks to calculate the SIRs for both HOB and CLABSI. We included multiple positive blood cultures from the same infection episode more than once and included “contaminants” in our HOB rate to make this measure as objective and simple as possible. Laboratory-based algorithms could be used in the future to delete repeat positive blood cultures. Although “contaminants” such as single episodes of coagulase-negative staphylococcus bacteremia may not represent HAI, it is likely that good blood culture drawing technique with adequate attention to sterile technique would result in low rates of these contaminants so the presence of these organisms may in a sense represent poor quality of care. We did not collect data on the microbiology or etiology of hospital-onset bacteremia, and were therefore unable to identify specific areas for improvement. However, the aim of this particular study was to evaluate this metric for external reporting rather than understand the causes of bacteremia in the hospital. Finally, both CLABSI and HOB are subject to the possibility of “surveillance bias” in that the harder you look the more you find.

A potential challenge for implementation is the lack of baseline data to derive national SIR benchmarks for HOB. However, if adopted, similar to other new outcome measures, hospitals could submit HOB data to NHSN during a baseline period of only data collection without public reporting. Based on these data, benchmark HOB rates could be established for future SIR calculations. In summary, in this multi-center study we showed that HOB rates are strongly associated with CLABSI rates and that HOB has much better power than CLABSI to discriminate between ICU HAI-related quality performances. These results, in addition with the objectiveness, simplicity, and global nature of the HOB measure may make it more attractive than CLABSI as an outcome measure of hospital quality of care. However, this study also highlights areas worthy of future research prior to utilization of HOB as a quality outcome measure. These include identifying the causes of hospital-onset bacteremia, understanding how often and for what indication duplicate positive blood cultures occur, appropriate ways of risk adjustment of HOB rates, and assessment of hospital-onset bacteremia in non-ICU locations and in the community hospital setting which was under-represented in our study.