The effect of adding comorbidities to current CDC central line-associated bloodstream infection (CLABSI) risk adjustment methodology

Abstract

Background

Risk adjustment is necessary to fairly compare central line-associated bloodstream infections (CLABSI) rates between hospitals. Until 2017, CDC methodology adjusted CLABSI rates only by type of intensive care unit (ICU). 2017 CDC models also adjust for hospital size and medical school affiliation. We hypothesized that risk adjustment would be improved by including patient demographics and comorbidities from electronically-available hospital discharge codes.

Methods

Using a cohort design across 22 hospitals, we analyzed data from ICU patients admitted between January 2012 and December 2013. Demographics and ICD-9-CM discharge codes were obtained for each patient and CLABSI were identified by trained infection preventionists. Models adjusting only for ICU type and for ICU type plus patient case mix were built and compared using discrimination, and standardized infection ratio (SIR). Hospitals were ranked by SIR for each model to examine the changes in rank from one model to the other.

Results

85,849 ICU patients were analyzed and 162 (0.2%) developed CLABSI. The significant variables added to the ICU model were coagulopathy, paralysis, renal failure, malnutrition, and age. The C-statistics were 0.55 (95% CI: 0.51–0.59) for the ICU-type model and 0.64 (95% CI: 0.60, 0.69) for the ICU-type plus patient case mix model. When the hospitals were ranked by adjusted SIRs, 10 hospitals (45%) changed rank when comorbidity was added to the ICU type model.

Conclusions

Our risk adjustment model for CLABSI using electronically-available comorbidities demonstrated better discrimination than the CDC model. Comorbidity-based risk adjustment should be strongly considered by the CDC to more accurately compare CLABSI rates across hospitals.

Central line-associated bloodstream infections (CLABSI) are responsible for substantial morbidity and mortality among hospitalized patients. Patients with CLABSI are at a higher risk of death, have longer hospital stays, and incur more healthcare costs than patients without CLABSI.¹ Since January 2012, hospital reimbursement by the Centers for Medicare and Medicaid Services (CMS) is dependent on public reporting of CLABSI rates. CMS hospitals use the operational system of the Centers for Disease Control and Prevention’s National Healthcare Safety Network (CDC NHSN) to facilitate reporting.²

The CDC employs risk adjustment in order to fairly compare CLABSI rates across hospitals. Until 2017, CDC NSHN adjusted CLABSI rates only by type of intensive care unit (ICU). In 2017, the CDC added hospital size (i.e. licensed beds) and medical school affiliation as additional risk adjustment variables.³ However, neither of these CDC models adjust for individual patient level factors including comorbid conditions. We hypothesized that risk adjustment could be improved by including demographics and comorbid conditions from electronically-available hospital discharge codes.

Methods

Using a cohort design, we retrospectively analyzed ICU patients admitted between January 1, 2012 and December 31, 2013 from 22 US hospitals. Facilities were recruited as part of a partnership between Premier, Inc., the Society for Healthcare Epidemiology of America (SHEA) Research Network, and the University of Maryland, School of Medicine. For those facilities volunteering to participate in the study, Institutional Review Board IRB and facility consent were obtained.

Using Premier’s Quality Advisor™ database, we obtained demographic and International Classification of Diseases, Ninth Edition, Clinical Modification (ICD-9-CM) discharge codes on each adult ICU patient. Patients with CLABSI were identified by trained infection preventionists at each hospital using CDC NHSN definitions.⁴ We also obtained information on the size of the hospital (i.e., number of beds) and whether the hospital was associated with an academic medical school.

Risk adjustment models were built using discrete survival analysis, a method that accounts for time at risk.⁵ Specifically, acquisition of CLABSI on each day in the ICU was used as the outcome of a binary regression model with a complementary log-log link. A random intercept for hospital was included in the model to account for the clustering of patients within hospitals.

Two models were constructed: 1) a model containing only ICU-type (CDC methodology prior to 2017) and 2) a model containing ICU-type plus patient case-mix variables model. For this latter model, we identified candidate comorbidity variables using expert consensus, which has been reported elsewhere.⁶ Using a modified Delphi method, nine infectious disease and infection control experts were asked to rate the 35 comorbid conditions found in the Charlson and Elixhauser Comorbidity Indices from 1 (not at all related) to 5 (strongly related), based on perceived relatedness to CLABSI. These experts rated the following 14 conditions in terms of causality with CLABSI as 3 (somewhat related) or higher: coagulopathy, dementia, diabetes without complications, diabetes with complications, drug abuse, hemiplegia or paraplegia, HIV/AIDS, lymphoma, malignancy, solid tumor with metastasis, severe liver disease, obesity, renal disease, and weight loss (malnutrition). These 14 conditions (identified using ICD-9-CM codes), along with ICU type, age, gender, race, hospital size, and medical school affiliation were entered into the model as potential predictors of CLABSI. Hospital size was defined in the 2017 CDC NHSN model as a binary variable indicating the number of beds in the hospital ≥276.³ Variables were retained using backwards selection if they met the significance level of alpha <0.05.

For both models, we estimated the marginal predicted probabilities of a CLABSI for each patient-day in the ICU without including the random effect in the prediction so that hospital characteristics did not influence these values. These predicted probabilities were then used to generate the C-statistic and 95% confidence interval (CI) for both models. The C-statistic is a measure of discrimination, or the model’s ability to discriminate between those with and without the outcome. The C-statistic is the chance that the model will assign a higher probability to patients with CLABSI than without.⁷ Values for the C-statistic range from 0.50, a probability no different from chance, to 1.0, which is perfect prediction. Calibration, which is the model’s ability to accurately quantify the probability of the outcome, was assessed with a calibration plot. The predicted probabilities were plotted against the observed proportion of CLABSI in deciles and a 45-degree line was added to visually inspect how well the model was calibrated. In a perfectly calibrated model, the points would rest exactly on the 45-degree line, implying that the predicted risks are equal to the observed rate.^8,9

Unadjusted CLABSI rates were calculated for each hospital by dividing the number of CLABSI by the total number of ICU days. To calculate risk-adjusted rates, the predicted probabilities from the risk adjustment model were summed to estimate the expected number of CLABSI events for each hospital. Standardized infection ratios (SIR) for each hospital were calculated by dividing the observed number of CLABSI by the expected number predicted by the ICU-type plus patient case mix model. An SIR above 1 indicates the hospital reported a greater number of CLABSI than expected, while an SIR below 1 indicates the hospital reported a lower number of events than expected by the model.¹⁰ Hospitals were then ranked by the case mix risk-adjusted SIRs and compared to the rankings when ordered by the ICU-type only risk adjusted SIRs.

All analyses were conducted using SAS version 9.4 (The SAS Institute, Inc., Cary, NC). The calibration plots were generated using the “ggplot2” package in R studio (Version 0.99.902).

Results

Twenty two hospitals contributed ICU data. The analysis included 85,849 ICU patients of whom 162 (0.2%) developed a CLABSI. Sixteen (73%) of the hospitals were large (≥296 beds), 11 (50%) were affiliated with medical schools, and 20 (90%) were located in urban areas. Across hospitals, 22,560 (26%) patients were from nine medical cardiac critical care units, 18,157 (21%) from eight medical critical care units, 34,537 (40%) from 14 medical/surgical critical care units, and 10,595 (12%) from six surgical critical care units based on CDC ICU definitions. All patients had a minimum of nine ICD-9-CM codes with a median of 27 and a maximum of 65 codes.

Table 1 presents a bivariate analysis of the relationship between CLABSI and patient demographics and comorbidities. ICU type, age, coagulopathy, paralysis, liver disease, renal failure, and malnutrition were significant at the p<0.10 level in the bivariate analysis. Using the medical cardiac care ICU as the reference category, medical/surgical critical care (p=0.06) and surgical critical care (p=0.03) ICUs were predictive of CLABSI, but the medical critical care (p=0.40) ICU was not. Table 2 presents the results of the ICU-type plus patient case mix model. The variables added to the ICU-type only model were coagulopathy (p<0.01), paralysis (p=0.03), renal failure (p<0.01), malnutrition (p=0.01), and patient age in 10 year increments (p<0.01). Facility hospital size (p=0.33) and medical school affiliation (p=0.152) were not significant predictors of CLABSI and were therefore dropped from both models.

Table 1.

Characteristics of 85,849 patients with and without CLABSI, hazard ratios (95% confidence intervals), and p-values admitted to the ICU between January 1, 2012 and December 31, 2013

Variable	CLABSI n=162 Mean (SD) n (%)	Non-CLABSI n=85,687 Mean (SD) n (%)	Hazard Ratio (95%CI)	p-value
Age in years	60.2 (17.2)	63.0 (16.9)	0.99 (0.98, 1.00)	0.012
Sex
Female	71 (0.17)	39,094 (99.8)	Reference
Male	91 (0.18)	46,590 (99.8)	0.96 (0.70, 1.31)	0.793
Race
Black	42 (0.29)	13,225 (99.7)	1.70 (1.15, 2.52)	0.008
Other	17 (0.17)	6,998 (99.8)	1.44 (0.85, 2.42)	0.175
White	103 (0.15)	65,364 (99.8)	Reference
ICU type
Medical cardiac	36 (0.14)	22,524 (99.8)	Reference
Medical critical care	32 (0.16)	18,125 (99.8)	1.52 (0.85, 2.70)	0.156
Medical/surgical critical care	64 (0.18)	34,473 (99.8)	1.82 (1.03, 3.22)	0.040
Surgical critical care	30 (0.24)	10,565 (99.7)	1.98 1.14 (3.46)	0.016
Coagulopathy	52 (0.39)	12,258 (99.6)	1.70 (1.22, 2.37)	0.002
Dementia	2 (0.24)	751 (99.7)	1.36 (0.34, 5.12)	0.665
Diabetes uncomplicated	39 (0.15)	23,236 (99.8)	0.87 (0.61, 1.26)	0.468
Diabetes complicated	17 (0.23)	6,696 (99.8)	1.22 (0.74, 2.01)	0.446
Drug abuse	9 (0.15)	5,726 (99.8)	0.79 (0.40, 1.55)	0.489
Paralysis	17 (0.45)	3,659 (99.5)	1.89 (1.14, 3.14)	0.013
HIV/AIDS	2 (0.45)	411 (99.5)	1.58 (0.39, 6.40)	0.524
Lymphoma	4 (0.34)	1,057 (99.6)	1.60 (0.59, 4.31)	0.355
Malignancy	9 (0.11)	6,773 (99.9)	0.63 (0.32, 1.24)	0.185
Metastatic cancer	10 (0.24)	3,538 (99.7)	1.42 (0.75, 2.70)	0.281
Liver disease	31 (0.37)	7,667 (99.6)	1.68 (1.13, 2.49)	0.010
Obesity	32 (0.20)	14,956 (99.8)	1.02 (0.69, 1.50)	0.927
Renal disease	56 (0.28)	19,822 (99.7)	1.38 (1.00, 1.92)	0.050
Weight loss (malnutrition)	55 (0.47)	10,804 (99.5)	1.74 (1.25, 2.42)	0.001

Table 2.

Hazard ratios with 95% confidence intervals, p-values and the C-statistic for the ICU-type plus patient case mix model.

Variable	HR (95%CI)	p-value	C-Statistic (95% CI)
ICU type			0.64 (0.60, 0.69)
Medical cardiac	Reference
Medical critical care	1.28 (0.72, 2.26)	0.400
Medical/surgical critical care	1.70 (0.98, 2.95)	0.060
Surgical	1.83 (1.04, 3.20)	0.034
Coagulopathy
No	Reference
Yes	1.65 (1.17, 2.30)	0.004
Paralysis
No	Reference
Yes	1.76 (1.06, 2.93)	0.029
Renal disease		0.009
No
Yes	1.59 (1.13, 2.22)
Weight loss
No	Reference
Yes	1.56 (1.12, 2.19)	0.007
Age (per 10 year increase)	0.88 (0.80, 0.96)	0.006

The C-statistic (shown in Figure 1) was 0.55 (95% CI: 0.51, 0.59) for the ICU-type only model and 0.64 (95% CI: 0.60, 0.69) for the ICU-type plus patient case mix model, a statistically significant difference (p<0.001). When the hospitals were ranked by adjusted SIRs and compared (Table 3), 10 hospitals (45%) changed rank (4 increased in rank and 6 decreased) when comorbidities were added to the ICU-type only model. Figures 2 and 3 show the calibration of the ICU-type only model and the ICU-type plus patient case-mix model. Our final model shows better calibration than the ICU-type only model, which overestimates the expected rate relative to the observed CLABSI rate in some subgroups.

Figure 1.

ROC curves comparing the ICU-type only model to the ICU-type plus patient Case Mix

Table 3.

Ranking of hospitals^a (in order of CDC ranking) with the ICU-type only model and ICU-type plus patient case mix risk adjustment

Hospital	ICU-type only Model SIR	ICU-type only model Rank	ICU-type + Case Mix Model SIR	ICU-type only + Case Mix Rank	Difference in Rank	Direction
A	0.15	1	0.15	1	0
B	0.17	2	0.17	2	0
C	0.20	3	0.23	3	0
D	0.38	4	0.44	4	0
E	0.62	5	0.67	5	0
F	0.68	6	0.70	6	0
G	0.83	7	0.83	7	0
H	0.88	8	0.87	8	0
I	1.03	10	0.94	9	1
J	0.93	9	0.95	10	−1
K	1.06	11	1.00	11	0
L	1.10	12	1.16	12	0
M	1.53	18	1.29	13	5
N	1.30	13	1.30	14	−1
O	1.34	14	1.30	15	−1
P	1.36	16	1.37	16	0
Q	1.61	19	1.38	17	2
R	1.48	17	1.44	18	−1
S	1.35	15	1.50	19	−4
T	2.94	20	2.66	20	0
U	3.32	22	2.73	21	1
V	3.29	21	3.50	22	−1

NOTE. SIR, standardized infection ratio.

^aIn order of ICU-type-only model ranking.

Figure 2.

Calibration curve for the ICU-type only model

Figure 3.

Calibration curve for the ICU-type plus patient case mix model

Discussion

In this retrospective cohort study, we illustrate the importance of adjusting for patient case-mix variables including comorbid conditions when comparing CLABSI across hospitals. Other than the existing CDC model, this is the first analysis to develop risk adjustment models for CLABSI. Further, the CDC models do not incorporate comorbid conditions or other significant patient factors such as age. Though our model incorporating these factors showed modest discrimination, it showed better discrimination than a model with only ICU-type (CDC risk model until 2017). The additional 2017 CDC variables of medical school affiliation and facility hospital size were not statistically significant predictors of CLABSI in our cohort.

We further demonstrate the importance of risk adjustment by showing the change in rankings of the hospitals that resulted when the risk adjustment model including comorbid conditions was applied. Hospitals with a large burden of patients with more comorbid conditions are expected to have a larger rate of CLABSI and their ranking will improve once the risk adjustment model is applied. Likewise, hospitals that serve healthier patients with fewer comorbidities may decline in their performance ranks when SIRs are adjusted for patient case-mix. These shifts may have consequences on payments and penalties for an individual hospital when all U.S. hospitals are included in this ranking, as currently done by CMS.

CDC models prior to 2017 only adjusted for type of ICU.¹⁰ The new 2017 CDC model added medical school affiliation and facility hospital size as variables.³ Although these variables are unlikely causally related to CLABSI occurrence, they were probably selected as proxy variables for patient case-mix. However, while medical school affiliation may represent a case-mix of patients who have more comorbid conditions and higher severity of illness that merits risk adjustment, it may also represent more inexperienced providers that should not be adjusted for when the intent is to use those adjusted rates for quality of care comparisons. Similarly, facility hospital size is likely associated with several patient case-mix and care delivery factors making the direction of influence on CLABSI difficult to predict. Indeed, in our large and diverse cohort, neither medical school affiliation nor facility hospital size were significantly associated with CLABSI. Therefore, we suggest that it is better to directly adjust for patient demographics and comorbid conditions when possible.

Our analysis has a number of strengths. Infection preventionists used standardized CDC NHSN criteria to identify CLABSI such that outcome assessment is comparable across hospitals. We were able to use comorbid conditions from discharge codes already collected routinely for other purposes, and therefore incorporation of these variables into current national risk adjustment would not require any additional data collection burden on the part of hospitals. In fact, ICD diagnostic codes are already routinely transmitted to CMS by hospitals. Use of discharge codes may also encourage the use of risk adjustment as ICD diagnostic codes are easier to access and are collected on every patient by trained individuals in a standardized fashion.

Our approach has some limitations. The majority of our sample were large, urban facilities, which may limit the generalizability of our findings to other hospitals. The Premier database did not have data on central-line days so we were unable to use this measure for our denominator, nor account for patients with more than one line. Our use of ICU days as the denominator may have underestimated the overall CLABSI rate in each unit. While this may have misclassified patient time at risk, we have no reason to believe that this misclassification is differential. Work by Horstman et al.¹¹ has shown that ICU days correlates strongly with device days and that hospital performance rankings using either measure were also strongly correlated. A criticism of the use of ICD-9-CM codes in research is that they fail to capture all patient comorbidities or could reflect codes that maximize reimbursement.^12,13 Research comparing the Charlson and Elixhauser Comorbidity Indices derived from ICD-9-CM codes to those same scores extracted from chart review has found that the sensitivity of the individual components varies greatly but that specificity is nearly 100%.^14,15 Therefore, while some patient comorbidities may have been missed due to low sensitivity of the ICD codes, a condition assigned to a patient is likely to be correct.^14–16 As such, we may be underestimating the prevalence of these conditions in our study resulting in smaller rank changes after adjustment. Despite this limitation, our models still demonstrated good discrimination. Another limitation is that we used ICD-9-CM codes and hospitals have recently switched to ICD-10 codes; however, this is unlikely to affect the discrimination of our model as the identified comorbid conditions can be directly cross-walked between ICD-9-CM and ICD-10.¹⁷

Our analyses demonstrate the importance of using individual demographic data and comorbidities in risk adjustment models. We believe that the CDC and CMS should strongly consider incorporating comorbid conditions obtained by electronically-available ICD codes into their risk adjustment models for CLABSI.