Improving Risk Adjustment Above Current Centers for Disease Control and Prevention Methodology Using Electronically Available Comorbid Conditions

Abstract

OBJECTIVE.

To identify comorbid conditions associated with surgical site infection (SSI) among patients undergoing renal transplantation and improve existing risk adjustment methodology used by the Centers for Disease Control and Prevention National Healthcare Safety Network (NHSN).

PATIENTS.

Patients (≥18 years) who underwent renal transplantation at University of Maryland Medical Center January 1, 2010-December 31, 2011.

METHODS.

Trained infection preventionists reviewed medical records to identify surgical site infections that developed within 30 days after transplantation, using NHSN criteria. Patient demographic characteristics and risk factors for surgical site infections were identified through a central data repository. International Statistical Classification of Disease, Ninth Revision, Clinical Modification codes were used to analyze individual component comorbid conditions and calculate the Charlson and Elixhauser comorbidity indices. These indices were compared with the current NHSN risk adjustment methodology.

RESULTS.

A total of 441 patients were included in the final cohort. In bivariate analysis, the Charlson components of cerebrovascular disease, peripheral vascular disease, and rheumatologic disorders and Elixhauser components of obesity, rheumatoid arthritis, and weight loss were significantly associated with the outcome. A model utilizing the variables from the NHSN methodology had a c-statistic of 0.56 (95% CI, 0.48–0.63), whereas a model that also included comorbidities from the Charlson and Elixhauser indices had a c-statistic of 0.65 (95% CI, 0.58–0.73). The model with all 3 risk adjustment scores performed best and was statistically different from the NHSN model alone, demonstrated by improvement in the c statistic (0.65 vs 0.56).

CONCLUSION.

Risk adjustment models should incorporate electronically available comorbid conditions.

Healthcare outcomes, including healthcare-associated infections (HAIs), are often used in quality improvement for public reporting and with the intent of comparing rates across different facilities. However, patients at one facility may be significantly different from patients at another facility and thus, risk adjustment is necessary in order to make comparisons meaningful across providers and healthcare centers. Numerous groups have outlined weaknesses with the current risk adjustment methodology for HAIs.^1–3 A major weakness in the current method is a lack of adjustment for patient-level factors.^1–3

The Centers for Disease Control and Prevention (CDC) modifies its risk adjustment methodology for surgical site infections (SSIs) on an ongoing basis. Prior methodology included adjustment for the American Society of Anesthesiologists score; the duration of surgery; and whether the procedure was clean, clean contaminated, or dirty.⁴ Present methodology is procedure specific. For example, for renal transplantation, more recent methodology takes into account age, American Society of Anesthesiologists score, procedure duration, and size of facility.⁵ However, for each procedure, the CDC’s updated SSI risk adjustment does not take into account patient-level comorbid conditions.

The aim of this study was to identify comorbid conditions associated with SSI and to develop risk adjustment scores from electronically available comorbidity data. We hypothesized that these scores and their components would improve upon the current CDC risk adjustment methodology. To test this hypothesis we used a data set that included a cohort of patients undergoing renal transplantation, from which the main results have been previously published.⁶

METHODS

We identified a retrospective cohort of patients (≥18 years old) who underwent renal transplantation at the University of Maryland Medical Center from January 1, 2010, through December 31, 2011. Study recruitment has been reported elsewhere.⁶ Briefly, eligible patients were identified from the transplant database of the University of Maryland Medical Center. Trained infection preventionists reviewed the inpatient and outpatient medical records of all eligible renal transplant patients to identify SSIs that had developed within 30 days of transplantation, using CDC National Healthcare Safety Network (NHSN) criteria.^7,8 Patients who underwent dual transplantations were excluded from our cohort. Supplemental surveillance through medical chart review was conducted to identify additional patient demographic characteristics and risk factors for SSIs. Covariates analyzed with the outcome of SSI included age, sex, race, body mass index (calculated as weight in kilograms divided by height in meters squared), American Society of Anesthesiologists score, procedure duration (time from incision to wound closure), and discharge International Statistical Classification of Disease, Ninth Revision, Clinical Modification (ICD-9-CM) codes. Electronically available ICD-9-CM codes were used to analyze individual component comorbid conditions and calculate the Charlson (Deyo) and Elixhauser scores, 2 previously validated comorbidity scores.^9,10

The most widely used measure in studies of infectious diseases is the Deyo adaptation of the Charlson comorbidity index.¹¹ The Charlson consists of 17 comorbid conditions, assigned a weight of 1, 2, 3, or 6, and summed to create a total score between 0 and 33.⁹ Originally developed to predict 1-year in-hospital mortality, the Charlson has been used to predict outcomes such as postoperative mortality and infectious disease outcomes, in a number of patient populations.^12,13 The Elixhauser index, on the other hand, is the sum of 30 unweighted comorbid conditions in which possible scores range from 0 to 30.^10,14 Though a relatively new risk adjustment method, the Elixhauser has been validated in hospital administrative data¹⁴ and has a larger list of comorbid conditions than the Charlson.

Means and standard deviations or medians and interquartile ranges were generated for the continuous variables including Charlson and Elixhauser scores. The t test or Wilcoxon rank sum test was used to compare those who developed an SSI and those who did not. Frequencies and proportions were generated for the categorical variables, including individual components of Charlson and Elixhauser scores, and the χ² test was used to compare distributions between those who developed an SSI and those who did not. Components significant at the alpha = .10 level were candidates for inclusion in the logistic regression models to predict occurrence of SSI.

Five separate logistic regression models were built as follows: (1) the “CDC NHSN model,” which is based on the variables in the current CDC NHSN SSI risk adjustment model⁵; (2) the “Charlson model” using components of the Charlson Comorbidity Index; (3) the “Elixhauser model” using components of the Elixhauser Index; (4) the “Charlson/ Elixhauser model” using nonoverlapping components of both Charlson and Elixhauser model; and (5) the “Charlson/ Elixhauser/CDC NHSN” model using nonoverlapping components of Charlson and Elixhauser, as well as components of the NHSN model.

The CDC NHSN model for kidney transplantation (procedure code KTP) is based on age (>59 vs ≤59), American Society of Anesthesiologists score (>3 vs ≤3), and bed size (>500 vs ≤500) and codes the continuous variable procedure duration as the number of 10-minute increases in duration. Bed size was dropped from our model because these data come from a single site and thus did not vary among this patient population. For the Charlson, Elixhauser, and combined models, components from the bivariate analyses were selected if P ≤ .10 and were entered into the model and retained if P ≤ .05. All 4 models were compared with the referent CDC NHSN model.

Comparison of risk adjustment models is generally performed via both discrimination and calibration. Model discrimination refers to the ability of the model to discriminate between patients who had the event and those who did not based on the predictor variables. The c-statistic is defined by the area under the receiver operating characteristic curve, a plot of sensitivity versus “1 – specificity” for all values of the predictor variable. Discrimination was assessed by calculating the c-statistic and associated 95% confidence interval to test the equality of the model’s c-statistics for each model. A c-statistic of 0.50 is no better than random chance, whereas 1.0 is perfect discrimination. Calibration of the models, or how well the predicted risk correlates to the observed risk, was assessed with the Hosmer-Lemeshow goodness of fit statistic. This statistic divides the sample into 10 deciles of increasing risk to compare the number of observed events with the number of events predicted by the model. Thus a smaller value between observed and expected is desired. Therefore, a model with good fit would have a low Hosmer-Lemeshow χ² statistic and a nonsignificant P value (>.05).¹⁵ All analyses were performed using SAS, version 9.4 (SAS Institute).

RESULTS

This cohort consisted of 441 adult patients who underwent renal transplantation. Patient characteristics are presented in Table 1. Of the 441 patients, 66 (15%) developed an SSI. Among the 66 SSIs, 31 (47%) were superficial-incisional and 35 (53%) were deep-incisional or organ-space SSIs. There was no difference between those who developed an SSI and those who did not with respect to age, sex, or race. Body mass index was significantly higher in those who developed an SSI than those who did not. The median (interquartile range) Charlson score was 3.0 (2), and the mean (SD) Elixhauser score was 4.0 (2). These scores did not differ significantly between the outcome groups.

TABLE 1.

Patient Characteristics in Study of Improving Risk Adjustment Methodology

Variable	Entire cohort (N = 441)	SSI+ (N = 66)	SSI− (N = 375)	P value
Age, mean (SD), y	53.0 (13)	52.2 (11)	53.2 (13)	.5502
Male sex, n (%)	257 (58)	39 (59)	218 (58)	.8843a
White race, n (%)	210 (48)	28 (42)	182 (49)	.3631a
BMI, mean (SD)	29.7 (6)	31.7 (6)	29.4 (6)	.0045
ASA score, n (%)				.3304a
2	6/401 (1)	2/61 (3)	4/340 (1)
3	323/401 (81)	46/61 (75)	277/340 (81)
4	72/401 (18)	13/61 (21)	59/340 (17)
Duration, median (IQR), min	182 (78)	179 (80)	184 (80)	.7992b
Charlson comorbidity index, median (IQR)	3.0 (2)	4.0 (3)	3.0 (2)	.0577b
Elixhauser index, mean (SD)	4.0 (2)	4.1 (2)	4.0 (2)	.5841

NOTE. ASA, American Society of Anesthesiologists; BMI, body mass index (calculated as weight in kilograms divided by height in meters squared); IQR, interquartile range; SSI, surgical site infection.

^aχ² test.

^bWilcoxon rank sum test.

In the bivariate analysis (Table 2), the Charlson components of cerebrovascular disease, peripheral vascular disease, and rheumatologic disorders and Elixhauser components of obesity, rheumatoid arthritis, and weight loss were significantly associated with the outcome. Owing to sparse cell counts, cerebrovascular disease was dropped from the final Charlson model whereas all 3 Elixhauser components were significant and retained in the final Elixhauser model. The final combined model of nonoverlapping Charlson and Elixhauser components included obesity, peripheral vascular disease, rheumatologic disorders, and weight loss.

TABLE 2.

Comparison of Charlson and Elixhauser Component Conditions in 66 SSI Positive and 375 SSI Negative Patients in Study of Improving Risk Adjustment Methodology

	Charlson (Deyo)			Elixhauser (original)
Variable	Value in SSI+ n (%)	Value in SSI− n (%)	P value	Value in SSI+ n (%)	Value in SSI− n (%)	P value
Alcohol abuse				0 (0)	4 (1)	>.999
Anemia (blood loss)				3 (5)	27 (7)	.5981
Anemia (deficiency)				28 (42)	149 (40)	.6809
Cardiac arrhythmia				5 (8)	44 (12)	.3216
Cerebrovascular	0 (0)	20 (5)	.0552a
Chronic pulmonary	10 (15)	39 (10)	.2574	10 (15)	39 (10)	.2574
Coagulopathy				9 (14)	62 (17)	.5549
Depression				7 (11)	42 (11)	.8874
Dementia	0 (0)	0 (0)	…
Diabetes	5 (8)	45 (12)	.2958	45 (68)	5 (1)	.2958
Diabetes w/complications	26 (39)	110 (29)	.1027	26 (39)	110 (29)	.1027
Drug abuse				2 (3)	11 (3)	>.99
Fluid and electrolyte disorders				42 (64)	234 (62)	.8482
Heart failure	5 (8)	30 (8)	.9064	5 (8)	30 (8)	.9604
Hemi-paraplegia	0 (0)	0 (0)	…	0 (0)	0 (0)	…
HIV/AIDS	2 (3)	7 (2)	.6292	2 (3)	7 (2)	.6292
Hypertension				2 (3)	11 (3)	>.99
Hypertension complicated				0 (0)	2 (1)	>.99
Hypothyroidism				8 (12)	45 (12)	.9777
Liver disease				2 (3)	13 (3)	.8569
Lymphoma				1 (2)	2 (1)	.3859
Malignancy	0 (0)	3 (1)	>.99	8 (12)	35 (9)	.4814
Metastatic tumor	0 (0)	0 (0)	…	0 (0)	0 (0)	…
Mild liver	1 (2)	5 (1)	>.99
Myocardial infarction	9 (14)	36 (10)	.3178
Neurologic (other)				2 (3)	15 (4)	.7059
Obesity				12 (18)	41 (11)	.0949
Peptic ulcer	0 (0)	7 (2)	.6009	0 (0)	7 (2)	.6009
Peripheral vascular	8 (12)	21 (6)	.0594a	9 (14)	32 (9)	.2171
Psychosis				1 (2)	2 (1)	.3859
Pulmonary circulation disorders				3 (5)	32 (9)	.2691
Renal disease	66 (100)	374 (99)	>.99	66 (100)	375 (100)	…
Rheumatologic	7 (11)	14 (4)	.0156a	7 (11)	15 (4)	.0326a
Severe liver	0 (0)	2 (1)	>.99
Valvular				6 (9)	29 (8)	.7067
Weight loss				2 (3)	36 (10)	.0794a

NOTE. HIV, human immunodeficiency virus.

^aSignificant at the P = .10 level.

Table 3 presents the calibration and discrimination results, and Figure 1 shows the receiver operating characteristic curves for the CDC NHSN model and the full model. For the current CDC NHSN model, the c-statistic was 0.56 (95% CI, 0.48–0.63). For the Charlson model the c-statistic was 0.56 (0.51–0.61). The c-statistic was 0.59 (95% CI, 0.53–0.65) for the Elixhauser model and compared with the NHSN model there was no difference in discrimination between the 2 models. For the combined model that included non-overlapping Charlson/Elixhauser components, the c-statistic was 0.63 (95% CI, 0.56–0.69). This model was not different from the NHSN model. The Charlson/Elixhauser/CDC NHSN model had a c-statistic of 0.65 (95% CI, 0.58–0.73), which does represent a statistically significant difference in discrimination compared with the NHSN model. The Hosmer-Lemeshow χ² test indicated that all the models had good calibration as demonstrated by nonsignificant P values (see Table 3).

TABLE 3.

Comparison of Discrimination Abilities and Calibration Between NHSN, Charlson, Elixhauser, and Combined Models in Study of Improving Risk Adjustment Methodology

Model	c-statistic (95% CI)	Hosmer Lemeshow χ2	P valuea
CDC NHSN model	0.56 (0.48–0.63)	12.88	.12
Charlson model (peripheral vascular and rheumatologic disorders)	0.56 (0.51–0.61)	0.03	N/A
Elixhauser model (obesity, rheumatoid arthritis, and weight loss)	0.59 (0.53–0.65)	0.12	.73
Charlson/Elixhauser model (peripheral vascular, obesity, rheumatoid arthritis, and weight loss)	0.63 (0.56–0.69)	0.40	.82
Charlson/Elixhauser/CDC NHSN model	0.65 (0.58–0.73)	2.12	.98

NOTE. CDC, Centers for Disease Control and Prevention; N/A, not applicable; NHSN, National Healthcare Safety Network.

^aA nonsignificant P value indicates good model fit.

FIGURE 1.

The receiver operating characteristic (ROC) curves for the Centers for Disease Control and Prevention National Healthcare Safety Network (NHSN) model and the full model.

DISCUSSION

We found that the addition of comorbid conditions obtained from ICD-9-CM codes improved the current CDC NHSN SSI risk adjustment model among a cohort of patients undergoing renal transplantation. Further, the current CDC NHSN SSI model had little ability to discriminate between patients with an SSI and those without following renal transplantation based on a c-statistic of 0.56. In fact, since the confidence interval for the c-statistic included 0.50, the data are consistent with the possibility that the NHSN score performs no better than random chance in this setting. The models that included the Elixhauser comorbid components alone and the combination Charlson/Elixhauser indices had better discrimination than the NHSN model (as indicated by a higher c-statistic). The model that included all 3 risk adjustment scores performed the best and was statistically different from the CDC NHSN model alone as demonstrated by improvement in the c-statistic (0.65 compared with the CDC model of 0.56).

The improved performance of the Charlson/Elixhauser/ CDC NHSN model may be because the CDC’s updated SSI risk adjustment for renal transplantation does not take into account patient-level comorbid conditions. Numerous studies have shown that comorbid conditions such as obesity and diabetes have been shown to increase risk for SSI.^16–18 Because the Elixhauser Index includes additional comorbidities such as obesity, this score may have greater predictive ability than the either the NHSN or the Charlson models and allows the exploration of additional comorbid conditions.^19,20

A common criticism of using ICD-9 codes in risk adjustment measures is that these codes represent patient comorbidities recorded at discharge. Often the condition under study, such as a HAI, occurs during the hospital stay. Because comorbid conditions are measured after the HAI has occurred, they may not be able to serve as predictors of the outcome. However, in our sample, more than 80% of SSIs developed after the patient was discharged from the hospital. Further, the conditions we found to be associated with SSI, such as obesity, peripheral vascular disease, and rheumatologic disorders, were likely preoperative conditions. Therefore, comorbid conditions recorded at discharge can be used as valid predictors of SSI.

The comorbid condition variables found to be statistically significant have biologic plausibility of being risk factors for SSI and have been demonstrated as risk factors in other studies. For example, previous studies have identified obesity as a risk factor for the development of SSI.^21,22 As well, patients with peripheral vascular disease likely have altered blood flow that could increase the risk of SSI.²³ Many patients with rheumatoid arthritis receive medications that affect the immune system and may place patients at higher infection risk.

The major limitations of our study are that it is a single-site study and it addresses a single surgical procedure. Further, the SSI rate for deep incisional of 7.9% found in our analysis is higher than the CDC NHSN estimate of 3.8% to 6.6% for patients undergoing renal transplantation between 2006 and 2008; therefore, the generalizability of our results may be limited.²⁴ Another limitation is that our c-statistics are low (close to 0.50) owing to similar patient case-mix. Patients who undergo renal transplantation often have similar comorbid conditions that lead them to a transplant. Even with this limitation, Merkow et al¹⁵ has demonstrated that the c-statistic is still a valid measure of discrimination in small sample sizes of homogenous patient populations. Though low c-statistics do not necessarily mean inferior risk adjustment, we also looked at calibration to assess model performance. However, we recommend that these indices be validated in samples with a more varied patient case-mix, different surgical procedures, and larger number of outcomes, and for different HAI outcomes, to assess their predictive utility across HAIs.

Future research and healthcare policy in risk adjustment should start incorporating comorbid conditions. As hospitals move from ICD-9 to ICD-10 and as electronic health records become more readily available, there are unique research opportunities and improvements in risk adjustment that can be explored and implemented.