Abstract
Background
The Centers for Medicare and Medicaid Services (CMS) use colon surgical site infection (SSI) rates to rank hospitals and apply financial penalties. The CMS’ risk-adjustment model omits potentially impactful variables that might disadvantage hospitals with complex surgical populations.
Methods
We analyzed adult patients who underwent colon surgery within facilities associated with HCA Healthcare from 2014 to 2016. SSIs were identified from National Health Safety Network (NHSN) reporting. We trained and validated 3 SSI prediction models, using (1) current CMS model variables, including hospital-specific random effects (HCA-adapted CMS model); (2) demographics and claims-based comorbidities (expanded-claims model); and (3) demographics, claims-based comorbidities, and NHSN variables (claims-plus–electronic health record [EHR] model). Discrimination, calibration, and resulting rankings were compared among all models and the current CMS model with published coefficient values.
Results
We identified 39 468 colon surgeries in 149 hospitals, resulting in 1216 (3.1%) SSIs. Compared to the HCA-adapted CMS model, the expanded-claims model had similar performance (c-statistic, 0.65 vs 0.67, respectively), while the claims-plus-EHR model was more accurate (c-statistic, 0.70; 95% confidence interval, .67–.73; P = .004). The sampling variation, due to the low surgical volume and small number of infections, contributed 74% of the total variation in observed SSI rates between hospitals. When CMS model rankings were compared to those from the expanded-claims and claims-plus-EHR models, 18 (15%) and 26 (22%) hospitals changed quartiles, respectively, and 10 (8.3%) and 12 (10%) hospitals changed into or out of the lowest-performing quartile, respectively.
Conclusions
An expanded set of variables improved colon SSI risk predictions and quartile assignments, but low procedure volumes and SSI events remain a barrier to effectively comparing hospitals.
Keywords: surgical site infection, risk adjustment, colon surgery
Using additional variables can improve risk adjustments of hospital-level colon surgical site infection rates, but have limited impact on financial penalties. Low procedure and infection rates contribute to much of the between-hospital variation.
Colon surgeries are common procedures in US hospitals and are associated with high surgical site infection (SSI) rates, compared to other procedure types, with estimated attributable costs exceeding $3 billion annually [1, 2]. Because of both the high volume of these procedures and the relatively high rate of infectious complications, the Centers for Medicare and Medicaid Services (CMS) requires acute-care hospitals to report colorectal SSI outcomes in order to receive full CMS reimbursement as part of the Inpatient Prospective Payment System [3, 4]. Hospital rankings based on colorectal SSI outcomes—specifically, those ranking in the lowest-performing quartile—are used along with other quality metrics to determine hospital reimbursements through both the CMS Hospital Acquired Conditions and Value-Based Purchasing pay-for-performance systems. In fiscal year 2015, the total amount of reimbursement at risk in the Hospital Acquired Conditions and Value-Based Purchasing programs was approximately $1.4 billion [5].
In order to account for differences in patient complexity and baseline SSI risk, hospital performance is risk-adjusted, and hospitals with less than 1 expected SSI are excluded from rankings. For colon surgery, CMS uses a risk adjustment model that includes age, gender, American Society of Anesthesiologists (ASA) score, diabetes, body mass index (BMI), primary closure, and oncology hospital (Table 1). This scoring omits several risk factors that have been associated with increased SSI risk; for example, tobacco use [6–10] and comorbidities such as renal disease [11], vascular disease [12], cirrhosis [13], and malignancy [13]. Electronically available comorbidities derived from International Classification of Disease (ICD) codes have been shown to improve risk adjustment and lead to significant changes in hospital rankings [14]. Additionally, concomitantly performed colon or other noncolon intraabdominal surgeries might affect the SSI risk [15], and are available to CMS via procedural billing codes.
Table 1.
Eligible Covariates for Each Risk Adjustment Model
| CMS | HCA-Adapted-CMS | Expanded Claims | Claims Plus EHR |
|---|---|---|---|
| Age | Age | Age | Age |
| ASA score | ASA score | ASA score | ASA score |
| Diabetes status | Diabetes status | Diabetes status | Diabetes status |
| Body mass index | Body mass index | Body mass index | Body mass index |
| Use of primary closure | Use of primary closure | Use of primary closure | Use of primary closure |
| Oncology hospital | Oncology hospital | Oncology hospital | Oncology hospital |
| Sex | Sex | Sex | Sex |
| … | … | Charlson/Elixhauser comorbiditiesa | Charlson/Elixhauser comorbiditiesa |
| … | … | Concomitant colon proceduresa | Concomitant colon proceduresa |
| … | … | Concomitant noncolon intraabdominal proceduresa | Concomitant noncolon intraabdominal proceduresa |
| … | … | … | Use of anesthesia b |
| … | … | … | Procedure durationb |
| … | … | … | Laparoscopic techniqueb |
| … | … | … | Medical school affiliationb |
| … | … | … | Hospital size ≥/<500 bedsb |
| … | … | … | Wound classb |
| … | … | … | Emergency surgeryb |
| … | … | … | Smoking status (current, former, or never) |
| … | … | … | Race |
Abbreviations: ASA, American Society of Anesthesiologists; CMS, Centers for Medicare and Medicaid Services; EHR, electronic health record; HCA, HCA Healthcare.
aData obtained from diagnostic or procedural claims codes.
bData obtained from the National Healthcare Safety Network.
Less than optimal risk adjustment affects the ability to use SSI outcomes to identify opportunities for improvement or to meaningfully compare SSI performance between health-care facilities. By omitting recognized risk factors for SSI, CMS’ adjustment model might be unfairly weighted against hospitals that perform colorectal surgery on more complex patients with multiple comorbidities. We hypothesized that better risk adjustment models could be made using additional data relevant to the SSI risk, including both claims-based and other patient-level clinical data that are available in electronic health records (EHRs). We set out to determine whether an expanded set of preoperative data could help build more effective risk adjustment models that yield significantly different hospital rankings than those derived from the current CMS model, especially with respect to the lowest-performing quartile.
METHODS
We performed a retrospective analysis of adult patients who underwent colon surgery within facilities affiliated with HCA Healthcare (HCA) in 21 US states from January 2014 through December 2016. The median HCA hospital size is 260 beds (range 32–1013). Only the first eligible episode of colon surgery for each individual was included.
We used data that HCA-affiliated hospitals had submitted to the Centers for Disease Control and Prevention National Healthcare Safety Network to identify patients who underwent colon surgery. We also obtained patient- and hospital-level data from both the NHSN submissions and from HCA’s central data repository. The full set of variables is shown in Table 1. Deep incisional or organ/space colon SSIs were determined by each hospital’s infection prevention staff using NHSN criteria [16]. Each HCA facility validates NHSN data twice per year (3 cases), and CMS validation is performed at 30 facilities per year. Superficial incisional SSIs were excluded from the analyses, as these are not included in the CMS SSI outcome measures [17]. All SSIs present at the time of surgery were also excluded. Using ICD-9 and ICD-10 procedure codes (Supplementary Appendix 1), we identified instances of secondary colon surgery or other noncolon intraabdominal surgery that occurred on the same date as the index colon procedure for use as covariates, but did not count them as separate procedures. The Harvard Pilgrim Healthcare Institutional Review Board approved this study.
We computed Charlson and Elixhauser comorbidities using diagnosis codes, according to the methods of Quan et al [18]. Baseline demographics and clinical variables were compared between patients with and without SSI using mixed effects models. The sample was randomly split into a training set (2/3) and a validation set (1/3) within each hospital. We used mixed-effects logistic models to assess the SSI risk attributable to each patient-level variable, with a random intercept for facility to account for the clustering effect at the hospital level. We used a backwards selection approach with a threshold of P < .05 for elimination. Using the training set, we built an HCA-adapted CMS model using the variables in the existing CMS model; an expanded-claims model using only variables selected from a set including basic demographics (age, gender, race, ASA score) and claims-based data (comorbidity scores and concomitant colon or other noncolon intraabdominal surgeries); and a claims-plus-EHR model using all available NHSN patient- and hospital-level data, in addition to demographics and claims-based data. For the claims-plus-EHR model, the laparoscopic variable was included by default [19, 20]. For comparison of the resulting hospital rankings, we also used the CMS risk-adjustment model with published coefficient values for each variable [17]. This differed from the HCA-adapted CMS model, which had coefficients built from our training set and used random effects.
In the model-building stage, we included random effects to account for clustering within hospitals in order to compare model performance. Without including the random effects, in general we would not expect the predicted risks to agree with the observed risk, unless there was no heterogeneity across hospitals (ie, no clustering within a hospital). Patient-level predicted probabilities of SSI were generated from each model in validation data sets. We calculated c-statistics with 95% confidence intervals (CIs) and calibration plots for each model [21]. Hospitals’ predicted and observed SSI rates were graphed by the hospital procedure volume to assess differences in prediction accuracy for large versus small hospitals.
For ranking purposes, we calculated the expected individual-level SSI risk from each model using the validation data set. The random effects were not included, in order to remove the influence of hospital characteristics so that the resulting value represented the expected risk for an individual in the same reference hospital after adjusting only for individual characteristics. Based on the expected individual SSI risk, we estimated the expected number of SSI events and created standardized infection ratios for each hospital by dividing the observed number of SSI events by the expected number estimated from each model. We ranked hospitals according to the standardized infection ratios calculated from the different models and assessed the differences in ordinal ranks and by quartile in comparison to the CMS model [14, 22]. Hospitals were excluded if they had <1 expected SSI. Bubble plots were made to show changes in quartile ranks between models. Finally, we decomposed the total variability in observed SSI rates across hospitals into the between-hospital variability and the within-hospital sampling variability, and estimated the coefficient of variation in SSI rates across hospitals using the methods of Hayes and Moulton [23]. Analyses were performed using SAS version 9.3.
RESULTS
We identified 39 468 instances of eligible colon surgery at 149 hospitals. In total, 1216 (3.1%) subjects developed a deep incisional or organ/space SSI. Subjects with SSIs differed significantly from those without for many attributes; those with an SSI were slightly younger, had higher ASA scores, were less likely to have primary closure, were less likely to undergo laparoscopic procedures, had higher BMIs, had longer procedure durations, were less likely to be in hospitals with medical school affiliations, were more likely to have trauma, were more likely to have an emergency procedure, were more likely to have malnutrition, were more likely to have contaminated or dirty wounds, were more likely to have a concomitant noncolon intraabdominal procedure, and had higher Elixhauser and Charlson comorbidity scores. No significant differences were observed for race, gender, smoking status, concomitant colon surgery, hospital bed size, or several other comorbidities (Table 2).
Table 2.
Baseline Demographics by Surgical Site Infection Status
| Total, N = 39 468 | Subjects without SSI, n = 38 252 | Subjects with SSI, n = 1216 | P Value | |
|---|---|---|---|---|
| Mean age (SD) | 62.7 (15.31) | 62.7 (15.31) | 60.9 (15.33) | <.0001 |
| Race | … | … | … | .0789 |
| Asian | 741 (1.9%) | 723 (1.9%) | 18 (1.5%) | |
| Black | 4264 (10.8%) | 4115 (10.8%) | 149 (12.3%) | |
| Hispanic | 694 (1.8%) | 672 (1.8%) | 22 (1.8%) | |
| Other | 3025 (7.7%) | 2933 (7.7%) | 92 (7.6%) | |
| White | 30 746 (77.9%) | 29 809 (77.9%) | 935 (76.9%) | |
| Female gender | 21 345 (54.1%) | 20 707 (54.1%) | 638 (52.5%) | .3606 |
| ASA score | … | … | … | .0008 |
| 1 | 836 (2.1%) | 816 (2.1%) | 20 (1.6%) | |
| 2 | 12 643 (32%) | 12 302 (32.2%) | 341 (28%) | |
| 3 | 20 118 (51%) | 19 479 (50.9%) | 639 (52.5%) | |
| 4 | 5487 (13.9%) | 5278 (13.8%) | 209 (17.2%) | |
| 5 | 384 (1%) | 377 (1%) | 7 (0.6%) | |
| Use of anesthesia | 38 953 (98.7%) | 37 740 (98.7%) | 1213 (99.8%) | .1145 |
| Primary closure | 39 083 (99%) | 37 898 (99.1%) | 1185 (97.5%) | <.0001 |
| Laparosopic technique | 14 233 (36.1%) | 13 883 (36.3%) | 350 (28.8%) | <.0001 |
| Mean BMI (SD) | 28.6 (6.69) | 28.5 (6.67) | 30.2 (7.22) | <.0001 |
| Mean procedure length, minutes (SD) | 137.3 (72.19) | 136.6 (71.63) | 160.3 (84.83) | <.0001 |
| Medical school affiliation | 16 499 (41.8%) | 16 041 (41.9%) | 458 (37.7%) | .0532 |
| Trauma | 171 (0.4%) | 141 (0.4%) | 30 (2.5%) | <.0001 |
| Emergency surgery | 5299 (13.4%) | 5096 (13.3%) | 203 (16.7%) | .0012 |
| Hospital bed size ≥500 | 7415 (18.8%) | 7178 (18.8%) | 237 (19.5%) | .5836 |
| Malnutrition | 3579 (9.1%) | 3364 (8.8%) | 215 (17.7%) | <.0001 |
| Wound class | … | … | … | <.0001 |
| Clean-contaminated | 31 469 (79.7%) | 30 624 (80.1%) | 845 (69.5%) | |
| Contaminated | 4707 (11.9%) | 4523 (11.8%) | 184 (15.1%) | |
| Dirty | 3293 (8.3%) | 3106 (8.1%) | 187 (15.4%) | |
| Tobacco use category | … | … | … | .1433 |
| Current smoker | 6488 (16.4%) | 6274 (16.4%) | 214 (17.6%) | |
| Former smoker | 11 813 (29.9%) | 11 441 (29.9%) | 372 (30.6%) | |
| Never smoker | 18 248 (46.2%) | 17 710 (46.3%) | 538 (44.2%) | |
| Missing | 2921 (7.4%) | 2829 (7.4%) | 92 (7.6%) | |
| ≥1 concomitant colon procedure | 20 639 (52.3%) | 19 998 (52.3%) | 641 (52.7%) | .9034 |
| ≥1 concomitant intraabdominal procedure | 15 728 (39.9%) | 15 135 (39.6%) | 593 (48.8%) | <.0001 |
| Elixhauser components | … | … | … | |
| Congestive heart failure | 3043 (7.7%) | 2921 (7.6%) | 122 (10%) | .0026 |
| Cardiac arrhythmia | 7590 (19.2%) | 7256 (19%) | 334 (27.5%) | <.0001 |
| Cardiac valve disease | 1085 (2.7%) | 1049 (2.7%) | 36 (3%) | .5811 |
| Pulmonary vascular disease | 832 (2.1%) | 787 (2.1%) | 45 (3.7%) | .0002 |
| Peripheral vascular disease | 2820 (7.1%) | 2695 (7%) | 125 (10.3%) | <.0001 |
| Hypertension | 18 052 (45.7%) | 17 506 (45.8%) | 546 (44.9%) | .7705 |
| Complicated hypertension | 3981 (10.1%) | 3824 (10%) | 157 (12.9%) | .0006 |
| Paralysis | 709 (1.8%) | 698 (1.8%) | 11 (0.9%) | .0108 |
| Neurological disease | 2186 (5.5%) | 2070 (5.4%) | 116 (9.5%) | <.0001 |
| Chronic pulmonary disease | 6707 (17%) | 6457 (16.9%) | 250 (20.6%) | .0003 |
| Diabetes | 6196 (15.7%) | 5975 (15.6%) | 221 (18.2%) | .0129 |
| Complicated diabetes | 1571 (4%) | 1507 (3.9%) | 64 (5.3%) | .0194 |
| Hypothyroidism | 4912 (12.4%) | 4752 (12.4%) | 160 (13.2%) | .3785 |
| Renal disease | 4070 (10.3%) | 3917 (10.2%) | 153 (12.6%) | .0062 |
| Liver disease | 1464 (3.7%) | 1400 (3.7%) | 64 (5.3%) | .0038 |
| Peptic ulcer disease | 476 (1.2%) | 448 (1.2%) | 28 (2.3%) | .0005 |
| AIDS | 59 (0.1%) | 58 (0.2%) | 1 (0.1%) | .5807 |
| Lymphoma | 304 (0.8%) | 294 (0.8%) | 10 (0.8%) | .7802 |
| Metastatic cancer | 3829 (9.7%) | 3675 (9.6%) | 154 (12.7%) | .0019 |
| Solid tumor | 13 318 (33.7%) | 12 891 (33.7%) | 427 (35.1%) | .6026 |
| Rheumatoid arthritis | 1033 (2.6%) | 994 (2.6%) | 39 (3.2%) | .1674 |
| Coagulopathy | 2262 (5.7%) | 2131 (5.6%) | 131 (10.8%) | <.0001 |
| Obesity | 5176 (13.1%) | 4932 (12.9%) | 244 (20.1%) | <.0001 |
| Weight loss | 5060 (12.8%) | 4757 (12.4%) | 303 (24.9%) | <.0001 |
| Fluid and electrolyte disorder | 7385 (18.7%) | 7058 (18.5%) | 327 (26.9%) | <.0001 |
| Blood loss anemia | 1105 (2.8%) | 1059 (2.8%) | 46 (3.8%) | .0369 |
| Iron deficiency anemia | 2028 (5.1%) | 1958 (5.1%) | 70 (5.8%) | .3658 |
| Alcohol abuse | 1533 (3.9%) | 1463 (3.8%) | 70 (5.8%) | .0004 |
| Illicit drug abuse | 679 (1.7%) | 647 (1.7%) | 32 (2.6%) | .0102 |
| Psychosis | 248 (0.6%) | 236 (0.6%) | 12 (1%) | .1532 |
| Depression | 4405 (11.2%) | 4225 (11%) | 180 (14.8%) | <.0001 |
| Median Elixhauser score (IQR) | 3.0 (1.0–4.0) | 2.0 (1.0–4.0) | 3.0 (2.0–5.0) | <.0001 |
| Charlson components | ||||
| Myocardial infarction | 1087 (2.8%) | 1057 (2.8%) | 30 (2.5%) | .6107 |
| Coronary heart disease | 2905 (7.4%) | 2791 (7.3%) | 114 (9.4%) | .0076 |
| Cerebrovascular disease | 842 (2.1%) | 811 (2.1%) | 31 (2.5%) | .3650 |
| Dementia | 629 (1.6%) | 613 (1.6%) | 16 (1.3%) | .4165 |
| Chronic pulmonary disease | 6532 (16.5%) | 6294 (16.5%) | 238 (19.6%) | .0018 |
| Rheumatologic disease | 928 (2.4%) | 896 (2.3%) | 32 (2.6%) | .4707 |
| Liver disease | 613 (1.6%) | 585 (1.5%) | 28 (2.3%) | .0248 |
| Diabetes | 7082 (17.9%) | 6863 (17.9%) | 219 (18%) | .8639 |
| Complicated diabetes | 876 (2.2%) | 846 (2.2%) | 30 (2.5%) | .5363 |
| Paraplegia | 556 (1.4%) | 547 (1.4%) | 9 (0.7%) | .0299 |
| Renal disease | 4059 (10.3%) | 3906 (10.2%) | 153 (12.6%) | .0056 |
| Malignancy | 4095 (10.4%) | 3931 (10.3%) | 164 (13.5%) | .0014 |
| Severe liver disease | 178 (0.5%) | 169 (0.4%) | 9 (0.7%) | .1561 |
| Metastatic cancer | 3829 (9.7%) | 3675 (9.6%) | 154 (12.7%) | .0017 |
| HIV/AIDS | 59 (0.1%) | 58 (0.2%) | 1 (0.1%) | .5807 |
| Peripheral vascular disease | 1111 (2.8%) | 1061 (2.8%) | 50 (4.1%) | .0047 |
| Median Charlson score (IQR) | 1.0 (0–3.0) | 1.0 (0–3.0) | 2.0 (0–3.0) | <.0001 |
The P values reflect the comparisons of subjects with and without SSI.
Abbreviations: ASA, American Society of Anesthesiologists; BMI, body mass index; HIV, human immunodeficiency virus; IQR, interquartile range; SD, standard deviation; SSI, surgical site infection.
The analysis of the training data set yielded an expanded-claims multivariable model that included age, ASA score, presence of concomitant colon procedure(s), total Elixhauser score, and the Elixhauser components of cardiac arrhythmia, paralysis, neurological disorder, coagulopathy, obesity, weight loss, and fluid/electrolyte disorder (Table 3). Of these, only age and ASA score are present in the CMS model. The claims-plus-EHR model included laparoscopy (by default), age, ASA score, total Elixhauser score, and the Elixhauser components of cardiac arrhythmia, paralysis, neurological disorder, and weight loss, as well as wound class, procedure length, BMI, and use of primary closure. The c-statistic for the CMS model was 0.59 (95% CI .56–.61) in the validation set. A significant improvement in the prediction accuracy was observed using the HCA-adapted CMS model, with a c-statistic of 0.65 (95% CI, .62–.69; P = .001 for comparison). The expanded-claims model yielded similar accuracy as the HCA-adapted CMS model, with a c-statistic of 0.67 (95% CI, .64–.71; P = .15). The claims-plus-EHR model further improved the prediction accuracy, as compared to the HCA-adapted CMS model, with a c-statistic of 0.70 (95% CI, .67–.73; P = .004).
Table 3.
Associations Between Variables and Surgical Site Infection in Each Prediction Model
| Odds Ratio (95% CI) | C-statistic (95% CI) from Validation Data Set | ||
|---|---|---|---|
| CMS model, fixed coefficients | … | .59 (.56–.61) | |
| HCA-adapted CMS model | .65 (.62–.69) | ||
| ASA scorea | 1 | .86 (.51–1.45) | |
| 2 | .80 (.67–.94) | ||
| 3/4/5, reference | … | ||
| Diabetes | 1.08 (.89–1.31) | ||
| Male gender | 1.01 (.87–1.16) | ||
| Age / 10 | .93 (.88–.97) | ||
| BMI ≥0 | 1.45 (1.25–1.68) | ||
| Non-primary closure | .37 (.23–.59) | ||
| Expanded-claims model | .67 (.64–.71) | ||
| ASA score | 1 | 1.09 (.65–1.85) | |
| 2, reference | … | ||
| 3 | 1.01 (.84–1.21) | ||
| 4 | .77 (.60–1.00) | ||
| 5 | .24 (.09–.67) | ||
| Age | .99 (.98–.99) | ||
| Cardiac arrhythmia | 1.59 (1.34–1.90) | ||
| Paralysis | .22 (.09–.53) | ||
| Neurological disease | 1.42 (1.09–1.84) | ||
| Coagulopathy | 1.74 (1.37–2.21) | ||
| Obesity | 1.81 (1.51–2.17) | ||
| Weight loss | 2.27 (1.89–2.72) | ||
| Fluid and electrolyte disorder | 1.34 (1.13–1.59) | ||
| Depression | 1.28 (1.05–1.60) | ||
| Multiple colon procedures on the index date | 1.30 (1.13–1.51) | ||
| Claims-plus-EHR Model | .70 (.67–.73) | ||
| ASA score | 1 | 1.20 (.71–2.03) | |
| 2, reference | … | ||
| 3 | .90 (.75–1.08) | ||
| 4 | .67 (.51–.88) | ||
| 5 | .22 (.08– .60) | ||
| Wound class | CC, reference | … | |
| CO | 1.48 (1.20–1.83) | ||
| D | 1.77 (1.41–2.22) | ||
| Age | .99 (.99–1.00) | ||
| Cardiac arrhythmia | 1.28 (1.06–1.54) | ||
| Coronary heart disease | .67 (.50–.90) | ||
| Diabetes, Charlson | .69 (.56–.85) | ||
| Complicated diabetes, Elixhauser | .55 (.32–.95) | ||
| Hypertension | .80 (.67–.96) | ||
| Paralysis | .23 (.09–.57) | ||
| Weight loss | 1.78 (1.46–2.18) | ||
| Hypothyroidism | .76 (.60–.96) | ||
| Renal disease, Elixhauser | .72 (.55–.96) | ||
| Total Elixhauser score | 1.24 (1.18–1.30) | ||
| Laparoscopy | .85 (.72–1.01) | ||
| Trauma | 3.76 (1.99–7.11) | ||
| Primary closure | .59 (.35–.98) | ||
| BMI, continuous | 1.03 (1.02–1.04) | ||
| Procedure length, minutes | 1.00 (1.00–1.01) |
Abbreviations: ASA, American Society of Anesthesiologists; BMI, body mass index; CC, clean-contaminated; CI, confidence interval; CMS, Centers for Medicare and Medicaid Services; CO, contaminated; D, dirty; EHR, electronic health record; HCA, HCA Healthcare.
aThe ASA score is treated as a continuous variable with a value of 1, 2, or 3/4/5 in the CMS model.
For comparisons of ranking, 29 hospitals were excluded for having 0 observed SSIs and <1 expected, based on the claims-plus-EHR model, leaving 120 hospitals. When rankings from the CMS model were compared to those from the expanded-claims and the claims-plus-EHR models, 89 (74%) and 93 (78%) of 120 hospitals had a change in rank, respectively; the median absolute changes were 3 and 3.5 ranks, respectively, and the number of hospitals that changed quartiles were 18 (15%) and 26 (22%), respectively. There were 10 (8.3%) and 12 (10%) hospitals, respectively, that were reclassified into or out of the lowest-performing quartile (Figure 1). Comparing the expanded-claims and the claims-plus-EHR models, 10 (8.3%) hospitals were reclassified into or out of the lowest-performing quartile.
Figure 1.
A–D, Comparison of hospital rankings, in quartiles, between the CMS model, HCA-adapted CMS model, expanded claims model, and claims-plus-EHR model. The number in each circle represents the number of hospitals that fall into the quartiles indicated on each axis, with “1” being the best-performing quartile and “4” being the worst-performing quartile. Each axis represents the resulting hospital rankings of 1 of the 4 modeling approaches used in this study. Abbreviations: CMS, Centers for Medicare and Medicaid Services; EHR, electronic health record; HCA, HCA Healthcare.
Graphing of the hospitals’ predicted and observed SSI rates demonstrated modest agreement using the HCA-adapted CMS, expanded-claims, and claims-plus-EHR models (Figure 2; Supplementary Appendix 2). In our sample, the median total surgical volume for the entire 3-year period was 207 and the median number of SSIs was only 5. The total procedure volume was 101 or less for 25% of the hospitals. The sampling variation, due to a low surgical volume and a small number of infections, contributed to 74% of the total variation in observed SSI rates across hospitals. The heterogeneity in SSI rates across hospitals was substantial: the coefficient of variation was estimated to be 47%, which corresponds to an intraclass correlation of 26%.
Figure 2.
A–F, Predicted versus observed SSI rates, stratified by median procedure volume (validation data set). Abbreviations: CMS, Centers for Medicare and Medicaid Services; EHR, electronic health record; HCA, HCA Healthcare; SSI, surgical site infection.
DISCUSSION
Optimizing the risk adjustment is essential for meaningful interhospital comparisons and appropriately assigning financial penalties. The current CMS risk adjustment model for colon surgery omits several important variables that proved to be significant risk factors in our models. The addition of claims data and clinical data that hospitals already submit to NHSN improved the discrimination of the model using variables currently employed by CMS from a c-statistic of 0.65 (95% CI, .62–.70) to 0.70 (95% CI, .67–.73). This confirms the importance of previously identified SSI risk factors included in our models. This improved model assigned 22% of hospitals to a different quartile than did the CMS model, including 10% of hospitals that moved into or out of the lowest-performing quartile.
The agreements between the predicted SSI rates based on these models and the observed SSI rates were moderate for all models, as shown in Figure 2. Low procedure volumes and the small number of predicted events in individual hospitals are likely to be major limiting factors. Even though our sample included 3 years of data for each hospital, the median number of procedures included in our study was 207 (69 per year) and the median number of colon SSIs was only 5 (1.67 per year). The low surgical volume and small number of infections led to a large amount of uncertainty in estimating hospital-specific SSI rates. The heterogeneity in SSI rates across hospitals was substantial. While adjustment for patient, surgeon, and hospital characteristics (eg, surgeon’s personal procedure volume or surgical process measures) might be useful, the variation resulting from low procedure volumes and event rates will remain a serious barrier to effectively characterizing hospitals’ “true” SSI rates.
The improved performance of the HCA-adapted CMS model versus the CMS model suggests that coefficients obtained from a different patient population may have low portability. This also highlights the disadvantage of using a “1-size-fits-all” national model, as different populations may be better risk-adjusted using different variables. Additionally, the enhanced prediction demonstrated by the expanded-claims model versus the CMS model cannot be attributed to better variable selection, but instead is due to training this model in our specific population.
Our findings come with certain caveats. Our models were trained using data from a large network of community hospitals; therefore, our findings may not be applicable to populations at traditional academic centers or specialty hospitals. However, these results are likely to apply to a large proportion of US hospitals. As our primary goal was to build models for prediction, the calculated coefficients used for adjustment in our models do not have a causal interpretation. SSI diagnoses were based on NHSN reporting by hospital-based infection prevention staff, and SSI detection methods may vary by hospital. We did not exclude patients who died within the 30-day postoperative period; this is consistent with the limitations of current surveillance practices at CMS, and a possible explanation for the lower SSI rates seen with increasing ASA scores in the expanded-claims and claims-plus-EHR models.
This study’s strengths include a large sample of individual patients and SSI outcomes, as well as a geographically diverse network of hospitals that represent a wide range of facility sizes and care environments.
CONCLUSIONS
An expanded set of claims-based comorbidities and patient-level clinical data improved SSI risk prediction, compared to current CMS methods, in a population of adult colon surgery patients. This improved predictive ability changed hospital rankings for the majority of hospitals, and resulted in a change in quartile for 22%, including 10% that moved into or out of the lowest-performing quartile. However, all adjusted ranking systems were of limited value, because of low hospital-specific procedure volumes and the small number of predicted infections. Because SSI following colon surgery is a relatively rare outcome, coupled with low procedure volumes, we believe that current ranking systems and readily foreseeable improvements will be largely incapable of distinguishing hospitals based on their SSI rates. This limitation may justify returning to assessing compliance with evidence-based process measures; for example, appropriate selection and timing of the administration of perioperative antibiotic prophylaxis.
Supplementary Data
Supplementary materials are available at Clinical Infectious Diseases online. Consisting of data provided by the authors to benefit the reader, the posted materials are not copyedited and are the sole responsibility of the authors, so questions or comments should be addressed to the corresponding author.
Notes
Acknowledgments. The authors thank Dr Jon Perlin, President of the Clinical Services Group and Chief Medical Officer at HCA Healthcare, for his support of the study, and Julia Moody, Clinical Director of Infection Prevention at HCA Healthcare, for her assistance in data retrieval.
Disclaimer. The views expressed in this publication represent those of the author(s) and do not necessarily represent the official views of HCA Healthcare or any of its affiliated entities.
Financial support. This work was supported by the Harvard Pilgrim Health Care Institute.
Potential conflicts of interest. The authors: No reported conflicts of interest. All authors have submitted the ICMJE Form for Disclosure of Potential Conflicts of Interest.
References
- 1. Zimlichman E, Henderson D, Tamir O, et al. . Health care-associated infections: a meta-analysis of costs and financial impact on the US health care system. JAMA Intern Med 2013; 173:2039–46. [DOI] [PubMed] [Google Scholar]
- 2. Kirkland KB, Briggs JP, Trivette SL, Wilkinson WE, Sexton DJ. The impact of surgical-site infections in the 1990s: attributable mortality, excess length of hospitalization, and extra costs. Infect Control Hosp Epidemiol 1999; 20:725–30. [DOI] [PubMed] [Google Scholar]
- 3. Centers for Medicare and Medicaid Services, Dept of Health and Human Services. Medicare program; hospital inpatient prospective payment systems for acute care hospitals and the long-term care hospital prospective payment system and policy changes and fiscal year 2017 rates; quality reporting requirements for specific providers; graduate medical education; hospital notification procedures applicable to beneficiaries receiving observation services; technical changes relating to costs to organizations and Medicare cost reports; finalization of interim final rules with comment period on LTCH PPS payments for severe wounds, modifications of limitations on redesignation by the Medicare Geographic Classification Review Board, and extensions of payments to MDHs and low-volume hospitals. Final rule. Fed Regist 2016; 81:56761–7345. [PubMed] [Google Scholar]
- 4. Blumenthal D, Jena AB. Hospital value-based purchasing. J Hosp Med 2013; 8:271–7. [DOI] [PubMed] [Google Scholar]
- 5. CMS. CMS to improve quality of care during hospital inpatient stays Available at https://www.cms.gov/newsroom/fact-sheets/cms-improve-quality-care-during-hospital-inpatient-stays. Accessed 22 December 2016.
- 6. Augenstein VCP, Wormer B, Lincourt A, Heniford BT. CeDAR: Carolinas equation for determining associated risks. J Am Coll Surg 2015; 221:565–6. [Google Scholar]
- 7. Bakkum-Gamez JN, Dowdy SC, Borah BJ, et al. . Predictors and costs of surgical site infections in patients with endometrial cancer. Gynecol Oncol 2013; 130:100–6. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8. Kaoutzanis C, Leichtle SW, Mouawad NJ, et al. . Risk factors for postoperative wound infections and prolonged hospitalization after ventral/incisional hernia repair. Hernia 2015; 19:113–23. [DOI] [PubMed] [Google Scholar]
- 9. Kurz A, Sessler DI, Lenhardt R. Perioperative normothermia to reduce the incidence of surgical-wound infection and shorten hospitalization. Study of wound infection and temperature group. N Engl J Med 1996; 334:1209–15. [DOI] [PubMed] [Google Scholar]
- 10. Moreno Elola-Olaso A, Davenport DL, Hundley JC, Daily MF, Gedaly R. Predictors of surgical site infection after liver resection: a multicentre analysis using national surgical quality improvement program data. HPB (Oxford) 2012; 14:136–41. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11. Daley J, Khuri SF, Henderson W, et al. . Risk adjustment of the postoperative morbidity rate for the comparative assessment of the quality of surgical care: results of the National Veterans Affairs Surgical Risk Study. J Am Coll Surg 1997; 185:328–40. [PubMed] [Google Scholar]
- 12. Karamanos E, Kandagatla P, Watson J, Schmoekel N, Siddiqui A. Development and validation of a scoring system to predict surgical site infection after ventral hernia repair: a Michigan surgical quality collaborative study. World J Surg 2017; 41:914–8. [DOI] [PubMed] [Google Scholar]
- 13. Darouiche RO, Wall MJ Jr, Itani KM, et al. . Chlorhexidine-alcohol versus povidone-iodine for surgical-site antisepsis. N Engl J Med 2010; 362:18–26. [DOI] [PubMed] [Google Scholar]
- 14. Jackson SS, Leekha S, Magder LS, et al. . Electronically available comorbidities should be used in surgical site infection risk adjustment. Clin Infect Dis 2017; 65:803–10. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15. Calderwood MS, Kleinman K, Murphy MV, Platt R, Huang SS. Not all colon procedures are equal: implication for risk adjustment in publicly reported SSI rates. In: Program and abstracts of the Society of Healthcare Epidemiology of America (SHEA) Annual Spring Meeting (Atlanta, GA) 2016. Abstract number 598. [Google Scholar]
- 16. Centers for Disease Control and Prevention. Surgical site infection (SSI) event [procedure-associated module]. 2019. Available at https://www.cdc.gov/nhsn/pdfs/pscmanual/9pscssicurrent.pdf. Accessed 1 April 2019. [Google Scholar]
- 17. Centers for Disease Control and Prevention. Your guide to the standardized infection ratio (SIR). NHSN e-News: SIRs special edition. Atlanta, GA: Centers for Disease Control and Prevention, 2010. [Google Scholar]
- 18. Quan H, Sundararajan V, Halfon P, et al. . Coding algorithms for defining comorbidities in ICD-9-CM and ICD-10 administrative data. Med Care 2005; 43:1130–9. [DOI] [PubMed] [Google Scholar]
- 19. Howard DP, Datta G, Cunnick G, Gatzen C, Huang A. Surgical site infection rate is lower in laparoscopic than open colorectal surgery. Colorectal Dis 2010; 12:423–7. [DOI] [PubMed] [Google Scholar]
- 20. Kiran RP, El-Gazzaz GH, Vogel JD, Remzi FH. Laparoscopic approach significantly reduces surgical site infections after colorectal surgery: data from national surgical quality improvement program. J Am Coll Surg 2010; 211:232–8. [DOI] [PubMed] [Google Scholar]
- 21. Crowson CS, Atkinson EJ, Therneau TM. Assessing calibration of prognostic risk scores. Stat Methods Med Res 2016; 25:1692–706. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22. Jackson SS, Leekha S, Magder LS, et al. . The effect of adding comorbidities to current centers for disease control and prevention central-line-associated bloodstream infection risk-adjustment methodology. Infect Control Hosp Epidemiol 2017; 38:1019–24. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23. Hayes RJ, Moulton LH.. Cluster randomised trials. 2nd ed. New York: CRC Press, 2017. [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.