Comparison of Comorbidity Scores in Predicting Surgical Outcomes

Abstract

Introduction

The optimal methodology for assessing comorbidity to predict various surgical outcomes such as mortality, readmissions, complications and failure to rescue (FTR) using claims data has not been established.

Objective

Compare diagnosis- and prescription-based comorbidity scores for predicting surgical outcomes.

Methods

We used 100% Texas Medicare data (2006–2011) and included patients undergoing coronary artery bypass grafting (CABG), pulmonary lobectomy, endovascular repair of abdominal aortic aneurysm, open repair of abdominal aortic aneurysm, colectomy, and hip replacement (N=39,616). The ability of diagnosis-based (Charlson comorbidity score, Elixhauser comorbidity score, Combined Comorbidity Score, Centers for Medicare & Medicaid Services-Hierarchical Condition Categories [CMS-HCC]) vs. prescription-based chronic disease (CDS) score in predicting 30-day mortality, 1-year mortality, 30-day readmission, complications, and FTR were compared using c-statistics (c) and integrated discrimination improvement (IDI).

Results

The overall 30-day mortality was 5.8%, 1-year mortality was 17.7%, 30-day readmission was 14.1%, complication rate was 39.7%, and FTR was 14.5%. CMS-HCC performed the best in predicting surgical outcomes (30-day mortality, c=0.791, IDI=4.59%; 1-year mortality, c=0.798, IDI=9.60%; 30-day readmission, c=0.630, IDI=1.27%; complications, c=0.766, IDI=9.37%; FTR, c=0.811, IDI=5.24%) followed by Elixhauser comorbidity index/disease categories (30-day mortality, c=0.750, IDI=2.37%; 1-year mortality, c=0.755, IDI=5.82%; 30-day readmission, c=0.629, IDI=1.43%; complications, c=0.730, IDI=3.99%; FTR, c=0.749, IDI=2.17%). Addition of prescription-based scores to diagnosis-based scores did not improve performance.

Conclusions

The CMS-HCC had superior performance in predicting surgical outcomes. Prescription-based scores, alone or in addition to diagnosis-based scores, were not better than any diagnosis-based scoring system.

INTRODUCTION

In the hierarchy of study designs, randomized controlled trials (RCTs) are considered the gold standard because they can establish causal effect between the treatment and the outcome. However, surgical treatment decisions are half as likely to be based on RCTs compared to medical treatment decisions due to several unique aspects of surgery. (1) It is often challenging, or sometimes unethical, to randomize patients to surgical treatment or no treatment. Likewise, it is difficult to blind patients to receipt of surgery. In addition, the surgical learning curve and the complex process of coordinating care across disciplines and individuals make RCTs in surgery challenging. (2–4) As such, the use of observational studies to evaluate surgical outcomes research is increasing. (5)

Observational studies include the challenge of controlling for patient risk factors. (6) Patient selection based on comorbidity can significantly bias estimates of the treatment effect. As such, it is critical to control for comorbidity. The most commonly used diagnosis-based comorbidity scores include the Charlson comorbidity score (CCS) and the Elixhauser comorbidity score. (7, 8) The Combined Comorbidity Score, developed by combining disease conditions from Charlson and Elixhauser, was superior to either score alone in predicting mortality in Medicare patients. (9) The Centers for Medicare and Medicaid Services (CMS) have also developed a risk adjustment model, the CMS Hierarchical Condition Categories (CMS-HCC), to adjust capitation payments to health plans. (10) Prescription-based comorbidity scores include the chronic disease score (CDS). (11)

Previous studies have compared the ability of different claim-based methodologies to evaluate patient comorbidities. (12) However, no study has established the optimal methodology using claims data for assessing comorbidity in order to predict various surgical outcomes such as mortality, readmissions, complications and failure to rescue (FTR). The goal of our study was twofold: (i) to conduct a review of studies in surgery to describe the use of comorbidity sores, and (ii) to compare the performance of commonly used comorbidity scores in predicting mortality, readmission rates, complications and FTR in surgical patients.

METHODS

Use of Comorbidity Scores in Surgical Studies Using Medicare Claims Data

We performed a review of surgical studies to describe how comorbidity scores were used to control for confounding in Medicare claims data. In order to discuss results of our study in the context of Medicare data, we limited the literature review to Medicare data only. We selected articles from PubMed using following search criteria: “surgery” OR “surgical” AND “SEER” OR “Medicare”. All articles from 2012 onwards published in English were kept for further review (N=1,723). We included articles in the final review that met the following criteria: (i) the study was an original research article using Medicare data, (ii) the study included surgical patients or compared surgical interventions and (iii) the study controlled for comorbidities in multivariable models. Full text articles were reviewed and summarized for the use of comorbidity scores.

Assessment of the Performance of Comorbidity Scores

Data Source

In this retrospective study, we used 100% Texas Medicare claims data from 2006 to 2011. Medicare claims data collects patients’ demographic and enrollment information, outpatient visits, and inpatient visits. Specifically, we used the Medicare Provider Analysis and Review (MEDPAR) file, the outpatient file, the carrier file and the denominator file.

Study sample

The study sample included patients from 2007 to 2010 who underwent a primary surgical procedure for one of the following: coronary-artery bypass grafting (CABG), pulmonary lobectomy, endovascular repair of abdominal aortic aneurysm, open repair of abdominal aortic aneurysm, colectomy or hip replacement. (Appendix 1) We selected these procedures because they are common for Medicare beneficiaries and they reflect several surgical subspecialties (cardiac, thoracic, vascular, colorectal and orthopedic), enhancing the generalizability of results across the spectrum of surgical procedures. (13) Patients over 65 years of age who were continuously enrolled in Medicare part A, B and D without health maintenance organization (HMO) enrollment in the year prior to the surgery were included in the cohort.

Outcome

We included five commonly-used surgical outcomes:(i) 30-day mortality, defined as death within 30 days from the procedure date (in-hospital or after discharge); (ii) 1-year mortality, defined as mortality within 1 year of the procedure date; (iii) 30-day readmission, defined as readmission within 30 days from the index hospital discharge date; (iv) complications occurring in-hospital or within 30 days of the surgery date, defined using the International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) codes previously validated by chart review in The Complication Program; and (v) FTR, defined as the number deaths in patients who developed a postoperative complication (numerator) relative to the total number of patients who developed a postoperative complication (denominator). (13–16) In evaluating readmissions, we excluded patients who died during the index hospitalization or after admission to a non-prospective payment system hospital from both index and readmission causes. We considered eight major complications: pulmonary failure, pneumonia, myocardial infarction, deep venous thrombosis/pulmonary embolism, acute renal failure, hemorrhage, surgical site infection and gastrointestinal bleeding. (Appendix 1)

Covariates

Age, gender, race/ethnicity, type of surgical procedure, Medicare and Medicaid dual eligibility status and original reason for entitlement were included as covariates in this study.

Comorbidity scores

Diagnosis-based scores

Medicare inpatient and outpatient claims data were used to identify comorbidities. All claims in the year prior to the index surgery date were queried for the presence of comorbidities. We did not include comorbidities found only at the index hospitalization, to ensure that we captured comorbidities only, and not complications. Patients were considered as having a given comorbidity if they had (i) at least one diagnosis from the inpatient file or (ii) at least two distinct diagnoses recorded more than 30 days apart from the outpatient file. (17)

The Charlson comorbidity score (CCS) adapted for the use with claims data includes 17 disease conditions. (7, 18, 19) Weights are assigned to each disease condition and summed for each patient to obtain a single summary score. (20) Four different versions of CCS weights are available for prediction of 1-year mortality. (Appendix 2) The original weights for CCS were derived by Charlson et al. using all inpatient records from New York Hospital Cornell Medical Center and validated in breast cancer patients. (7) Schneeweiss et al. derived CCS weights using New Jersey Medicare data and validated their model using Pennsylvania Medicare data. (21) Quan et al. updated the CCS weights using hospital data from Canada and validated their model with hospital data from six different countries. (22, 23) The Royal College of Surgeons Charlson score included 14 disease conditions (peptic ulcer disease was omitted; mild and severe liver disease were combined; and diabetes with and without complications were combined) and assigned an equal weight to all conditions to develop a summary score. (24) We also constructed CCS by summing up the 17 disease conditions for each patient without applying any type of weights. Therefore, five different versions of summary CCS were constructed.

The Elixhauser comorbidity score includes 30 disease conditions. The original comorbidity index was not developed as a summary score, and it included 30 indicator variables for disease conditions. (8, 25) Walraven et al. used inpatient records from the Ottawa Hospital, Canada and derived weights for the Elixhauser comorbidity score to make it as a summary score. (26) We also constructed an Elixhauser score by summing the number of the 30 disease conditions for each patient.

The Combined Comorbidity Score was developed by including disease conditions from the Charlson and the Elixhauser comorbidity scores. The Combined Comorbidity Score includes 20 unique disease conditions from the possible 37 disease conditions obtained from Charlson and Elixhauser. Weights were derived for these disease conditions using Pennsylvania Medicare data and validated in New Jersey Medicare data to predict one year mortality. (9) We also constructed the Combined Comorbidity number of conditions for each patient by summing up the 20 disease conditions.

The CMS-HCC risk index was developed by the CMS to calculate the risk of adjustment to capitation payments to Medicare Advantage health plans. The CMS-HCC uses 87 HCCs (of a total of 189 HCCs) to predict Part A and Part B medical expenditures. In addition to 87 HCCs, it also uses certain HCC interactions, age-gender interactions, an indicator for at least one month of Medicaid enrollment in the base year and an indicator for original disability status. The CMS-HCC assigns expenditure-related weights to 87 HCC and other demographic variables to create a summary core. In this study, we used CMS-HCC version 21 using a publicly available SAS code. (10) We did not use the 87 indicator variables as this method would have limited application to smaller datasets or for patients with rare outcomes; data over fitting and lack of convergence can also be issues. CMS-HCC cannot be included as a count variable because it uses interactions of HCCs and demographic variables.

Prescription medication-based score

Medicare part D prescription claims data were used to define disease categories. The Red Book and American Hospital Formulary System (AHFS) were used to classify drugs into appropriate categories, each of which represents a disease condition.

The Chronic disease score (CDS) includes 29 disease conditions identified using prescription drug classes. The original CDS derived weights for total cost, outpatient cost and primary care visit weights. In this study, we used primary care visit weights to make a summary comorbidity score. Prior studies have used a physician-visit weight-based CDS to predict 1-year mortality. (11) We constructed the CDS number of disease conditions for each patient by summing up the 29 disease conditions. The CDS has been used in Medicare patients to predict different outcomes. (27, 28) However, no prior study has determined the performance of CDS in surgical patients for surgical outcomes.

Statistical analysis

Descriptive statistics were used to describe the study cohort and the distribution of summary comorbidity scores. Spearman correlation coefficients were used to describe the correlation between comorbidity scores.

Logistic regression models were constructed for five surgical outcomes. The baseline model included age, gender, race/ethnicity, type of surgical procedure, Medicare and Medicaid dual eligibility and original reason for entitlement. For each comorbidity score model, we included the baseline model covariates plus a particular comorbidity score. In addition to summary comorbidity score models, we also considered models in which comorbidities were entered as indicator variables, except for CMS-HCC. A total of 30 logistic regression models were constructed for each outcome (1-baseline model, 13-diagnosis-based score models, 3-prescription-based score model and 13-diagnosis plus prescription-based score models). (Appendix 3)

The performance of comorbidity scores were evaluated with c-statistics and integrated discrimination improvement (IDI). (29) The discrimination ability of all logistic models was also compared using c-statistics with asymptotic 95% confidence intervals. A general guideline to evaluate different models based on the c-statistics is as follows: 0.5=chance prediction; 0.7–0.8=acceptable; 0.8–0.9=excellent; 1.0=perfect prediction. (30) The IDI is the difference in discrimination slopes between two models, where the discrimination slope for a model is the mean difference in predicted probability of mortality between cases and controls. (31, 32) A higher IDI value indicates that the new model performs better than the referent model.

All statistical analyses were performed using SAS 9.4. This study was considered non-human subjects research and was deemed exempt by the Institutional Review Board at the University of Texas Medical Branch, Galveston.

RESULTS

Use of comorbidity scores in surgical studies

A total of 1,723 studies were identified using the initial search strategy; of these, 245 studies met the inclusion criteria. Fully 147 (60%) studies used Charlson comorbidity scores, 47 (19.2%) used Elixhauser comorbidity scores, 46 (18.8%) studies used some other method such as controlling clinically relevant comorbidities and 5 (2.0%) studies used the CMS-HCC score. None of the studies used a Combined Comorbidity Score or a prescription-based comorbidity score. Of the studies that used the Charlson comorbidity score, 8 (5.4%) used 17 disease conditions as separate covariates, 133 (90.5%) used the summary score and 6 (4.1%) used the total number of conditions. For studies using the Elixhauser comorbidity score, 34 (72.3%) used distinct disease conditions, 10 (21.3%) used the total number of conditions and 3 (6.4%) used a summary score. (Appendix 4)

Comparison of comorbidity scores

The study included 39,616 patients (Figure 1). The mean age of the cohort was 77.3 ± 7.5 years and 60.7% were females. The cohort predominantly consisted of non-Hispanic whites (86.3%), non-Hispanic Blacks (5.8%) and Hispanics (5.8%). Hip replacement was the most common procedure (44.4%), followed by colectomy (24.6%), CABG (21.8%), endovascular repair of abdominal aortic aneurysm (4.8%), lobectomy (2.9%) and open repair of abdominal aortic aneurysm (1.5%). The overall 30-day mortality was 5.8%, 1-year mortality was 17.7%, 30-day readmission was 14.1%, the complication rate was 39.7% and FTR was 14.5%. (Table 1)

Figure 1.

Cohort Selection

Table 1.

Descriptive Statistics of the Cohort

Characteristic	Number (%)
N	39,616
Age, mean (SD)	77.3 (7.5)
Race/ethnicity
Non-Hispanic whites	34,189 (86.3)
Non-Hispanic blacks	2,312 (5.8)
Hispanics	2,282 (5.8)
Others	833 (2.1)
Female	24,036 (60.7)
Surgery type
Coronary-artery bypass grafting	8,643 (21.8)
Lobectomy	1,164 (2.9)
Endovascular repair of abdominal aortic aneurysm	1,895 (4.8)
Open repair of abdominal aortic aneurysm	575 (1.5)
Colectomy	9,744 (24.6)
Hip replacement	17,595 (44.4)
Medicare and Medicaid dual eligible patients	10,944 (27.6)
Original reason for entitlement
Old age	34,812 (90.4)
Disability or end stage renal disease	3,804 (9.6)
Outcomes
30-day mortality	2,282 (5.8)
1-year mortality	6,991 (17.7)
30-day readmissions†	4,433 (14.1)
Complications	15,718 (39.7)
Failure to rescue‡	2,282 (14.5)

^†Denominator is 30,848. Patient died in the index hospitalization or admissions to non-prospective payment system hospital from both index and readmission were excluded (n=8,768).

^‡Denominator is 15,718.

Table 2 reports the distribution of summary comorbidity scores. The mean score was highest for the Elixhauser summary score (4.7 ± 6.8) and lowest for the Royal College of Surgeon’s CCS (1.2 ± 1.1). More than 30% of patients had zero score for the CCS, Elixhauser comorbidity score or the Combined Comorbidity Score. In contrast, less than 10% of patients had a zero score for the CDS and no patient had a zero score for the CMS-HCC. Diagnosis-based comorbidity scores were highly correlated (ranging from 0.57 to 0.97), whereas the correlation between diagnosis- and prescription-based comorbidity scores was low to moderate (ranging from 0.14 to 0.32). (Appendix 5)

Table 2.

Distribution of Summary Comorbidity Scores in the Cohort

Comorbidity Score	Mean (SD)	% with 0	25th Percentile	Median	75th Percentile	Minimum	Maximum
CCS original	1.7 (2.0)	34.9	0	1	3	0	17
CCS/Schneeweiss	2.3 (2.7)	36.3	0	1	4	0	20
CCS/Quan	1.4 (1.9)	51.6	0	0	2	0	15
Royal College of Surgeons CCS	1.6 (1.1)	35.1	0	1	2	0	3
Elixhauser summary score	4.7 (6.8)	35.3	0	2	7	−11	49
Combined Comorbidity Score	1.2 (2.3)	30.4	0	0	2	−2	17
CMS-HCC	2.7 (1.7)	0	1.4	2.2	3.5	0.3	15.4
CDS	2.4 (1.5)	3.7	1.3	2.2	3.2	−1.2	10.7

Abbreviations: CCS, Charlson comorbidity score; CMS-HCC, Center for Medicare & Medicaid – Hierarchical Conditions Categories; CDS, chronic disease score

Table 3 reports the c-statistics for different comorbidity scores. The baseline model which included age, gender, race/ethnicity, type of surgery, dual eligibility status and original reason for entitlement performed poorly in predicting 30-day mortality (c=0.698), 1-year mortality (0.689), 30-day readmission (c=0.591), complications (c=0.697) and FTR (c=0.701). The addition of any comorbidity score to the baseline model improved the model performance significantly. Among models in which individual disease conditions were included as indicator variables, Elixhauser consistently performed the best in predicting the five surgical outcomes studied, followed by the Combined Comorbidity Score. Among diagnosis- and prescription-based summary comorbidity measures, the CMS-HCC performed the best, followed by the Combined Comorbidity Score for all five surgical outcomes. Overall, the ranking of the top four comorbidity scores were as follows: CMS-HCC > Elixhauser disease categories > Combined Comorbidity Score disease categories > Combined Comorbidity Score. When comorbidities were controlled for (as the number of disease conditions in the Elixhauser or the Combined Comorbidity Score), the instruments performed poorly compared to their respective weighted comorbidity score; the CCS/number of disease conditions performed poorer than the CCS/Schneeweiss summary score. The performance of prescription-based summary comorbidity scores was no better than any diagnosis-based summary comorbidity scores. The c-statistics results showed that addition of CDS to diagnosis-based scores improved c-statistics by less than 3%. (Appendix 6) A large proportion of patients were excluded because of Part D eligibility criteria. In order to address concerns regarding generalizability, we tested the performance of diagnosis-based scores in 89,317 patients who met Part A and B criteria. The relative performance of diagnosis-based comorbidity scores remained the same in the 89,317 patients as it was in the 39,616 patients. (Appendixes 7 and 8)

Table 3.

Comparative Performance of Comorbidity Scores in Predicting Surgical Outcomes

Comorbidity Score C-Statistics (95% CI)	30-Day Mortality	1-Year Mortality	30-Day Readmission	Complications	Failure To Rescue
Baseline model (age, gender, race/ethnicity, procedure type, dual eligibility status, original reason for entitlement)	0.698 (0.687, 0.709)	0.689 (0.682, 0.696)	0.591 (0.582, 0.600)	0.697 (0.692, 0.703)	0.701 (0.689, 0.713)
Comorbidity scores with disease categories †
Charlson disease categories	0.742 (0.732, 0.752)	0.746 (0.740, 0.753)	0.624 (0.615, 0.633)	0.726 (0.721, 0.731)	0.741 (0.730, 0.752)
Elixhauser disease categories	0.750 (0.740, 0.760)	0.755 (0.749, 0.761)	0.629 (0.620, 0.637)	0.730 (0.725, 0.735)	0.749 (0.738, 0.760)
Combined disease categories	0.749 (0.739, 0.759)	0.753 (0.747, 0.759)	0.627 (0.618, 0.636)	0.729 (0.724, 0.734)	0.747 (0.736, 0.759)
CDS disease categories	0.725 (0.715, 0.736)	0.727 (0.721, 0.733)	0.617 (0.609, 0.627)	0.714 (0.709, 0.719)	0.726 (0.715, 0.737)
Diagnosis-based summary comorbidity score ‡
CCS original	0.729 (0.719, 0.739)	0.737 (0.731, 0.743)	0.615 (0.606, 0.624)	0.715 (0.709, 0.720)	0.729 (0.717, 0.740)
CCS/Schneeweiss	0.735 (0.725, 0.746)	0.742 (0.736, 0.749)	0.620 (0.611, 0.629)	0.719 (0.714, 0.724)	0.735 (0.724, 0.746)
CCS/Quan	0.730 (0.719, 0.741)	0.737 (0.730, 0.744)	0.614 (0.605, 0.623)	0.713 (0.708, 0.718)	0.730 (0.718, 0.741)
Royal College of Surgeons CCS§	0.731 (0.721, 0.741)	0.734 (0.727, 0.740)	0.616 (0.607, 0.625)	0.717 (0.712, 0.722)	0.731 (0.720, 0.742)
CCS/number of disease	0.731 (0.721, 0.742)	0.734 (0.728, 0.740)	0.617 (0.608, 0.626)	0.717 (0.712, 0.722)	0.731 (0.720, 0.742)
Elixhauser summary score	0.741 (0.731, 0.752)	0.745 (0.739, 0.751)	0.618 (0.609, 0.626)	0.716 (0.711, 0.721)	0.740 (0.729, 0.751)
Elixhauser/number of disease	0.720 (0.710, 0.730)	0.718 (0.711, 0.724)	0.615 (0.607, 0.624)	0.713 (0.708, 0.718)	0.722 (0.711, 0.734)
Combined Comorbidity Score	0.745 (0.735, 0.756)	0.751 (0.745, 0.757)	0.619 (0.610, 0.628)	0.721 (0.716, 0.726)	0.743 (0.732, 0.754)
Combined Comorbidity/number of disease	0.728 (0.718, 0.738)	0.728 (0.721, 0.734)	0.616 (0.608, 0.625)	0.716 (0.711, 0.721)	0.730 (0.719, 0.741)
CMS-HCC	0.797 (0.789, 0.806)	0.798 (0.793, 0.803)	0.630 (0.622, 0.639)	0.766 (0.762, 0.771)	0.811 (0.802, 0.820)
Prescription-based summary comorbidity score
CDS	0.700 (0.690, 0.710)	0.696 (0.689, 0.702)	0.603 (0.594, 0.612)	0.703 (0.698, 0.708)	0.704 (0.692, 0.716)
CDS/number of disease	0.700 (0.690, 0.710)	0.691 (0.684, 0.698)	0.595 (0.586, 0.604)	0.700 (0.695, 0.705)	0.704 (0.692, 0.717)

Abbreviations: CCS, Charlson comorbidity score; CMS-HCC, Center for Medicare & Medicaid – Hierarchical Conditions Categories; CDS, chronic disease score

^†Top 2 comorbidity scores from disease categories are highlighted in light gray color.

^‡Top 2 comorbidity scores from diagnosis-based summary score are highlighted in light gray color.

Table 4 reports the IDI for comorbidities scores. All comorbidity scores were compared against the baseline model. The results of IDI were in accordance with the c-statistics results. The relative ranking of the top four comorbidity scores were as follows: CMS-HCC > Elixhauser disease categories > Combined Comorbidity Score disease categories > Combined Comorbidity Score.

Table 4.

Integrated Discrimination Improvement (IDI)of Comorbidity Scores in Predicting Surgical Outcomes

Comorbidity Score IDI (%)	30-Day Mortality	1-Year Mortality	30-Day Readmission	Complications	Failure To Rescue
Baseline model (age, gender, race/ethnicity, procedure type, dual eligibility status, original reason for entitlement)	Ref	Ref	Ref	Ref	Ref
Comorbidity scores with disease categories †
Charlson disease categories	1.96	4.96	1.18	3.36	1.74
Elixhauser disease categories	2.37	5.82	1.43	3.99	2.17
Combined disease categories	2.31	5.66	1.36	3.85	2.07
CDS disease categories	1.24	3.17	0.96	1.91	1.05
Diagnosis-based summary comorbidity score ‡
CCS original	1.03	3.79	0.73	1.96	0.85
CCS/Schneeweiss	1.42	4.33	0.97	2.63	1.23
CCS/Quan	1.07	3.82	0.67	1.75	0.87
Royal College of Surgeons CCS	1.29	3.47	0.77	2.33	1.07
CCS/number of disease	1.30	3.47	0.79	2.36	1.08
Elixhauser summary score	1.71	4.74	0.90	2.27	1.51
Elixhauser/number of disease	0.71	1.96	0.78	1.84	0.63
Combined Comorbidity Score	1.88	5.45	1.00	2.82	1.60
Combined Comorbidity/number of disease	1.01	2.75	0.82	2.20	0.91
CMS-HCC	4.59	9.60	1.27	9.37	5.24
Prescription-based summary comorbidity score
CDS	0.10	0.46	0.35	0.63	0.11
CDS/number of disease	0.06	0.11	0.10	0.23	0.07

Abbreviations: CCS, Charlson comorbidity score; CMS-HCC, Center for Medicare & Medicaid – Hierarchical Conditions Categories; CDS, chronic disease score

^†Top 2 comorbidity scores from disease categories are highlighted in light gray color.

^‡Top 2 comorbidity scores from diagnosis-based summary score are highlighted in light gray color.

DISCUSSION

We compared the performance of commonly-used diagnosis- and prescription-based comorbidity scores in predicting 30-day mortality, 1-year mortality, 30-day readmission, complications and FTR in patients undergoing high-risk surgery. Our review of the use of comorbidity scores in surgical studies found that the majority of studies used Charlson comorbidity score; only five studies used CMS-HCC. Our study results showed that CMS-HCC performed the best in predicting all surgical outcomes, followed by Elixhauser disease categories, Combined Comorbidity Score disease categories and Combined Comorbidity Score.

A recent systematic review described the development and validation of comorbidity scores. The authors reviewed 76 articles; similar to our study, diagnosis-based measures performed better in predicting mortality, but prescription-based measures demonstrated better performance in predicting health care utilization. (33) Prior studies have shown that use of CCS+CDS, compared to CCS alone, improved c-statistics for mortality prediction by 1.7% and adjusted R2 in explaining healthcare expenditure variation by 25%. (27, 34) We found that, in surgical patients, diagnosis-based comorbidity scores performed better than prescription-based scores and the use of both did not offer any advantage in predicting surgical outcomes compared to diagnosis-based score alone.

Only three studies have compared the performance of comorbidity scores in surgical patients. Atherly et al. compared National Surgical Quality Improvement Program (NSQIP), DxCG and CCS in predicting mortality in surgery and concluded that NSQIP performed better (c=0.94) than DxCG (c=0.59) or CCS (c=0.52). (35) However, this is a case of comparing “apples to oranges”. NSQIP includes data on preoperative comorbidities, lab values and intraoperative events (45 preoperative and 17 intraoperative factors), whereas CCS only includes 17 comorbidities. (36) Two other studies compared Charlson and Elixhauser comorbidity scores in patients having orthopedic surgery and concluded that Elixhauser outperformed Charlson. (28, 37) In our study, Elixhauser had better performance compared to the Charlson comorbidity score.

Our results are consistent with findings evaluating comorbidity scores in patients with cardiovascular diseases, cancer, renal disease, respiratory disease and patients undergoing orthopedic surgery. (12) This meta-analysis (54 articles) compared the performance of diagnosis-based comorbidity indices using administrative data and the authors concluded that the Elixhauser comorbidity index performed better than CCS in predicting long-term mortality. In our study of surgical patients, when disease conditions were entered as indicator variables, Elixhauser performed better than the Combined Comorbidity Score and Charlson comorbidity score. Of disease conditions, the Elixhauser includes 30, the Combined Comorbidity Score includes 20 and the Charlson includes 17. This greater detail likely explains why the Elixhauser comorbidity measure performed the best. Ours is the first study that compared the ability of CMS-HCC to predict surgical outcomes. We found that the CMS-HCC score performed the best in predicting all surgical outcomes. Future studies evaluating surgical outcome using Medicare data can use the CMS-HCC to control for confounding due to comorbidity.

In the best case scenario, a summary score can perform only as well as individual disease categories. (38) A few studies have utilized CCS and Elixhauser by summing up the number of disease conditions without applying any kind of weights; such a method ignores the complexity of the various disease conditions. (39, 40) For example, patients with metastatic cancer and those with ulcer disease receive equal weight, a process which is intuitively incorrect. Therefore, it would appear to be better to use weights to construct a summary comorbidity score rather than using the number of disease conditions. Our study results support this argument, with the worst performance coming from scores summing unweighted comorbidities.

Weighted CCS summary score was the most popular measure for comorbidity control in surgical studies, with most studies using original CCS weights derived in 1987. (7) Four different versions of weights are available to construct the summary CCS. (Appendix 2) In our study, summary comorbidity scores performed slightly worse than comorbidity models that included indicator variables for individual disease conditions. Furthermore, the weights for comorbidity scores were not derived for surgical patients, which may explain the lower performance of summary scores in this population. We found that weights derived by Schneeweiss et al. using Medicare data performed better than other weights. (21) Prior studies have shown that a comorbidity score developed for specific a population and outcome of interest can perform better than generic scores. (12, 21, 30, 41)

Summary comorbidity scores can be very useful as they offer several advantages over other systems of measure. Summary comorbidity scores can be used to control for several comorbidities in smaller datasets without loss of statistical power. (26) The overall comorbidity burden of the cohort can be easily summarized using a single summary score, and summary scores allow investigators to model interactions between other variables and comorbidity as well as to construct time-dependent models of the comorbidity score. (42, 43) Furthermore, a summary comorbidity score can be easily incorporated when developing a prognostic nomogram. (44)

The study had the following limitations. We selected older adults and six major surgical procedures to represent the spectrum of surgical specialties. Therefore, results should be generalizable to older adults with similar surgical complexity. The study used the Medicare data from Texas and we recognize the differences in demographic differences between Texas and national population. However, the rates of outcomes and distribution of comorbidity scores are similar. (13) The CMS-HCC can be applied only to the Medicare data. The performance of comorbidity scores in other healthcare claims and electronic medical records datasets is yet to be tested. We did not include comorbidities from the index hospitalization to ensure that we do not inadvertently include complications. While including disease conditions from the index hospitalization may increase the overall predictive power of comorbidity scores, it runs the risk of misclassifying complications as comorbidities. Identifying complications from Medicare data without “present on admission” codes can be problematic. However, we chose to include a subset of complications that have high sensitivity and specificity. (14, 15, 45)

Conclusion

In conclusion, all diagnosis-based comorbidity scores performed better than prescription-based scores in predicting surgical outcomes, and combining both types of scores did not improve the performance. Based on the comparative performance of comorbidity scores, we recommend the use of CMS-HCC to control for confounding in surgical studies that use Medicare data.