The Charlson and Elixhauser Scores Outperform the American Society of Anesthesiologists Score in Assessing 1-year Mortality Risk After Hip Fracture Surgery

Abstract

Background

Risk adjustment has implications across orthopaedics, including informing clinical care, improving payment models, and enabling observational orthopaedic research. Although comorbidity indices (such as the American Society of Anesthesiologists [ASA] classification, Charlson comorbidity index [CCI], and Elixhauser comorbidity index [ECI]) have been examined extensively in the immediate perioperative period, there is a dearth of data on their three-way comparative effectiveness and long-term performance. Moreover, the discriminative ability of the CCI and ECI after orthopaedic surgery has not been validated in the ICD-10 era, despite new diagnosis codes from which they are calculated.

Question/purpose

Which comorbidity index (ASA, CCI, or ECI) is associated with the greatest accuracy on receiver operating curve (ROC) analysis with respect to the endpoint of death at 90 days and 1 year after hip fracture surgery in the ICD-10 era?

Methods

A retrospective study was conducted on all patients undergoing surgical fixation of primary hip fractures at two Level I trauma centers and three community hospitals from October 2016 to May 2019. This time frame allowed for a 1-year baseline period of ICD-10 data to assess comorbidities and at least a 1-year follow-up period to assess mortality. Initially 1516 patients were identified using Common Procedural Terminology and ICD codes, of whom 4% (60 of 1516) were excluded after manual review; namely, those with pathologic fractures (n = 38), periprosthetic fractures (n = 12), and age younger than 18 years (n = 10). Of the patients who were studied, 69% (998 of 1456) were women and the mean ± SD age was 77 ± 14 years; 45% (656 of 1456) were treated with intramedullary nails, 32% (464 of 1456) underwent hemiarthroplasties, 10% (149 of 1456) underwent THAs, 7% (104 of 1456) underwent percutaneous fixations, and 6% (83 of 1456) were treated with plates and screws. The mean ± SD ASA score was 2.8 ± 0.6, CCI was 3.1 ± 3.2, and ECI was 5.2 ± 3.5. Hip fracture fixation was chosen as the operation of interest given the high incidence of this injury, the well-documented effects of comorbidities on complications, and the critical importance of risk stratification and perioperative medical management for these patients. Demographics, comorbidities, surgical details, as well as 90-day and 1-year mortality were collected. Logistic regressions with ROC curves were used to determine the accuracy and comparative effectiveness of the three measures. The 90-day mortality rate was 7.4%, and the 1-year mortality rate was 15.0%.

Results

The accuracy (area under the curve [AUC]) for 1-year mortality was 0.685 (95% CI 0.656 to 0.714) for the ASA, 0.755 (95% CI 0.722 to 0.788) for the ECI, and 0.769 (95% CI 0.739 to 0.800) for the CCI. The CCI and ECI were more accurate than ASA (p < 0.001 for both), while the CCI and ECI did not differ (p = 0.30). The ECI (AUC 0.756 [95% CI 0.712 to 0.800]) was more accurate for 90-day mortality than the ASA (AUC 0.703 [95% CI 0.663 to 0.744]; p = 0.04), while CCI (AUC 0.742 [95% CI 0.698 to 0.785]) with ASA (p = 0.17) and CCI with ECI (p = 0.46) did not differ at 90 days.

Conclusion

Performance measures and research results may vary depending on what comorbidity index is used. We found that the CCI and ECI were more accurate than the ASA score for 1-year mortality after hip fracture surgery. Moreover, these data validate that the CCI and ECI can perform reliably in the ICD-10 era. If other studies from additional practice settings confirm these findings, as would be expected because of the objective nature of these indices, the CCI or ECI may be a useful preoperative measure for surgeons to assess 1-year mortality for hip fracture patients and should likely be used for institutional orthopaedic research involving outcomes 90 days and beyond.

Level of Evidence

Level III, diagnostic study.

Introduction

Assessing surgical risk is crucial in guiding preoperative decision-making, promoting reasonable expectations, maximizing resource allocation, providing appropriate bundled payments, enabling accurate public outcome reporting, and minimizing confounding of research results that may lead to incorrect or misleading conclusions [4, 13, 16, 17, 29, 37, 50, 51, 56]. Consequently, numerous risk-adjustment strategies have been developed, with the American Society of Anesthesiologists (ASA) physical status classification system [47], the Elixhauser comorbidity index (ECI) [14], and the Charlson comorbidity index (CCI) [8] being three of the indices most commonly used for this purpose. The ASA includes four to six physical status categories designed to assess a patient’s baseline physical status and immediate perioperative risk [2, 12, 47]. The ECI includes 31 unique conditions designed to predict inpatient mortality [14, 45], and the CCI includes 17 individually weighted conditions designed to predict 1-year mortality [8, 45].

Despite their varying aims, the rationale for using certain indices for specific studies often is not stated, which may be a function of the lack of research on their three-way comparative effectiveness and long-term performance. Prior work has shown the ECI generally outperforms the CCI in adjusting for inpatient mortality risk after orthopaedic surgery [24, 31, 34, 40, 41], and that the ASA typically outperforms the modified CCI for 30-day mortality in the National Surgical Quality Improvement Program database [19, 25, 38, 39]. However, data evaluating the performance of the ASA and CCI in terms of any longer-term outcome are scarce [43], and the performance of the ECI beyond the immediate perioperative period for any outcome is unknown. Moreover, data comparing the ASA and true CCI are extremely limited [43], the relative performance of the ASA and ECI has not been evaluated, and the comparative accuracy of all three leading indices (ASA, CCI, and ECI) for outcomes after orthopaedic surgery has never, to our knowledge, been examined at any timepoint. Finally, the ECI and CCI have not been validated for patients undergoing orthopaedic surgery in the ICD-10, era, despite an entirely new set of diagnosis codes from which the scores are calculated.

Therefore, we asked: Which comorbidity index (ASA, CCI, or ECI) is associated with the greatest accuracy on receiver operating curve (ROC) analysis with respect to the endpoint of death at 90 days and 1 year after hip fracture surgery in the ICD-10 era?

Patients and Methods

Study Design and Setting

This retrospective study included all patients undergoing surgical fixation of primary (that is, nonperiprosthetic, nonpathologic) hip fractures at two Level I trauma centers (Brigham and Women’s Hospital [urban, Boston, MA, USA, 793 beds] and Massachusetts General Hospital [urban, Boston, MA, USA, 999 beds]) and three community hospitals (Brigham and Women’s Faulkner Hospital [suburban, Jamaica Plain, MA, USA, 162 beds], Newton Wellesley Hospital [suburban, Newton, MA, USA, 273 beds], and North Shore Medical Center [suburban, Salem, MA, USA, 395 beds]) between October 1, 2016 and May 1, 2019. We chose this timeframe to ensure that all patients had a 1-year baseline period of ICD-10 data to assess comorbidities and at least a 1-year follow-up period to assess postoperative mortality. Hip fracture fixation was chosen as the operation of interest given the increasingly high incidence of this injury [7, 42], the well-documented effects of comorbidities on complications for these patients [20, 48, 49], and the critical importance of risk stratification and perioperative medical management for this set of procedures [28, 32, 35].

Identification of Eligible Patients

Patients were identified by querying our institutional database using Common Procedural Terminology (CPT) and/or ICD-10 diagnosis codes for all hip fractures undergoing open reduction and internal fixation with an intramedullary device (CPT 27245) or plates and screws (CPT 27244), hemiarthroplasty for a hip fracture (CPT 27236), hip hemiarthroplasty and hip fracture (CPT 27125 and ICD-10 s72.*), THA and hip fracture (CPT 27130 and ICD-10 s72.*), and percutaneous fixation (CPT 27235), yielding 1516 potential primary hip fracture patients.

Medical records were reviewed to confirm the operative diagnosis and extract additional demographic information (age at the time of surgery, gender, patient-reported race, and primary language) and surgical details (hospital type [academic or community], time to surgery from admission, surgical procedure, fracture location, and date of operation). Patients with pathologic fractures (n = 38), periprosthetic fractures (n = 12), and age younger than 18 years (n = 10) were excluded, leaving 96% of the original dataset available for analysis here (n = 1456 patients). The primary exposures of this study were each patient’s ASA, CCI, and ECI scores. ASA scores were determined and documented by the primary anesthesiologist on the case. CCI and ECI scores were calculated based on documented diagnoses within 1 year preoperatively [44] using ICD-10 codes as determined by Quan et al. [45]. These ICD-10 codes were developed by an international consortium of researchers and have been validated in a nonoperative setting [45]. The CCI and ECI were algorithmically calculated directly from our electronic medical record. Importantly, (1) these scores were derived to be applicable across all adult inpatients (regardless of condition) [45] and (2) we assessed all scores on all patients. As such, including all adult hip fracture patients serves only to increase the generalizability of our findings (nonetheless, for reference, more than 95% of patients were 50 years or older).

Patient Demographics, Baseline Data, and Summary Mortality Statistics

A total of 1456 patients underwent hip fracture surgery during the 31-month data collection period (Table 1 ). The mean ± SD age was 77 ± 14 years, and 69% (998 of 1456) of the patients were women. Time to surgery from admission was under 48 hours for 66% of procedures. There were 45% (656 of 1456) fixations with intramedullary nails, 32% (464 of 1456) hemiarthroplasties, 10% (149 of 1456) THAs, 7% (104 of 1456) percutaneous fixations, and 6% (83 of 1456) fixations with plates and screws. The mean ± SD ASA score was 2.8 ± 0.6, CCI score was 3.1 ± 3.2, and ECI score was 5.2 ± 3.5. Mortality rates were 1.9% inpatient, 7.4% at 90 days, and 15% at 1 year.

Table 1.

Baseline demographic and mortality data

	Total
	(n = 1456)
American Society of Anesthesiologists Classification
Mean ± SD	2.8 ± 0.6
1	2 (30)
2	25 (365)
3	63 (919)
4	10 (142)
Elixhauser comorbidity index
Mean ± SD	5.2 ± 3.5
0-3	38 (547)
4-7	39 (561)
8+	24 (348)
Charlson comorbidity index
Mean ± SD	3.13 ± 3.19
0-3	66 (954)
4-7	23 (331)
8+	12 (171)
Age in years
Mean ± SD	77 ± 14
< 50	5 (72)
50-59	6 (93)
60-69	16 (231)
70-79	23 (333)
80-89	34 (501)
90-99	15 (222)
100+	0.3 (4)
% Women	69 (998)
Racea
Asian	3 (39)
Black	3 (49)
Hispanic	1 (20)
Other/unknown	5 (70)
White	88 (1278)
Hospital type
Academic	90 (1304)
Community	10 (152)
Time to surgery > 48 hours	35 (503)
Surgery type
Hemiarthroplasty	32 (464)
Intramedullary nail	45 (656)
Plates/screws	6 (83)
Percutaneous fixation	7 (104)
THA	10 (149)
90-day mortality	7 (108)
1-year mortality	15 (219)

Data presented as mean ± SD or % (n).

^aPatient self-reported.

Primary and Secondary Study Outcomes

The primary outcome of this study was accuracy for 1-year mortality, and accuracy for 90-day mortality was analyzed as a secondary outcome. Logistic regressions with ROC curve analyses were implemented to determine the accuracy, as reflected by the area under the curve (AUC). Mortality and death dates were collected via a medical record review of documented deaths and a review of provider notes and telephone calls for any documentation of death, even if the death had not been formally marked in our electronic medical record. Deaths were subsequently validated, and any missing deaths and/or dates were obtained using the Social Security Index Master Death Index file. The Master Death Index file is updated monthly at our institution, and this review occurred more than 2 months after the final patient reached 1 year of follow-up. We believe this multipronged approach successfully obtained the verified death status and dates for all deceased patients in this study.

Ethical Approval

Ethical approval for this study was obtained from Brigham and Women’s Hospital, Boston, MA, USA (IRB approval number 2020P003243).

Statistical Analysis

We compared categorical variables using the chi-square or Fisher exact tests as appropriate, and we compared continuous variables using t-tests or the Mann-Whitney U tests. Statistical comparisons between ROC curves were performed using the standard methods of DeLong et al. [11]. Initial analyses provided the overall accuracy of each index, both alone and compared with the others. Similarly to Menendez et al. [34], we then built a baseline-adjusted model accounting for age, gender, and year of surgery, and analyzed the improvement in performance with the addition of each comorbidity score individually. In addition to analyzing the absolute improvements in accuracy, we then calculated the difference between the indices in terms of percent improvement beyond the accuracy of the baseline model [27, 34]. For example, a difference in AUCs between an ASA of 0.7 and ECI of 0.8, assuming a base model AUC of 0.65, represents a 67% relative improve in accuracy for the ECI relative to the ASA: ([0.8-0.65]-[0.7-0.65])/(0.8-0.65). Next, to contextualize the performance of the comorbidity indices in reference to comorbidities commonly encountered in the preoperative medical management of patients with hip fractures in our clinical experience, we analyzed seven common comorbidities (congestive heart failure, cardiac arrhythmia [anticoagulation], valvular disease [anticoagulation], chronic pulmonary disease, pulmonary circulation disorders, uncomplicated diabetes, and complicated diabetes), both alone and in a single model including all seven conditions. Finally, we added all seven comorbidities to the base model accounting for age, gender, and year and again assessed their relative performance. All analyses were performed in SAS version 9.4 (SAS Institute), and p < 0.05 was considered significant.

Results

Analysis of Comorbidity Indices

The ECI and CCI outperformed the ASA for 1-year mortality and the ECI outperformed the ASA for 1-year mortality (Fig. 1). Specifically, the AUCs for 1-year mortality were 0.685 (95% CI 0.656 to 0.714) for the ASA, 0.769 (95% CI 0.739 to 0.800) for the CCI, and 0.755 (95% CI 0.722 to 0.788) for the ECI (Table 2), with both the CCI (AUC difference, 0.084 [95% CI 0.045 to 0.123]; p < 0.001) and ECI (0.070 [95% CI 0.032 to 0.109]; p < 0.001) being more accurate than the ASA (Table 3). The CCI and ECI did not differ (AUC 0.014 [95% CI -0.012 to 0.04]; p = 0.30). The AUCs for 90-day mortality were 0.703 (95% CI 0.663 to 0.744) for the ASA, 0.742 (95% CI 0.698 to 0.785) for the CCI, and 0.756 (95% CI 0.712 to 0.800) for the ECI (Table 2), with the ECI being more accurate than the ASA (AUC 0.053 [95% CI 0.001 to 0.105]; p = 0.04). The CCI and ECI did not differ (AUC -0.015 [95% CI -0.053 to 0.024]; p = 0.46) (Table 3).

Fig. 1.

These ROC curves represent the CCI, ECI, and ASA regarding 1-year mortality. The farther the curve is to the top-left, the better the performance; the farther the curve is to the bottom right, the worse the performance (the bottom right corner means perfectly predicting the opposite of what occurs). A diagonal line from the origin at 45° (that is, y=x) represents no discriminative ability. Accuracy of a given measure can be estimated by its area under the curve, which were 0.685 (95% CI 0.656 to 0.714) for the ASA, 0.769 (95% CI 0.739 to 0.800) for the CCI, and 0.755 (95% CI 0.722 to 0.788) for the ECI.

Table 2.

AUCs for the discriminative ability of the comorbidity indices alone in predicting 90-day and 1-year mortality

Score	90-day mortality		1-year mortality
	AUC (95% CI)	p value	AUC (95% CI)	p value
ASA	0.703 (0.663-0.744)	< 0.001	0.685 (0.656-0.714)	< 0.001
ECI	0.756 (0.712-0.800)	< 0.001	0.755 (0.722-0.788)	< 0.001
CCI	0.742 (0.698-0.785)	< 0.001	0.769 (0.739-0.800)	< 0.001

AUC = area under the curve; ASA = American Society of Anesthesiologists; ECI = Elixhauser comorbidity index; CCI = Charlson comorbidity index.

Table 3.

Differences in AUCs of the individual comorbidity indices

Score	90-day mortality		1-year mortality
	AUC (95% CI)	p value	AUC (95% CI)	p value
CCI - ECI	-0.015 (-0.053 to 0.024)	0.46	0.014 (-0.012 to 0.04)	0.30
ECI - ASA	0.053 (0.001 to 0.105)	0.04	0.07 (0.032 to 0.109)	< 0.001
CCI - ASA	0.038 (-0.016 to 0.093)	0.17	0.084 (0.045 to 0.123)	< 0.001

AUC = area under the curve; ASA = American Society of Anesthesiologists; ECI = Elixhauser comorbidity index; CCI = Charlson comorbidity index.

Similar results were seen when adding the comorbidity indices to an adjusted base model accounting for age, gender, and year of surgery (Table 4). Specifically, adding the ASA to the base model caused the AUC to increase by 0.102 for 1-year mortality (AUC 0.712 [95% CI 0.677 to 0.747]) and 0.118 for 90-day mortality (AUC 0.727 [95% CI 0.680 to 0.775]). In comparison, adding either the CCI (1-year: 0.170; AUC 0.780 [95% CI 0.750 to 0.809]; 90-day: 0.150; AUC 0.759 [95% CI 0.719 to 0.799]) or ECI (1-year: 0.153; AUC 0.763 [95% CI 0.731 to 0.796]; 90-day: 0.160; AUC 0.769 [95% CI 0.726 to 0.812]) to the base model led to greater improvement than adding the ASA at both timepoints (Table 4). In terms of relative improvement in accuracy at 1 year, the CCI performed 112% better than the ASA, and the ECI outperformed the ASA by 93%. The relative improvements versus ASA for 90-day mortality were 42% for the CCI and 56% for the ECI.

Table 4.

AUCs for the discriminative ability of the base model as well as the base model with comorbidity indices in predicting 90-day and 1-year mortality

Score	90-day mortality		1-year mortality
	AUC (95% CI)	p value	AUC (95% CI)	p value
Base model (age, gender, and year of surgery)	0.609 (0.557-0.661)	< 0.001	0.610 (0.571-0.648)	< 0.001
Base model and ASA score	0.727 (0.680-0.775)	< 0.001	0.712 (0.677-0.747)	< 0.001
Base model and ECI	0.769 (0.726-0.812)	< 0.001	0.763 (0.731-0.796)	< 0.001
Base model and CCI	0.759 (0.719-0.799)	< 0.001	0.780 (0.750-0.809)	< 0.001

AUC = area under the curve; ASA = American Society of Anesthesiologists; ECI = Elixhauser comorbidity index; CCI = Charlson comorbidity index.

Reference Performance

The CCI and ECI outperformed a model with common demographic factors and individual comorbidities for 1-year mortality, while there were no differences in performance at 90 days. Each comorbidity was associated with 1-year mortality (p < 0.003 for all), with odds ratios ranging from 3.49 (95% CI 2.59 to 4.72) for congestive heart failure to 1.70 (95% CI 1.21 to 2.38) for uncomplicated diabetes; however, only the presence of congestive heart failure or cardiac arrhythmia was an even minimally useful (poor) classifier of 1-year or 90-day mortality (AUC > 0.6) (Table 5). When taking all seven comorbidities together in a single model, the accuracy for 1-year and 90-day mortality was 0.709 (95% CI 0.672 to 0.746) and 0.725 (95% CI 0.679 to 0.772), respectively (Table 5). This performance was worse than both the CCI (-0.061; 95% CI -0.099 to -0.022; p = 0.002) and ECI (-0.047; 95% CI -0.075 to -0.018; p = 0.002) alone for 1-year mortality, while it was not better than the ASA (0.024; 95% CI -0.013 to 0.061; p = 0.21) (Table 6). There was no difference between the seven-condition model and the ASA, CCI, or ECI for 90-day mortality (p > 0.10 for all) (Table 6). Finally, when accounting for each of the seven comorbidities in addition to age, gender, and year in a single model, the accuracy for 1-year and 90-day mortality was 0.721 (95% CI 0.684 to 0.757) and 0.736 (95% CI 0.691 to 0.781), respectively. This performance remained worse than that of the CCI (-0.049; 95% CI -0.089 to -0.008; p = 0.02) and ECI (-0.035; 95% CI -0.004 to -0.066; p = 0.03) alone for 1-year mortality and was not different from each of the three indices alone for 90-day mortality (p > 0.18 for all).

Table 5.

AUCs and odds ratios for the individual comorbidities and a model including them all for the outcomes of 90-day and 1-year mortality

Comorbidities	90-day mortality			1-year mortality
	AUC (95% CI)	Odds ratio (95% CI)	p value	AUC (95% CI)	Odds ratio (95% CI)	p value
Congestive heart failure	0.636 (0.587-0.684)	3.43 (2.30-5.11)	< 0.001	0.633 (0.598-0.668)	3.49 (2.59-4.72)	< 0.001
Cardiac arrhythmia	0.637 (0.598-0.675)	3.82 (2.30-6.33)	< 0.001	0.626 (0.595-0.657)	3.20 (2.27-4.50)	< 0.001
Valvular disease	0.591 (0.544-0.639)	2.41 (1.60-3.62)	< 0.001	0.577 (0.544-0.611)	2.17 (1.60-2.96)	< 0.001
Chronic pulmonary disease	0.549 (0.512-0.586)	2.51 (1.47-4.27)	< 0.001	0.541 (0.516-0.566)	2.31 (1.52-3.54)	< 0.001
Pulmonary circulation disorders	0.585 (0.537-0.633)	2.18 (1.46-3.26)	< 0.001	0.573 (0.539-0.608)	2.01 (1.48-2.71)	< 0.001
Diabetes (uncomplicated)	0.547 (0.504-0.590)	1.74 (1.11-2.73)	0.02	0.544 (0.513-0.575)	1.70 (1.21-2.38)	0.002
Diabetes (complicated)	0.533 (0.493-0.573)	1.57 (0.96-2.54)	0.07	0.553 (0.523-0.583)	2.02 (1.42-2.86)	< 0.001
Model with all	0.725 (0.679-0.772)		< 0.001	0.709 (0.672-0.746)		< 0.001

AUC = area under the curve.

Table 6.

Differences in AUCs of the comorbidity indices alone and the model with seven common comorbidities

Score	90-day mortality		1-year mortality
	AUC (95% CI)	p value	AUC (95% CI)	p value
Comorbidities with ASA	0.022 (-0.027 to 0.071)	0.38	0.024 (-0.013 to 0.061)	0.21
Comorbidities with CCI	-0.016 (-0.07 to 0.038)	0.55	-0.061 (-0.099 to -0.022)	0.002
Comorbidities with ECI	-0.031 (-0.068 to 0.006)	0.10	-0.047 (-0.075 to -0.018)	0.002

AUC = area under the curve; ASA = American Society of Anesthesiologists; ECI = Elixhauser comorbidity index; CCI = Charlson comorbidity index.

Discussion

Although proper risk adjustment has received considerable attention in orthopaedics, evaluation of the three leading risk-adjustment indices (ASA, CCI, and ECI) has been largely limited to the immediate perioperative period and has never, to our knowledge, been simultaneously performed in the same patient population. As such, the best risk-adjustment strategy is unknown. Furthermore, the performance of claims-based risk adjustment in orthopaedic surgery has not been assessed in the ICD-10 era. In this study, we found that the CCI and ECI outperformed the ASA score in assessing 1-year mortality risk after hip fracture surgery. Our results also validated the performance of these indices for orthopaedic research in the ICD-10 era, showing their accuracy to be equivalent or superior to that observed previously in the ICD-9 era [33, 43]. Finally, we demonstrated that these comorbidity scores can outperform individual comorbidities and patient demographics that are frequently evaluated.

Limitations

This study has several limitations. First, this study specifically focused on hip fractures as a representative orthopaedic procedure given the high mortality risk [5, 22, 26, 53] and known influence of comorbidities on outcomes [20, 48, 49]. Although the relative performance of these robust indices would not be expected to markedly differ among procedures (particularly those with a similarly high mortality rate), the generalizability of these results to other orthopaedic procedures is nevertheless uncertain. Next, because of the infrequency of inpatient mortality (only 28 deaths, or 2% of patients), we were unable to evaluate the performance of these indices for inpatient death. The ASA may perform better in this setting than for longer-term outcomes [19, 38, 39].

Further, our study focused only on mortality. However, given that mortality is almost always a function of other complications and is the most important event to avoid, it is perhaps the single best all-encompassing marker of operative risk and has been used extensively in prior work on this subject [9, 18, 34, 36, 43]. Although there were no missing data given the study design, comorbidity indices are always subject to underperformance if patients have undocumented or undiagnosed conditions. Notably, this bias works more strongly against the ECI/CCI than the ASA, so the results could be even more in favor of the ECI/CCI if applied in a theoretical patient sample where 100% of all medical diagnoses were documented. Relatedly, institutional data are likely to capture more diagnoses than database studies, which may lack data on all of the specific comorbidities included in these indices [1, 19, 25, 38, 39]; be limited in the total number of comorbidities documentable (as low as 7-9) [10, 21]; be only able to capture diagnoses during the index hospitalization (less acute comorbidities may be underreported) [3, 23]; and muddy the distinction between chronic comorbidities and adverse events [36]. Consequently, the results of the current work are likely more generalizable to both institutional research and clinical scenarios than prior large database studies, while these results are likely less generalizable to big-data settings. Finally, all facilities were located in the same geographic region. However, the performance of these indices should not substantially differ as a function of geography; in fact, the updated (that is, ICD-10) scores were derived from data from a single province in Canada (Alberta) [45], the original CCI from a single hospital in Connecticut [8], and the original ECI from a single US state (California) [14], yet have been used successfully on an international basis [57].

Analysis of Comorbidity Indices

The most important finding of this study was that the CCI and ECI outperformed the ASA in assessing 1-year mortality risk. The superior performance of the CCI compared with the ASA score contrasts what has previously been reported in the perioperative period, where the ASA may be superior for inpatient or 30-day mortality after orthopaedic surgery [19, 38, 39]. These differences may not be surprising when considering the CCI was originally developed specifically to assess 1-year mortality risk [8] while the ASA was designed to stratify immediate perioperative risk [47]. Consistent with this time-varying performance, there was no difference (with the numbers available in this study) in CCI and ASA at 90 days, while the CCI was more accurate than the ASA at 1 year. Taken together, these results may reveal the importance of time when using certain risk-adjustment measures. Interestingly, in contrast to the CCI, the ECI performed roughly equivalently at 90 days and 1 year, with higher accuracy than the ASA at both timepoints. When taken in conjunction with the fact that the ECI was originally designed for [14, 45] and has been validated in the inpatient setting (for example, an AUC of 0.759 for inpatient complications after THA [40]), these results indicate that the ECI may be a reliable comorbidity index across the entire postoperative period. The high degree of specification of the ECI (31 comorbidities) might be responsible for this strong and stable performance across time periods (with AUCs from 0.755-0.759), as well as surgical fields [9, 18, 27, 34, 52]. Notably, however, in contrast to the inpatient setting, where the ECI appears to be more accurate than the CCI for mortality [24, 31, 34, 40, 41], we found no differences in performance between the CCI and ECI at 90 days or 1 year. This improved performance of the CCI at longer time points is again consistent with its original design [8]. Collectively, these results suggest that either the CCI or ECI should be used over the ASA to adjust for the risk of longer-term outcomes after orthopaedic surgery, at least in the ICD-10 era.

To our knowledge, this study is also the first to validate the ECI and CCI in orthopaedic surgery during the ICD-10 era. The ICD-10 has the merit of containing more codes with substantially greater specificity than the ICD-9 (approximately 68,000 diagnosis codes in ICD-10 versus approximately 13,000 codes in ICD-9), allowing for a richer, more accurate classification of clinical information [6, 54, 55]. However, the introduction of entirely new codes from which the ECI and CCI are calculated [45] carried the potential to substantially alter their performance. In this study, we found that the performance of the ECI (AUC of 0.755) appeared to be highly consistent with what was seen in the ICD-9 era. For instance, Mehta et al. [33] found an AUC of 0.755 for the ECI for 1-year mortality after common surgical procedures, while Chu et al. [9] found an AUC of 0.777 for 1-year mortality in patients hospitalized with common medical conditions. The CCI (AUC of 0.769), in contrast, may be improved in the ICD-10 era. In the only prior analysis examining the ASA and CCI for 1-year mortality in orthopaedic surgery, Quach et al. [43] reported an AUC of 0.607 for the CCI after hip fracture surgery. The improvement demonstrated in the current study may be at least partially explained by ICD-10 codes enabling more accurate documentation of specific comorbidities that are most associated with mortality risk. In line with this, the similar performance of the ECI between ICD-10 and ICD-9 might be because this index already included a substantial number of highly specific diagnoses (31 comorbidities), leaving less room for improvement from increased diagnostic code specificity.

Reference Performance

Critically, both the CCI and ECI outperformed models composed of basic patient characteristics and many comorbidities commonly considered in the preoperative workup of patients with hip fractures. Although studies have suggested that demographic factors (most notably age) may be more accurate for adverse outcomes than certain comorbidity indices in select databases [38, 39], our results indicate that in the ICD-10 era and in a clinical setting, a properly chosen comorbidity index can more effectively describe a patient’s physical state. Moreover, our results highlight the particular strength of comorbidity scores for assessing the risk of longer-term outcomes. Both the CCI and ECI, even without additional accounting for age and gender, were able to outperform a fully developed model of seven common comorbidities, age, gender, and year in assessing 1-year mortality risk. In contrast, the most commonly considered risk stratification score perioperatively (ASA score) had inferior longer-term performance. These results underscore the difficulty of preoperative risk stratification based on longer-term outcomes and reveal the potential of preoperative comorbidity indices to aid clinical practice, because the CCI and ECI were able to accurately predict 1-year mortality in more than 75% of patients.

Notably, at present, the ASA may be more readily available than the CCI and ECI, which raises the question of whether the differences in accuracy seen in this study are clinically meaningful. First, based on current standards, the ASA (AUC of 0.685) demonstrated “poor” or “unacceptable” accuracy for 1-year mortality, while the CCI and ECI both were on the upper half of “fair” or “acceptable” performance [15, 30, 34, 46]. Ultimately, an AUC threshold of 0.750 has been proposed for what is clinically useful, which both the CCI and ECI, but not the ASA, surpassed [15]. To further help contextualize the performance of these measures, we also calculated the relative improvement from a base model, similar to prior studies [27, 34]. Through this analysis, we showed that the CCI provided a 112% improvement in accuracy relative to the ASA for 1-year mortality, while the ECI yielded a 93% relative improvement compared with the ASA. Taken together, these differences clearly highlight the potential benefit that could exist from a clinical calculator in electronic medical records that provides these scores at bedside. As the CCI and ECI scores were calculated algorithmically directly from data exported from our electronic medical record (Epic), this is certainly technologically feasible, and future efforts may be warranted to develop such applets (that is, automatic calculators built into electronic medical record systems).

Conclusion

Although appropriate risk adjustment is vital to both clinical practice and orthopaedic research, which comorbidity index to use for risk assessment beyond the immediate perioperative period has yet to be identified. This study demonstrated that both the CCI and ECI substantially outperform the ASA for 1-year mortality risk after hip fracture surgery. It also validates that the CCI and ECI can be used reliably in the ICD-10 era. By selecting a comorbidity index that more accurately assesses mortality after hip fractures, patients, family members, anesthesiologists, and surgeons may be able to make more informed decisions about what surgical intervention should be performed and anticipated outcomes. If other studies from additional hospital systems confirm these findings, as would be expected because of the objective nature of these indices, the CCI or ECI should likely be used for institutional orthopaedic research involving 90-day outcomes and beyond. Moreover, in light of the potential for these indices to provide superior clinical benefit compared with the ASA, electronic medical records may consider creating applets to calculate these scores.