What Associations Exist Between Comorbidity Indices and Postoperative Adverse Events After Total Shoulder Arthroplasty?

Abstract

Background

Comorbidity indices like the modified Charlson Comorbidity Index (mCCI) and the modified Frailty Index (mFI) are commonly reported in large database outcomes research. It is unclear if they provide greater association and discriminative ability for postoperative adverse events after total shoulder arthroplasty (TSA) than simple variables.

Questions/purposes

Using a large research database to examine postoperative adverse events after anatomic and reverse TSA, we asked: (1) Which demographic/anthropometric variable among age, sex, and body mass index (BMI) has the best discriminative ability as measured by receiver operating characteristics (ROC)? (2) Which comorbidity index, among the American Society of Anesthesiologists (ASA) classification, the mCCI, or the mFI, has the best ROC? (3) Does a combination of a demographic/anthropometric variable and a comorbidity index provide better ROC than either variable alone?

Methods

Patients who underwent TSA from 2005 to 2015 were identified from the National Surgical Quality Improvement Program (NSQIP). This multicenter database with representative samples from more than 600 hospitals in the United States was chosen for its prospectively collected data and documented superiority over administrative databases. Of an initial 10,597 cases identified, 70 were excluded due to missing age, sex, height, weight, or being younger than 18 years of age, leaving a total of 10,527 patients in the study. Demographics, medical comorbidities, and ASA scores were collected, while BMI, mCCI and mFI were calculated for each patient. Though all required data variables were found in the NSQIP, the completeness of data elements was not determined in this study, and missing data were treated as being the null condition. Thirty-day outcomes included postoperative severe adverse events, any adverse events, extended length of stay (LOS, defined as > 3 days), and discharge to a higher level of care. ROC analysis was performed for each variable and outcome, by plotting its sensitivity against one minus the specificity. The area under the curve (AUC) was used as a measure of model discriminative ability, ranging from 0 to 1, where 1 represents a perfectly accurate test, and 0.5 indicates a test that is no better than chance.

Results

Among demographic/anthropometric variables, age had a higher AUC (0.587–0.727) than sex (0.520–0.628) and BMI (0.492–0.546) for all study outcomes (all p < 0.050), while ASA (0.580–0.630) and mFI (0.568–0.622) had higher AUCs than mCCI (0.532–0.570) among comorbidity indices (all p < 0.050). A combination of age and ASA had higher AUCs (0.608–0.752) than age or ASA alone for any adverse event, extended LOS, and discharge to higher level of care (all p < 0.05). Notably, for nearly all variables and outcomes, the AUCs showed fair or moderate discriminative ability at best.

Conclusion

Despite the use of existing comorbidity indices adapted to large databases such as the NSQIP, they provide no greater association with adverse events after TSA than simple variables such as age and ASA status, which have only fair associations themselves. Based on database-specific coding patterns, the development of database- or NSQIP-specific indices may improve their ability to provide preoperative risk stratification.

Level of Evidence

Level III, diagnostic study.

Introduction

With the recent proliferation of large database outcomes studies in total shoulder arthroplasty (TSA) research, various patient characteristics and comorbidity indices have been reported to correlate with adverse postoperative outcomes after TSA [10–13, 19, 28]. Variables associated with postoperative adverse events that have been identified include age, sex, body mass index (BMI), and individual comorbidities such as diabetes [11, 12]. In addition, in an attempt to capture an overall assessment of a patient’s comorbidity burden, indices have been developed such as the American Society of Anesthesiologists (ASA) physical status classification [18], as well as more complex existing indices that have been adapted for database research, including the modified Charlson Comorbidity Index (mCCI) [5] and the modified Frailty Index (mFI) [9], which are calculated based on individual patient comorbidities.

Although primary demographic variables such as age, sex, and BMI are commonly available, several studies using large clinical databases have used these more complex comorbidity indices for risk stratification in patients undergoing TSA [10–13, 20, 31]. It is unclear, however, whether these comorbidity indices provide greater discriminative ability in their associations with postoperative adverse events to aid clinicians in preoperative risk stratification. Furthermore, as the accurate determination of these indices relies on complete patient medical histories, the use of these indices based on information from large databases may be subject to potential concerns regarding data quality and completeness.

Therefore, within the setting of large database clinical outcomes research, this study sought to answer the following questions: (1) Which demographic/anthropometric variable among age, sex, and body mass index (BMI) has the best discriminative ability as measured by receiver operating characteristics (ROC) in its association with adverse events following TSA? (2) Which comorbidity index, among the American Society of Anesthesiologists (ASA) classification, the mCCI, or the mFI, has the best ROC in its association with adverse events following TSA? (3) Does a combination of the best demographic/anthropometric variable with the best comorbidity index provide better ROC than either variable alone?

Patients and Methods

Patient Sample

The study population was extracted from the American College of Surgeons National Surgical Quality Improvement Program (NSQIP) from the years 2005 to 2015. NSQIP is a multicenter surgical registry that includes a representative sample of patient data from more than 600 hospitals across the United States [1]. Clinical data is prospectively collected by trained clinical data reviewers from patient medical records, rather than administrative billing codes. Postoperative events are captured for 30 days postoperatively, regardless of hospital discharge date. Data elements are frequently audited to ensure data quality, with low reported interrater disagreement rates [1]. Use of this database is well established in orthopaedic outcomes research, and specifically for examining TSA outcomes [7, 11–13, 19, 28, 32]. As a research-oriented database with prospectively collected data, the NSQIP was chosen for this study as it has been demonstrated to be superior to other administrative databases in its determination of patient BMI and a number of medical comorbidities [3, 14].

Patients who underwent TSA were identified in the NSQIP using Current Procedural Terminology (CPT) code 23472, which includes both anatomic and reverse TSA. A total of 10,597 TSA cases were initially identified. Exclusion criteria were cases missing age (0 patients), sex (six patients), height (41 patients), and weight (23 patients), and those younger than 18 years of age (0 patients) (Fig. 1). Therefore, a total of 10,527 TSA patients were included in the current study.

Fig. 1.

This study cohort flow chart demonstrates the number of patients meeting inclusion and exclusion criteria.

Demographic Variables and Comorbidity Indices

Patient demographic information directly extracted from NSQIP included age and sex. BMI was calculated using height and weight data. The average patient age was 69.2 ± 10.0 years (mean ± SD), 56% of the patients were women, and the average BMI was 30.9 ± 6.8 kg/m². Medical histories were also directly extracted, which enabled the determination of the derived comorbidity indices, including the mCCI and the mFI. With both the mCCI and the mFI, higher index scores indicate a greater comorbidity burden. All required data variables for the calculation of mCCI and mFI were found in the NSQIP (Table 1). The completeness of the data elements for each individual patient was not determined in this study, as our research questions revolved around the utility of these comorbidity indices in the setting of large databases such as the NSQIP. Therefore, in the current study, missing data were treated as being null. For example, for a specific medical condition, patients in the NSQIP who did not have a documented history of that condition in NSQIP, were assumed to not have the condition. This is similar to previous studies examining the discriminative ability of these comorbidities using the NSQIP [22, 23]. In addition, ASA scores were also collected for each patient. The median ASA score was 3 (interquartile range, 2–3), mCCI was 0 (0–0), and mFI was 0.09 (0–0.09) (Table 2).

Table 1.

Comorbidity index definitions

Table 2.

Patient characteristics

Outcome Variables

Severe adverse events were defined as the occurrence of cardiac arrest, coma, death, deep vein thrombosis, myocardial infarction, postoperative intubation, pulmonary embolism, return to the operating room, stroke, or sepsis. Any adverse event included the occurrence of a severe adverse event and/or any other postoperative adverse event. With these definitions, 5.9% of patients experienced one or more postoperative adverse events, and 2.4% of patients had a severe adverse event (Table 3).

Table 3.

Incidence of adverse events in the study population (n = 10,527)

In addition, secondary outcome variables included extended hospital length of stay (defined as a hospital stay greater than the 75th percentile, or > 3 days), and discharge to a higher level of care than home (defined as discharge to a separate acute care, a skilled care facility that was not home, an unskilled facility that was not home, or a rehabilitation facility). In terms of these hospital metrics, 8.7% of patients had an extended length of hospital stay, and 12.0% of patients were discharged to a higher level of care (Table 3).

Statistical Analysis

Receiver operating characteristics (ROC) curves were generated for each demographic variable or comorbidity index against their diagnostic ability for the study outcome variables. ROC curves plot the sensitivity of a diagnostic or prognostic test on the y-axis, and one minus the specificity on the x-axis. Area under the curve (AUC) statistics were determined for each ROC curve, and represent the discriminative ability of a test, which in this case is a measure of how well a model can differentiate between those who are or are not likely to experience an adverse outcome [6, 30]. AUC values range from 0 to 1, where an AUC 1 represents a perfectly accurate test, and an AUC of 0.5 indicates a test that is no better than chance [17, 30]. An AUC of 0.7 can generally be considered fair or moderate discriminative ability. In the context of this study, the numeric AUC value can be further conceptualized as the probability that a randomly chosen patient who experienced an adverse event has a more abnormal demographic variable or comorbidity index than a randomly chosen patient who did not experience that adverse event [17].

AUCs for the pairing of each demographic variable or comorbidity index with a study outcome variable were compared for statistically significant differences using 95% confidence intervals (CI). If no overlaps in the 95% CI were observed, then the difference in AUCs was deemed to be statistically significant. If there was overlap in the 95% CI, we used the DeLong method for comparing AUCs of correlated ROC curves [8]. Note that while nonoverlapping 95% CIs means that p < 0.05 by definition, the reverse is not necessarily true – the p value may still be < 0.05 even with overlapping 95% CIs [27].

Statistical analyses were performed using Stata version 13.1 (StataCorp, LLC, College Station, TX, USA). The level of significance was set at a two-sided level of p < 0.05. Use of the publicly available NSQIP dataset was found to be exempt from our institutional review board.

Results

ROC of Demographic/Anthropometric Variables and Adverse Events After TSA

Among demographic variables, AUC analysis of the ROC curves showed that age demonstrated the best discriminative ability (AUC 0.609; 95% CI, 0.585–0.634) in its association with the occurrence of any postoperative adverse event compared with BMI (AUC 0.546; 95% CI, 0.522–0.570) and sex (AUC 0.541; 95% CI, 0.522–0.561) (Fig. 2). The difference was statistically significant between age versus BMI (p < 0.001) and age versus sex (p < 0.001), but not between BMI versus sex (p = 0.752). Similarly, age had higher discriminative ability than BMI and sex for severe adverse events, extended length of hospital stay, and discharge to a higher level of care (Table 4). It should be noted, however, that the AUC values for all demographic/anthropometric variables were below the range of what would be considered fair discriminative ability, and just above chance.

Fig. 2.

This figure shows the area under the ROC curve for any adverse event, demographic factors.

Table 4.

Area under the ROC curve analyses; demographic factors summary

ROC of Comorbidity Indices and Adverse Events After TSA

Among comorbidity indices, both ASA (AUC 0.607; 95% CI, 0.587–0.627) and mFI (AUC 0.586; 95% CI, 0.565–0.607) outperformed mCCI (AUC 0.555; 95% CI, 0.536–0.575) in their discriminative ability for any postoperative adverse event (p < 0.001 for ASA versus mCCI, p < 0.001 for mFI versus mCCI) (Fig. 3). Similarly, the ASA and mFI were no different from each other, but superior to the mCCI in their associations with severe adverse events (p = 0.443 for ASA versus mFI, p < 0.001 for both ASA and mFI vs. mCCI) and discharge to a higher level of care (p = 0.358 for ASA versus mFI, p < 0.001 for both ASA and mFI versus mCCI) (Table 5). For extended length of postoperative hospital stay, the ASA was superior to both mFI and mCCI (both p < 0.001) in terms of AUC. Again, these AUC values all fall below the range of what is considered to be fair discriminative ability.

Fig. 3.

Shown here is the area under the ROC curve for any adverse event, comorbidity indices.

Table 5.

Area under the ROC curve analyses; comorbidity indices summary

Combination of the Best Demographic/Anthropometric Variable with the Best Comorbidity Index Compared With Either Alone

Finally, the best performing demographic variable (age) was compared with the best performing comorbidity index (ASA). The AUCs for age and ASA were no different (p > 0.05) for all outcomes except for discharge to a higher level of care, for which age had a higher AUC (0.727 versus 0.630, p < 0.001) (Table 6). The combined variable of ASA + age (AUC = 0.645) was found to have greater discriminative ability than ASA (AUC = 0.607; p < 0.001) or age (AUC = 0.609; p < 0.001) in its association with any postoperative adverse event (Fig. 4). Similarly, ASA + age had larger AUCs for its association with extended length of hospital stay (AUC = 0.672) and discharge to higher level of care (AUC = 0.752) than either ASA or age alone (all p < 0.001). It should be noted, however, that with the exception of the association between age and ASA + age and discharge to higher level of care, that the AUC values continue to be below what is considered to be fair discriminative ability.

Table 6.

Summary of area under the ROC curve analyses

Fig. 4.

In this figure, we show the area under the ROC curve for any adverse event, combination.

Discussion

Large clinical research databases such as the NSQIP are commonly used in shoulder arthroplasty research for its large sample sizes, providing the statistical power necessary to examine rare postoperative outcomes. In risk stratification models using these databases, researchers often calculate comorbidity indices such as the mCCI and the mFI as a measure of overall medical comorbidity burden. Relative to more simple variables such as patient demographics, BMI, and ASA class, however, it is unclear whether such comorbidity indices provide any greater risk stratification or discriminative ability for postoperative adverse events. In this study using TSA cases from the NSQIP database with ROC analysis, we found that no single patient demographic variable, BMI, ASA class, or comorbidity index provided better than a fair discriminative ability in its association with postoperative adverse events, and that among measures of a patient’s overall comorbidity burden, ASA class outperformed the more complex mCCI index.

There are several important limitations to this study. First, the mCCI and mFI are derived from several variables from the patient medical history, and data limitations such as incorrectly coded or missing information may affect the calculated index [21]. This is made evident by the apparent floor effects seen in both the mCCI and mFI distribution among the study cohort. In particular with the NSQIP, given its continued evolution and maturation as a research database over time, systematic changes in its coding have led to missing data for a number of variables that are used to determine the mCCI and the mFI [29]. There may also be discrepancies in the coding and distribution of an index like the mCCI across different research databases [26]. While the treatment of missing data in the NSQIP is beyond the scope of this study, patients who were not coded to have a history of a specific medical condition were assumed to not have that condition for the purposes of calculating their comorbidity index, which is in line with previous studies comparing the discriminative abilities of comorbidity indices using NSQIP [22, 23, 25]. We believe an understanding of this limitation supports our finding that comorbidity indices like the mCCI and the mFI, as derived from the NSQIP database, are inadequate in clinical outcomes research. In other words, for all but one of the study outcomes, the highest AUC for any demographic variable or comorbidity index was below the level generally considered to be fair discriminative ability (AUC ≥ 0.70). Therefore, it is likely that there are additional components of patient morbidity risk that are not being captured adequately in these large database models. In addition to coding deficiencies, these factors may include surgical indication, preoperative functional scores, surgeon experience, and perioperative protocols, such as tranexamic acid use, venous thrombosis prophylaxis, and postoperative care pathways that are not captured in the NSQIP. Furthermore, an additional limitation is the lack of postoperative events beyond 30 days, which is a limitation inherent to the NSQIP database. In addition, TSA cases were extracted from the NSQIP using CPT coding, which meant that anatomic and reverse TSA cases could not be differentiated. However, since anatomic and reverse TSA are similar in that both require glenoid and humeral preparation and prostheses, in addition to the focus of the study being on general postoperative adverse events rather than shoulder-specific outcomes, this is likely not a significant limitation. Finally, the use of the AUC statistic is not without its own limitations. Though discrimination is an important aspect of model performance, it does not provide calibration and likelihood-based information from the model [6]. Nonetheless, AUC statistics through ROC analyses provide a valuable means of assessing the discriminative ability of the variables examined in this study, which has not been previously demonstrated in studies using the NSQIP database to examine TSA outcomes.

Among demographic and anthropometric variables, age was superior to BMI and sex for all study endpoints. This is in line with previous large database studies examining TSA outcomes. Using the NSQIP from 2006 to 2011, Waterman et al. [31] found with multivariate regression that age (as a continuous variable) and history of cardiac disease were the only variables associated with postoperative mortality. Similarly, using the Nationwide Inpatient Sample database from 2000 to 2008, Griffin et al. [15] found that age ≥ 80 years was associated with a small increase in inhospital mortality rate of 0.4% compared with younger patients. Based on these results, among demographic variables, age alone may be adequate in most cases for simple general risk stratification, as a rough measure of overall health and physiological reserve, although it had at best fair discriminative ability based on our analysis.

Among comorbidity indices, ASA was found to be the most discriminative of postoperative adverse events, although, again, the discriminative ability of using ASA was no better than age alone. ASA class has been shown to be associated with postoperative medical adverse events, blood transfusions, and prosthesis failure after TSA [2, 20]. However, in the aforementioned study by Waterman et al. [31], after taking into consideration other patient factors and medical conditions in a multivariate regression model, ASA was not associated with any postoperative adverse events. Nevertheless, despite the ASA being a subjective index assigned by the anesthesiologist, it outperformed the more complex indices in identifying patients in the NSQIP with postoperative adverse outcomes after TSA.

In terms of mCCI, despite its use in TSA outcomes research [4, 10–12, 16], the mCCI demonstrated the worst discriminative ability across all study outcomes relative to the mFI and the ASA. In a single center study of 127 TSA patients, Chalmers et al. [4] did find that among age, BMI, and CCI, that CCI was the only variable that was associated with postoperative complications. However, it is difficult to compare the CCI in the setting of a retrospective single center study with complete patient data with its modified use in a large database study that is subject to missing data and other coding errors. Regardless, since the CCI was originally designed as a prognostic indicator for mortality risk, it may not be sensitive enough to capture the lower comorbidity burden in patients undergoing elective orthopaedic surgery relative to other medical disciplines. For example, medical conditions such as hypertension, cardiac arrhythmias, and thyroid dysfunction, among others, are given no points in the mCCI calculation. To that point, in addition to the potential impact of missing data, in the current TSA study population, the mCCI exhibited significant floor effects with 77% of the cohort having an mCCI of 0. Therefore, CCI and its derivatives should not be used as the sole measure of overall comorbidity burden in studies examining patients undergoing TSA. Finally, mFI was no different from ASA in its discriminative ability for postoperative adverse events and discharge to a higher level of care. This is consistent with a similar study by Ondeck et al. [22], comparing the discriminative ability of ASA and mFI in postoperative adverse events after lumbar fusion.

By combining age with ASA, the ROC characteristics improved to a level that was superior to either age or ASA alone for postoperative adverse events. This suggests that age and ASA may capture slightly different aspects of a patient’s morbidity risk. However, even the combination of the best-performing variables in this study provided at best only a fair discriminative ability. At the current AUC levels found in this study, the utility of using age and/or ASA alone in building TSA risk stratification models using the NSQIP is questionable, despite their superiority over mCCI and mFI. To enhance the ability of providers to use large databases such as the NSQIP to effectively stratify risk, we believe that database- and NSQIP-specific indices should be developed, rather than modifying existing indices like the CCI and FI to work on NSQIP data. A NSQIP-specific index could theoretically account for the pattern of missing data in the database, perhaps placing more weight on variables with more complete information, while discounting variables with high rates of missing data. In addition, recent studies have examined various ways of handling missing laboratory data in NSQIP, specifically the use of multiple imputation as a statistical method of approximating missing data [24, 25], which may warrant future study in its application for examining postoperative TSA outcomes.

In summary, despite the use of derived comorbidity indices such as the mCCI and mFI in TSA outcomes research using NSQIP, the results of this study suggest that they have no greater discriminative ability for adverse events after TSA than more easily-obtainable variables such as age and ASA classification. Importantly, no demographic factor, combination of factors, or comorbidity index provided better than fair discrimination in our analysis. We believe there is questionable utility to using existing indices modified to adapt to NSQIP data, and future research efforts should be directed toward the development of database- and NSQIP-specific comorbidity indices based on its own data coding patterns and idiosyncrasies.