Improving the Performance of Risk-adjusted Mortality Modeling for Colorectal Cancer Surgery by Combining Claims Data and Clinical Data

Abstract

Objectives

The objective of this study was to evaluate the performance of risk-adjusted mortality models for colorectal cancer surgery.

Methods

We investigated patients (n=652) who had undergone colorectal cancer surgery (colectomy, colectomy of the rectum and sigmoid colon, total colectomy, total proctectomy) at five teaching hospitals during 2008. Mortality was defined as 30-day or in-hospital surgical mortality. Risk-adjusted mortality models were constructed using claims data (basic model) with the addition of TNM staging (TNM model), physiological data (physiological model), surgical data (surgical model), or all clinical data (composite model). Multiple logistic regression analysis was performed to develop the risk-adjustment models. To compare the performance of the models, both c-statistics using Hanley-McNeil pair-wise testing and the ratio of the observed to the expected mortality within quartiles of mortality risk were evaluated to assess the abilities of discrimination and calibration.

Results

The physiological model (c=0.92), surgical model (c=0.92), and composite model (c=0.93) displayed a similar improvement in discrimination, whereas the TNM model (c=0.87) displayed little improvement over the basic model (c=0.86). The discriminatory power of the models did not differ by the Hanley-McNeil test (p>0.05). Within each quartile of mortality, the composite and surgical models displayed an expected mortality ratio close to 1.

Conclusions

The addition of clinical data to claims data efficiently enhances the performance of the risk-adjusted postoperative mortality models in colorectal cancer surgery. We recommended that the performance of models should be evaluated through both discrimination and calibration.

INTRODUCTION

Measuring the quality of healthcare services facilitates decisions about the selection of providers, the creation of financial incentives, and quality improvement for consumers, purchasers, and providers. There is an increasing demand for precise comparative information about service quality. In this regard, risk-adjustment methods are frequently used for the evaluation of the quality of patient outcomes [1,2]. Risk-adjustment data are typically derived from claims data and clinical sources [3]. Claims data have the advantages of ready availability, low cost, and large volume. These advantages warrant the use of claims data for risk-adjustment [3,4].

In contrast, clinical data are more informative with regard to a patient's pathophysiological risk of death, rendering clinical data more accurate than claims data and a more popular basis for the evaluation of quality [3-8]. Some investigators have suggested an alternative approach to the use of data sources in risk adjustment: combining clinical data and claims data can overcome any disadvantages of one or the other and strengthen the advantages of both data sources [9,10].

To date, studies combining claims and clinical data have focused only on cardiovascular disease [4,8,11]. Although cancer is the most common cause of mortality in South Korea, a lack of evidence exists comparing the performance of risk-adjustment cancer-mortality models based on data sources [7,12-14]. Furthermore, the discriminatory power of various models has not been evaluated by a probability method (e.g., pair-wise test) in numerous studies. Many comparative studies measuring the performance of risk-adjustment models are based on discrimination alone, not with calibration simultaneously, which can lead to avoidable controversy [1,3,4,6-9,11].

Colorectal cancer had the third highest age-adjusted incidence (36.9 per 10 000 population) and was the fourth highest cause of age-adjusted mortality (12.0 per 10 000 population) in South Korea in 2011 [15]. Colorectal cancer was increasing in incidence relative to other types of cancer and as a cause of mortality in both men and women from 1999 to 2007 in South Korea [16]. The recent increase in the colorectal cancer incidence and mortality in South Korea means that measuring the quality of colorectal cancer surgery is of extreme importance. The purpose of this study was to improve the performance of risk-adjusted mortality models for colorectal cancer by combining claims and clinical data. Additionally, we attempted to compare both the discrimination and calibration performance of the models.

METHODS

Data Sources

The study sample consisted of patients who had undergone colorectal cancer surgery (colectomy, colectomy of the rectum and sigmoid colon, total colectomy, and total proctectomy) at five teaching hospitals in the Seoul metropolitan area during 2008. A total of 652 patients were treated who had a principal diagnosis of colorectal cancer according to the International Classification of Disease 10 code (C180-C189, C190, C200, D010-D019, and D099) and procedure codes identified by the Korean National Health Insurance billing system (Q1261, Q1262, Q2671, Q2672, Q2673, Q2679, QA671, QA672, QA673, QA679, Q2921, Q2922, Q2923, Q2924, QA921, QA922, QA923, QA924, Q2925, Q2926, QA925, and QA926). The sample size for comparison of the discriminatory power of the models was calculated with the assumption that type I error =0.05 and type II error =0.2. The mean discriminatory power of the claims data models (c=0.739) and clinical data models (c=0.881) in previous studies were applied [1,3-8,11,17,18]. Based on a pilot study, it was assumed that the Pearson correlation coefficient between the mortality probabilities of the two types of models was 0.8 [19] and that the ratio of the number of cases of survival to deaths was 27.3.

Measures of Outcome and Prognostic Factors

The outcome was defined as death occurring within 30 days of an operative procedure, either in the hospital or after discharge from the hospital, regardless of the cause [20,21]. From the claims data, we extracted prognostic factors including age, sex, health insurance, associated surgery, admission through the emergency room, lymph node dissection, and comorbidities (diabetes mellitus, hypertension, liver disease, other cardiac arrhythmia, and secondary cancer). Clinical prognostic factors were selected from the medical records by a full-time nurse reviewer who was trained in data extraction and included TNM staging, history of past surgery, emergency surgery, body mass index, serum albumin levels, the American Society of Anesthesiologists (ASA) score, and perforation and obstruction of the colon. The prognostic factors were selected by three faculty members who practiced colorectal cancer surgery in academic teaching hospitals through a review of previous literature and exploratory statistical results [7,12-14,20-29]. Meetings for the selection of prognostic factors were held four times, selection of claims data and clinical data and a review of the results of the models were performed. Dividing the value of the variables was also based on clinical judgment.

Study Design and Statistical Analysis

To compare the performance of models according to their data sources, five models were constructed using candidate prognostic factors. The basic model used standard claims data including age, sex, health insurance, associated surgery, admission through the emergency room, lymph node dissection, and comorbidities. The TNM model used claims data plus TNM staging. The physiological model used claims data plus the ASA score, serum albumin levels, and body mass index. The surgical model used claims data plus emergency surgery, TNM staging, past surgery history, perforation of the colon, and obstruction of the colon. The composite model used claims data plus all the clinical data.

To avoid the use of secondary diagnoses that develop during hospital stays as prognostic factors, we used only the secondary diagnoses that occurred during the 12 months prior to the first admission. We used multiple logistic regression to develop models and bootstrapped entire data sets by repeating the steps 1000 times to prevent overfitting [30-32].

We examined the performance of the models according to two criteria: discrimination and calibration [33]. The discriminatory power of the models was evaluated using c-statistics, or areas under the receiver operating characteristic curve, according to Hanley-McNeil pair-wise tests [34]. The goodness of fit of each model was evaluated using Hosmer-Lemeshow statistics. To compare the power of calibration among the models, we first created log-log scatter plots of the models compared with top-flight discrimination models and calculated the observed/expected (O/E) ratios and 95% confidence intervals within each quartile of mortality for each of the models [3]. The quartiles represented four categories of estimated mortality risk. Subgroup analyses were performed by comparing the observed and expected 30 day postoperative mortality according to age, TNM staging, ASA score, and obstruction of the colon [20].

This study was approved by the institutional review board of the Seoul National University College of Medicine. All statistical analyses were performed using SAS version 9.1 (SAS Institute, Cary, NC, USA).

RESULTS

Of the total sample (n=652), 3.5% of patients died (n=23). The mean age was 64.2 years, and more than 62.5% of the patients were male. The distribution of surviving persons differed by age, secondary cancer, admission through the emergency room, lymph node dissection, emergency surgery, TNM staging, body mass index, ASA score, perforation of the colon, and obstruction of the colon (Table 1). The odds ratios of the prognostic factors by model are shown in Table 2. The odds ratios of other cardiac arrhythmia, secondary cancer, admission through the emergency room, perforation, and ASA score had p-values less than 0.05.

Table 1.

Distribution of surgical mortality by basic characteristics

Values are presented as n (%).

LN, lymph node; ASA, American Society of Anesthesiologists.

¹Fisher's exact test.

²Chi-squared test

Table 2.

Odds ratios of prognostic factors by model

LN, lymph node; ASA, American Society of Anesthesiologists.

^*p<0.05.

Figure 1 presents the c-statistics for each risk-adjustment model, showing that the physiological model (c=0.915), surgical model (c=0.922), and composite model (c=0.928) had impressive results. These three models displayed similar levels of improvement in discrimination, whereas the TNM model (c=0.87) was only slightly better than the basic model (c=0.86). The discriminatory power did not differ among the developed models (p>0.05) based on the paired comparison of c-statistics of the models using the Hanley-McNeil test.

Figure 1.

Discriminatory power (c-statistics) of five models.

The data on mortality generated by the basic, TNM, physiological, and surgical models showed greater clustering among moderate-risk patients and greater variability among the lower- and higher-risk patients compared with the composite model (Figure 2). The surgical model and physiological model were more similar to the composite model in terms of the expected mortality than were the basic and TNM models. This was reflected in the degree of difference between the fitted and reference lines, which means the calibration abilities compared to the composite model (slope=1).

Figure 2.

Log-log scatter plots of the five models.

Table 3 shows the O/E ratios of the five models by quartiles based on the expected mortality. In the lower-risk quartiles (quartiles A & B), all models overestimated the mortality risk (O/E ratio <1), whereas most models underestimated mortality in the higher-risk quartiles (quartiles C & D) (O/E ratio >1). Overall, the composite and surgical models displayed O/E ratios closer to 1 than did the other models. In quartile A, the composite model displayed an O/E ratio closest to 1, and the surgical model showed the next closest estimate. In quartile B, the TNM model displayed an O/E ratio closest to 1, with the basic model providing the next closest estimate. In quartile C, the surgical model displayed an O/E ratio closest to 1, and the composite model had the next closest value. In quartile D, the composite model displayed an O/E ratio closest to 1, with the surgical model displaying the next closest value. The subgroup analyses revealed that the expected mortality by age, TNM staging, ASA score, and obstruction of the colon was well within the 95% confidence limits of the observed outcome and did not differ among the subgroups.

Table 3.

Observed to expected ratio of mortality by the mortality quartile

O/E, observed/expected; LCL, lower 95% confidence limit; UCL, upper 95% confidence limit.

DISCUSSION

The level of model discrimination in this study (c=0.863-0.928) was relatively higher than the models examined in previous risk-adjustment studies of colorectal cancer mortality (c=0.707-0.848) [20,21,28]. This was evident in the claims-data models as well as in the model using a combination of claims data and clinical data. The level of model discrimination in this study may be effective because more relevant postoperative complication factors (physiologic factors and surgical factors) were selected. We used a minimum of 13 prognostic factors and a maximum of 19 to construct the risk-adjusted models. In previous studies, 4 to 13 factors have been used. We speculated that the relevance of the prognostic factors for mortality, rather than the number of prognostic factors, affected the discriminatory power of the models. According to studies by Steyerberg et al. [35], c-statistic levels are not necessarily proportional to the number of prognostic factors. Even with a large number of prognostic factors, the possibility of a low discriminatory power exists. In small data sets, the likelihood of this possibility increases.

Although the addition of the TNM staging data alone to the claims data resulted in only marginal improvement in discriminatory power, the addition of further clinical data led to remarkable improvements. The extent of cancer spread reflected in TNM staging may predict long-term outcomes (5-year or 10-year survival) more strongly than short-term outcomes (30-day or in-hospital operative mortality) [36,37]. However, physiological parameters, emergency surgery, and obstruction and perforation of the colon were important predictors of the short-term outcome [12,37,38]. In previous studies, the ASA score was used as a proxy for physiological condition [20,38].

Beyond the quality of surgery (i.e., completeness of resection, adequacy of nodal examination, utilization of adjuvant treatments, aggressiveness of postresection cancer follow up), multiple factors including anesthesia care, preoperative evaluation, nurse staffing and training, intensive care unit management and staffing, 24-hour in-house resident or physician coverage, and emergency response can affect the short-term outcome of cancer surgery [36]. Further studies are required to evaluate the performance of risk-adjusted long-term mortality models. Although the c-statistics for these models were not statistically different, subtle differences were evident, suggesting that a larger sample size may lead to more apparent differences.

The enhancement of the discriminatory power declined with the addition of clinical data to the claims data. The saturation of discriminatory power reflected in our results was similar to the findings of previous studies [4,9]. Pine and colleagues showed that the performance of the claims data model was enhanced by the addition of presentation on admission (POA) codes and limited laboratory data [4]. However, the addition of POA codes resulted in the greatest improvement in discrimination, and this enhancement declined with the addition of other factors. Fry and colleagues additionally showed that saturation in discrimination was provided by POA codes with the addition of numerical laboratory data [9]. According to Hall et al. [3], the cost per patient to obtain key clinical findings from the medical charts was 50 dollars in 2007; this cost must be balanced against the minimal enhancement of model performance derived from the collection of these data.

To select a model for the evaluation of quality of care, it is necessary to examine the levels of both discrimination and calibration. It is important to select precise risk-adjustment models when measuring quality, but testing discrimination alone is insufficient. It is difficult to differentiate the discriminatory power of models despite thorough pair-wise testing, and it would therefore be useful to select a suitable model by testing the calibration through the log-log scatter or O/E ratio. Our results demonstrated little statistical difference in the discriminatory power of the models examined here. The absolute levels differed only slightly among the physiological, surgical, and composite models, but were clearly differentiated by the calibration of the models. Our models appeared to overestimate mortality in the low-risk patients and to underestimate mortality among high-risk patients. This was similar to the results of Hall et al. [3], and careful research efforts are needed to identify the reasons for these biases.

There were several limitations of this study. First, the confidence intervals were greater than 10, which had p-values of the odd ratios of prognostic factors of less than 0.05. This may have been dependent on the limited sample size of deaths (23 of 652). Second, we used non-sequential data from five volunteer institutions. This can limit generalization to other populations. Third, we constructed parsimonious risk-adjustment models for colorectal cancer only. The prognostic factors were specific only to colorectal cancer surgery, which makes them inappropriate for application to other forms of cancer surgery. Third, we only considered secondary diagnoses that occurred during the 12 months prior to initial admission. We have not yet evaluated our approach to the definition of secondary diagnoses relative to previous POA coding algorithms such as those applied in New York and California. Fourth, a more extensive systematic training program on clinical data extraction is needed to perform precise extraction of variables from clinical records in South Korea.

In summary, our data suggest that the addition of clinical data to claims data could efficiently enhance the performance of risk-adjusted postoperative mortality models of colorectal cancer surgery. The addition of TNM data could contribute to improved performance in combination with other surgical data including the ASA score and emergency surgery. We additionally recommend that the performance of risk-adjustment models should be evaluated based on both discrimination and calibration. Testing only discrimination may result in an insufficient comparison of the performance of risk-adjustment models.