Comorbidity Assessment in the National Cancer Database for Patients With Surgically Resected Breast, Colorectal, or Lung Cancer (AFT-01, -02, -03)

Abstract

Purpose:

Accurate comorbidity measurement is critical for cancer research. We evaluated comorbidity assessment in the National Cancer Database (NCDB), which uses a code-based Charlson-Deyo Comorbidity Index (CCI), and compared its prognostic performance with a chart-based CCI and individual comorbidities in a national sample of patients with breast, colorectal, or lung cancer.

Patients and Methods:

Through an NCDB Special Study, cancer registrars re-abstracted perioperative comorbidities for 11,243 patients with stage II to III breast cancer, 10,880 with stage I to III colorectal cancer, and 9,640 with stage I to III lung cancer treated with definitive surgical resection in 2006-2007. For each cancer type, we compared the prognostic performance of the NCDB code-based CCI (categorical: 0 or missing data, 1, 2+), Special Study chart-based CCI (continuous), and 18 individual comorbidities in three separate Cox proportional hazards models for postoperative 5-year overall survival.

Results:

Comorbidity was highest among patients with lung cancer (13.2% NCDB CCI 2+) and lowest among patients with breast cancer (2.8% NCDB CCI 2+). Agreement between the NCDB and Special Study CCI was highest for breast cancer (rank correlation, 0.50) and lowest for lung cancer (rank correlation, 0.40). The NCDB CCI underestimated comorbidity for 19.1%, 29.3%, and 36.2% of patients with breast, colorectal, and lung cancer, respectively. Within each cancer type, the prognostic performance of the NCDB CCI, Special Study CCI, and individual comorbidities to predict postoperative 5-year overall survival was similar.

Conclusion:

The NCDB underestimated comorbidity in patients with surgically resected breast, colorectal, or lung cancer, partly because the NCDB codes missing data as CCI 0. However, despite underestimation of comorbidity, the NCDB CCI was similar to the more complete measures of comorbidity in the Special Study in predicting overall survival.

INTRODUCTION

Health services research commonly leverages data from large cancer registries to examine practice patterns,^1-4 disparities,^5-7 and outcomes.^8-10 Such studies rely on accurate and complete measurement of comorbidities to account for patient differences or to understand the impact of comorbidities on clinical outcomes. Furthermore, comorbidities can affect treatment decision making, toxicity, and overall survival (OS),^11-17 making accurate measurement central to research. In the National Cancer Database (NCDB), which captures approximately 70% of newly diagnosed patients in the United States,^18,19 comorbidity is assessed using the Charlson-Deyo Comorbidity Index (CCI)^20,21 mapped from up to 10 International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) diagnosis codes typically from discharge abstracts or billing face sheets.²² The accuracy of the CCI variable available in the NCDB is often perceived to be limited, but data regarding its accuracy are limited.

One recent study²³ compared the prevalence of comorbidities reported in the NCDB with comorbidities in a separate Surveillance, Epidemiology, and End Results (SEER)-Medicare cohort of matched patients with breast, colorectal, or lung cancer. The comorbidities reported in the NCDB and SEER-Medicare database were similar overall except for congestive heart failure, chronic pulmonary disease, and renal disease, which were consistently underreported in the NCDB.²³ Although this study compared comorbidities in matched cohorts, a direct evaluation of comorbidity assessment in the NCDB using data from the same cohort of patients was not possible.

As part of an American College of Surgeons Commission on Cancer (CoC) Special Study,^24,25 comorbidities were re-abstracted from original medical records for more than 30,000 patients with surgically resected breast, colorectal, or lung cancer. The Special Study thus provided a unique opportunity to directly evaluate comorbidity assessment in the NCDB. Therefore, the purpose of this study was to determine the accuracy of comorbidity assessment in the NCDB compared with a gold standard of medical record abstraction. In addition, we compared the performance of three different methods for adjusting for comorbidity in postoperative survival models.

PATIENTS AND METHODS

Data Source

The NCDB, sponsored by American College of Surgeons and the American Cancer Society, is a clinical oncology outcomes database sourced from hospital-based cancer registries from more than 1,500 CoC-accredited programs.^18,19 The Special Study mechanism evaluates patient care, sets benchmarks, and provides feedback to improve patient care in CoC-accredited programs. The 2015 Special Study examined surveillance and recurrence for patients with surgically resected breast, colorectal, or lung cancer. Data collected during the Special Study supplemented existing NCDB data. De-identified data, which do not identify patients, hospitals, or providers, were provided to investigators according to Health Insurance Portability and Accountability Act guidelines. This study was considered exempt by the institutional review boards.

Patient Population

Patients age 18 years or older diagnosed with stage II to III breast cancer, stage I to III colorectal cancer, or stage I to III non–small-cell lung cancer who were treated with definitive surgery in 2006-2007 were eligible. Ten eligible patients for each cancer type were randomly selected from each facility. For breast cancer, sampling was limited to women and stratified by stage (7 women, stage II; 3 women, stage III) to approximate the national distribution.²⁶ For facilities with fewer than 10 eligible patients per cancer type, data for all eligible patients were abstracted. Randomly selected patients with postoperative residual disease or unavailable records or those who were lost to follow-up within 90 days after surgery were excluded. Excluded patients were replaced with randomly selected eligible patients from the same facility if available. Patients were observed for 5 years from the time of initial definitive surgery or until death, whichever came first. The study years 2006-2007 were selected to allow for 5-year follow-up.

Data Collection and Measurement

Demographic and clinical characteristics were obtained from the NCDB. Comorbidity was assessed by using both existing NCDB data and new Special Study data. NCDB data were originally abstracted from ICD-9-CM codes from discharge abstracts or billing face sheets at the time of the primary surgery. To re-abstract comorbidity information for the Special Study, cancer registrars reviewed the medical record from 30 days before up to 90 days after surgery including notes from surgeons, anesthesiologists, oncologists, and primary care providers, which provided a more thorough assessment of comorbidity than required for the NCDB. The Special Study evaluated 18 comorbidities: 14 selected from the CCI^20,21 and 4 selected from the Adult Comorbidity Evaluation-27 (ACE-27)^17,27 (ie, morbid obesity, other neurologic conditions, psychiatric disorder, history of or active substance abuse). Substance abuse included alcohol and illicit drugs but did not include tobacco use. These additional comorbidities from the ACE-27 were abstracted to assess whether their inclusion would increase the ability to predict survival.

Two CCI scores were evaluated for each patient, which weights comorbidities according to their relative risk of death.^20,21 First, we obtained the existing NCDB CCI score, which is categorized as 0, 1, or 2+. Of note, patients with missing comorbidity information are categorized as CCI 0 in the NCDB.²⁸ Second, we calculated a Special Study CCI score as a continuous variable using the comorbidities re-abstracted from the medical record. Neither the NCDB nor the Special Study CCI includes cancer-related conditions in the score. Unlike the NCDB CCI, the Special Study CCI differentiated between patients with no documented comorbidities (Special Study CCI 0) and those with comorbidity information not available (Special Study CCI missing).

Statistical Analyses

For each cancer type, demographic and clinical characteristics were summarized with medians and interquartile ranges (IQRs) or percentages, as appropriate, and compared across cancer types using χ² tests and analysis of variance. All patients were included in the demographics and clinical characteristics table whereas patients for whom cancer registrars indicated that comorbidity information was not available during the Special Study were excluded from subsequent analyses. Sensitivity analyses including patients with missing Special Study comorbidity information were performed where appropriate, substituting known NCDB CCI scores.

For each cancer type, NCDB and Special Study CCI scores (both categorized as 0, 1, or 2+ for direct comparison) were compared using Spearman’s rank correlation coefficient and kappa statistic. For patients with discordant CCI scores, we determined whether the NCDB underestimated or overestimated comorbidity compared with the Special Study. In addition, we performed a sensitivity analysis of kappa statistics by excluding facilities that did not report any comorbidities to the NCDB for all of the randomly selected patients in this analysis whereas the Special Study re-abstracted data did report comorbidities.

Postoperative 5-year survival was assessed for each cancer type using the Kaplan-Meier method. Patients who were lost to follow-up were censored at the time of last contact. To evaluate the prognostic performance of the NCDB CCI, Special Study CCI, and Special Study individual comorbidities, we created three separate Cox proportional hazards models for each cancer type: Model 1 was for categorical NCDB CCI, Model 2 was for continuous Special Study CCI, and Model 3 was for individual comorbidities. Generalized estimating equations were used to account for patient clustering within a treatment facility. Patients with missing model covariates were excluded from this analysis. Within each cancer type, all three models adjusted for the same set of demographic, clinical, and tumor characteristics relevant to that cancer type. Harrell’s C index²⁹ and Akaike Information Criterion (AIC) were used to describe the explained variation and model performance. A lower AIC value suggests better model fit. The threshold of P < .05 was used to determine statistical significance for all two-sided comparisons. Analyses were performed using R v3.3.2 (Vienna, Austria) and STATA v15.1 (STATA, College Station, TX).

RESULTS

Demographic and clinical characteristics of 11,243 patients with breast cancer, 10,880 with colorectal cancer, and 9,640 with lung cancer treated with definitive surgery in 2006-2007 from a total of 1,214 CoC-accredited facilities are listed in Table 1. Median age was youngest for patients with breast cancer (58 years; IQR, 48 to 70 years) and oldest for those with colorectal cancer (68 years; IQR, 58 to 78 years). Comorbidity was highest among patients with lung cancer (13.2% with NCDB CCI 2+; median Special Study CCI, 1; IQR, 0 to 2) and lowest among patients with breast cancer (2.8% with NCDB CCI 2+; median Special Study CCI, 0; IQR, 0 to 1).

Table 1.

Demographic and Clinical Characteristics of Patients With Breast, Colorectal, or Non–Small-Cell Lung Cancer Treated With Definitive Surgical Resection in 2006-2007 (N = 31,763)

CCI scores from existing NCDB data and re-abstracted Special Study data were compared in Figure 1. For breast cancer, the Spearman rank correlation coefficient was 0.50 with a kappa statistic of 0.39 and 77.1% agreement between the NCDB and Special Study CCI scores. Compared with the Special Study, the NCDB underestimated comorbidity for 19.1% of patients with breast cancer and overestimated it for 3.8%. For colorectal cancer, the Spearman rank correlation coefficient was 0.51 with a kappa statistic of 0.36 and 64.1% agreement between the NCDB and Special Study CCI scores. The NCDB underestimated comorbidity for 29.3% of patients with colorectal cancer and overestimated it for 6.6%. For lung cancer, the Spearman rank correlation coefficient was the lowest at 0.40 with a kappa statistic of 0.27 and only 51.9% agreement between the NCDB and Special Study CCI scores. The NCDB underestimated comorbidity for 36.2% of patients with lung cancer and overestimated it for 11.9%.

Fig 1.

Comparison of existing National Cancer Database (NCDB) Charlson-Deyo Comorbidity Index (CCI) with re-abstracted Special Study CCI scores for patients with breast, colorectal, or non–small-cell lung cancer treated with definitive surgical resection in 2006-2007. (A) Breast cancer: Spearman rank correlation, 0.50; kappa statistic, 0.39; agreement between NCDB and Special Study CCI scores, 77.1%; NCDB underestimated comorbidity, 19.1%; NCDB overestimated comorbidity, 3.8%. (B) Colorectal cancer: Spearman rank correlation, 0.51; kappa statistic, 0.36; agreement between NCDB and Special Study CCI scores, 64.1%; NCDB underestimated comorbidity, 29.3%; NCDB overestimated comorbidity, 6.6%. (C) Non–small-cell lung cancer: Spearman rank correlation, 0.40; kappa statistic, 0.27; agreement between NCDB and Special Study CCI scores, 51.9%; NCDB underestimated comorbidity, 36.2%; NCDB overestimated comorbidity, 11.9%.

Among the 1,214 facilities included in the Special Study, 65 facilities (5.4%) did not report any comorbidities to the NCDB for all of the randomly selected patients in this analysis whereas the Special Study re-abstracted data included a report of comorbidities. When these 65 facilities were excluded, the kappa statistics only minimally improved: 0.40 for breast, 0.37 for colorectal, and 0.28 for lung cancer.

Postoperative 5-year survival was highest for patients with breast cancer: 85.8% for stage II and 69.6% for stage III. Among patients with colorectal cancer, postoperative 5-year survival was 81.4% for stage I, 72.5% for stage II, and 64.1% for stage III. Postoperative 5-year survival was lowest for patients with lung cancer: 62.9% for stage I, 44.4% for stage II, and 35.3% for stage III. During the 5-year follow-up period, there was loss to follow-up of 3.5%, 4.2%, and 8.5% of patients with breast, colorectal, or lung cancer, respectively.

The NCDB CCI (Model 1) and Special Study CCI (Model 2) were statistically significant predictors of postoperative 5-year survival for breast, colorectal, and lung cancer (Table 2). Models 1 and 2 performed similarly (breast cancer C index was 0.738 with NCDB CCI and 0.745 with Special Study CCI; colorectal cancer C index was 0.708 with NCDB CCI and 0.713 with Special Study CCI; lung cancer C index was 0.655 with NCDB CCI and 0.659 with Special Study CCI). For all three cancer types, the AIC for Models 1 and 2 were similar. In sensitivity analyses that included patients with missing Special Study CCI patients, C indices for Models 1 and 2 for each cancer type remained unchanged.

Table 2.

Multivariable Cox Proportional Hazards Models for Postoperative 5-Year OS for Patients With Breast, Colorectal, or Non–Small-Cell Lung Cancer Treated With Definitive Surgical Resection in 2006-07 Using Three Models to Measure Comorbidity

In Model 3 with individual comorbidities, chronic pulmonary disease, congestive heart failure, dementia, and moderate or severe renal disease were statistically significant predictors of postoperative 5-year survival for all three cancer types (Table 2). Among patients with breast cancer, coronary artery disease, diabetes with and without end organ damage, hemiplegia/paraplegia, and mild liver disease were also prognostic comorbidities. Cerebrovascular disease, moderate or severe liver disease, other neurologic conditions, and substance abuse were prognostic for colorectal cancer. Coronary artery disease, diabetes without end organ damage, moderate or severe liver disease, peripheral vascular disease, and psychiatric disorder were prognostic for lung cancer (Appendix Table A1, online only). For all three cancer types, the C index and AIC for Model 3 were similar to those for Models 1 and 2.

DISCUSSION

In a large nationally representative cohort study of more than 30,000 patients with surgically resected breast, colorectal, or lung cancer, comorbidity in the NCDB was underestimated in 19.1% (breast cancer) to 36.2% (lung cancer) of patients with only moderate agreement with re-abstracted Special Study comorbidity information. Agreement between the NCDB and Special Study CCI scores was highest for patients with breast cancer who had the lowest burden of comorbidity, whereas agreement was lowest for patients with lung cancer who had the highest burden of comorbidity. However, despite underestimation of comorbidity with the NCDB CCI, application of the re-abstracted Special Study CCI and individual comorbidities produced similar prognostic models for postoperative 5-year OS. This suggests that the current approach of the NCDB for capturing comorbidities is sufficient for risk adjustment for survival outcomes among surgical patients receiving the same treatment. However, it remains unknown whether the systematic underestimation of comorbidity in the NCDB has a significant impact on the comparative effectiveness studies needing to balance heterogeneous groups of patients receiving different treatments.

Underestimation of comorbidity in the NCDB compared with the Special Study was likely the result of several factors. Notably, the Special Study CCI differentiated between patients with no documented comorbidity (Special Study CCI 0) and patients for whom comorbidity information was not available. In contrast, the NCDB does not differentiate between no documented comorbidities (NCDB CCI 0) and missing or unavailable comorbidity information (also coded as NCDB CCI 0). The lack of a missing value category in the NCDB CCI variable limits the ability of researchers to identify and analyze these patients differently from patients who truly have no comorbidities that make up the CCI.

Surprisingly, we found that approximately 5.4% of CoC-accredited facilities did not report comorbidity information for any patients randomly selected for our study, whereas the re-abstracted Special Study did report comorbidities. This finding suggests that these facilities systematically underreport comorbidities to the NCDB. Because facilities are de-identified in the NCDB, additional characterization of these facilities was not possible. This apparent systematic under-reporting of comorbidity to the NCDB by a small subset of facilities is a limitation of the NCDB CCI that researchers need to understand. However, because the kappa statistics only minimally improved after exclusion of these facilities, underestimation of comorbidity in the NCDB still remained from other factors.

In addition, the Special Study chart-based CCI score was re-abstracted from provider notes in the original medical record, unlike the NCDB code-based CCI, which was mapped from ICD-9-CM codes from discharge abstracts or billing face sheets.²² Previous research has shown that chart-based comorbidity measures identify comorbidities in a larger number of patients than code-based measures do.^30,31 However, increased ascertainment of comorbidities with chart-based measures does not necessarily translate to improved prognostic ability. For example, in a study of more than 6,000 newly diagnosed patients with cancer, the chart-based ACE-27 improved comorbidity capture compared with the code-based CCI, but there was no statistically significant difference in model discrimination when predicting 2-year OS.³⁰ Likewise, our study found similar model performance as measured by the C index and AIC when using the Special Study chart-based CCI.

Unlike the NCDB CCI variable, which is categorized as 0, 1, or 2+, we modeled the Special Study CCI as a continuous variable. Treatment of comorbidity as a continuous variable allowed for better discrimination between patients with moderate and severe levels of comorbidity, which was most relevant for the lung cancer cohort in our study in which a larger proportion of patients had higher CCI scores. The NCDB recently recognized the value of adding a fourth category to the CCI variable, which is now reported as 0, 1, 2, or 3+ starting with the 2015 NCDB Public Use File released in 2017.²⁸ The addition of the fourth category will improve identification of patients with the highest level of comorbidities in future research.

When re-abstracted Special Study individual comorbidities were examined in Model 3, the comorbidities that were prognostic for each cancer type varied. Four comorbidities were prognostic for breast, colorectal, and lung cancer, whereas other comorbidities were only prognostic for one cancer type. Although the NCDB publicly releases only the composite CCI data and does not release data on individual comorbidities, inclusion of data on specific comorbidities of interest may be of value for researchers, depending on the cancer type and outcomes being studied.

Consistent with previous research on the differential prognostic impact of comorbidity for different types of cancer,¹² we found that severe comorbidity was a stronger predictor of lower postoperative 5-year OS for cancer types with higher OS. For example, patients with breast cancer with an NCDB CCI score of 2+ had a hazard ratio of 2.10 (95% CI, 1.80 to 2.47; reference CCI 0) for postoperative 5-year OS, whereas patients with lung cancer with an NCDB CCI score of 2+ had a hazard ratio of only 1.43 (95% CI, 1.31 to 1.56). Therefore, when studying cancer types with more favorable OS, it is especially important to include accurate measures of comorbidity.

Our study has several limitations. The Special Study included only those patients with locoregional stage breast, colorectal, or lung cancer who were treated with definitive surgery. The accuracy of comorbidity measurement in the NCDB for patients with advanced disease or other cancer types or who were poor candidates for surgery may differ from our results. In addition, we compared the predictive ability of the three comorbidity measures for only one outcome: postoperative 5-year OS. We chose this outcome because 5-year OS is the gold standard by which cancer treatments are evaluated. However, if different outcomes are modeled, comparison of these three comorbidity measures may identify clinically meaningful differences because the current NCDB CCI variable underestimates comorbidity, which may lead to incomplete risk adjustment and biased results in future research.

For example, in comparative effectiveness studies that assess treatment options in populations with differing levels of comorbidity (eg, surgery v stereotactic body radiation for early-stage lung cancer,^32,33 breast-conserving surgery v mastectomy,³⁴ multimodality treatment for bladder cancer^35,36), accurate and complete comorbidity measurement is critical to yield valid results. Because the Special Study re-abstracted comorbidities only for surgical patients, we are unable to evaluate how a more complete measure of comorbidity in the NCDB would perform in such comparative effectiveness analyses.

In conclusion, to our knowledge, this is the first direct comparison of comorbidity measurement in the NCDB using unique, re-abstracted comorbidity information on the same cohort of patients with cancer. Because the chart-based CCI abstracted during the Special Study is too resource intensive to implement throughout the NCDB, we have identified a practical strategy to improve how the NCDB reports comorbidity data that can in turn improve how researchers analyze comorbidity in future studies, particularly comparative effectiveness research, in which incomplete risk adjustment may lead to biased results. We believe the addition of a missing value category to the NCDB CCI variable will help distinguish between patients with true CCI scores of 0 and those with missing information. This relatively small change, which will not require increased cancer registrar resources, will allow researchers to decide for each study how to handle missing comorbidity data, especially because a small subset of CoC-accredited facilities seem to systematically under-report this information. Improved accuracy of comorbidity measurement in the NCDB may have an impact on numerous future studies to better understand different aspects of cancer care and patient outcomes.

Appendix

Table A1.

Prognostic Comorbidities by Cancer Type

AUTHOR CONTRIBUTIONS

Conception and design: Melisa L. Wong, Timothy L. McMurry, Jessica R. Schumacher, George J. Stukenborg, George J. Chang, Daniel P. McKellar, Benjamin D. Kozower

Financial support: Benjamin D. Kozower

Administrative support: Amanda B. Francescatti

Provision of study materials or patients: Amanda B. Francescatti, Caprice C. Greenberg, Benjamin D. Kozower

Collection and assembly of data: Melisa L. Wong, Timothy L. McMurry, Jessica R. Schumacher, Amanda B. Francescatti, Caprice C. Greenberg, George J. Chang, Benjamin D. Kozower

Data analysis and interpretation: Melisa L. Wong, Timothy L. McMurry, Jessica R. Schumacher, Chung-Yuan Hu, Daniel P. McKellar, Louise C. Walter, Benjamin D. Kozower

Manuscript writing: All authors

Final approval of manuscript: All authors

Accountable for all aspects of the work: All authors

AUTHORS' DISCLOSURES OF POTENTIAL CONFLICTS OF INTEREST

Comorbidity Assessment in the National Cancer Database for Patients With Surgically Resected Breast, Colorectal, or Lung Cancer (AFT-01, -02, -03)

The following represents disclosure information provided by authors of this manuscript. All relationships are considered compensated. Relationships are self-held unless noted. I = Immediate Family Member, Inst = My Institution. Relationships may not relate to the subject matter of this manuscript. For more information about ASCO's conflict of interest policy, please refer to www.asco.org/rwc or ascopubs.org/jop/site/ifc/journal-policies.html.

Melisa L. Wong

Employment: Genentech (I)

Stock and Other Ownership Interests: Genentech (I)

Timothy L. McMurry

Consulting or Advisory Role: Quidel, Diffusion Pharmaceuticals

Jessica R. Schumacher

No relationship to disclose

Chung-Yuan Hu

No relationship to disclose

George J. Stukenborg

No relationship to disclose

Amanda B. Francescatti

No relationship to disclose

Caprice C. Greenberg

Consulting or Advisory Role: Johnson & Johnson

Research Funding: Covidien (Inst)

George J. Chang

Consulting or Advisory Role: Johnson & Johnson, MORE Health

Daniel P. McKellar

No relationship to disclose

Louise C. Walter

No relationship to disclose

Benjamin D. Kozower

No relationship to disclose