Abstract
Background
Individualized postoperative survival calculators for patients with cancer can be an aid for predicting prognosis and clinical decision making, such as the use of adjuvant chemotherapy. The aim of this study was to compare existing survival calculators for colon cancer and determine their performance using an independent cohort of patients.
Methods
A retrospective analysis of a multi-site institutional experience was performed on patients diagnosed with stage II–III colon cancer between January 2012 and March 2013. Patient survival rates were estimated using Roswell Park Comprehensive Cancer Center (RPCCC), Memorial Sloan Kettering Cancer Center (MSKCC), and MD Anderson Cancer Center (MDACC) calculators. These calculators vary in the number and breadth of variables that are included. The agreement between selected models was obtained through a scatter plot matrix and related intra-class correlation coefficient (ICC). Calculators’ performances were compared using time-dependent receiver operating characteristic (ROC) curves and corresponding area under the curve (AUC) values.
Results
After the application of inclusion and exclusion criteria, a total of 97 patients were included in the analysis. Survival data were available for all patients. Median follow-up was 57.6 months, and the overall 5-year survival rate was 0.74 (95% CI: 0.64–0.82). Overall, the different calculators tended to predict survival similarly (ICC =0.017). However, there was variation among calculator performance with the RPCCC calculator showing the highest performance (AUC =0.913), followed by the MSKCC calculator (AUC =0.803), and the MDACC calculator (AUC =0.644).
Conclusions
Prognostic models incorporating a more comprehensive amount of patient and tumor specific variables may provide a more accurate estimate of individual patient survival rates. These tools can be an actual aid in the clinical practice, allowing physicians to personalize treatment and follow-up for patients with colon cancer.
Keywords: Colon cancer, overall survival, calculators
Introduction
Colorectal cancer (CRC) is the second leading cause of cancer-related death in the United States and the third most common cancer in both men and women. In 2014, over fifty thousand people died from CRC in the United States (1). An additional 140,250 new cases are expected in 2018 (2). The role and efficacy of adjuvant chemotherapy for CRC have been well established. Modern chemotherapeutic regimens have been demonstrated to improve overall survival (OS), especially for stage II and III patients (3,4).
Accurate prognosis for patients diagnosed with cancer is critical. Stage of the disease has been demonstrated to be strongly associated with OS (5,6). However, predicting prognosis for individual patients remains imprecise. In recent years, several studies have included more clinicopathological features in order to enhance the accuracy of the predicted prognosis. Different prediction models have been developed to assess survival rate for individual patients, who have been treated for stage II and III colon cancer: (I) Roswell Park Comprehensive Cancer Center (RPCCC) calculator (7); (II) Memorial Sloan-Kettering Cancer Center calculator (MSKCC, available at https://www.mskcc.org/nomograms/colorectal) (8); (III) MD Anderson Cancer Center calculator (MDACC, available at http://www3.mdanderson.org/app/medcalc/index.cfm?pagename=coloncancer) (9); (IV) Mayo Clinic ACCENT calculator (available at https://www.mayoclinic.org/medical-professionals/cancerpredictiontools/colon-cancer) (10); and (V) Adjuvant! Online (available at http://www.adjvantonline.com/) (11). All of these offers post-operative survival calculation, with the exception of the MDACC calculator, which generates an estimate of 5-year conditional survival (CS).
While these models have been validated and assistance with clinical decision making (12,13), concordance in predicting survival rates can often be variable among each of these calculators. The purpose of this study was to compare these selected calculators and determine which, if any, was the most accurate in predicting the 5-year OS rate for individual patients with stage II/III colon cancer. We hypothesized that the more extensive inclusion of patient, tumor, and outcome-specific variables would optimize prediction of OS.
Methods
Patients
This was a retrospective review of a multi-site institutional experience for all patients diagnosed with clinical stage II or III colon cancer. Mayo Clinic Institutional Review Board (IRB) approval was obtained for this study; informed consent was waived by our IRB as this retrospective study was deemed minimal risk to patients. Patient selection was limited to the diagnosis made from January 1, 2012 to March 31, 2013. This criterion was influenced by the RPCCC OS calculator. The RPCCC calculator includes patients from National Cancer Data Base (NCDB) up to 2011. Due to the participation of Mayo Clinic in the NCDB, a cohort of patients outside the years used in the calculator (specifically 2004–2011) was required in order to perform appropriate external validation of this specific calculator. This criterion was not applicable to the other calculators, as these were developed using patients prior to 2011. Specifically, the Mayo Clinic calculator included patients accrued to 12 phase III clinical trials from 1989 to 2002 (10). The MSKCC and MDACC are based on the Surveillance, Epidemiology and End Results (SEER) Program and included patients from 1996 to 2005, and from 1988 to 2000, respectively (8,9). The Adjuvant! Online tool is also based on SEER prior to 2011 (11), although it is currently being updated.
As the Adjuvant! Online calculator is currently being revised, not every calculator could be included in this study. Adjuvant! Online was neither available nor accessible because of website maintenance. The Mayo Clinic ACCENT calculator could not be used as well, because of insufficient or incompatible data within our cohort. Specifically, 65 patients did not receive adjuvant chemotherapy, which is a mandatory variable for estimating survival rate using Mayo Clinic ACCENT calculator. Therefore, the Mayo Clinic and Adjuvant! Online calculators were excluded from the analysis. Figure 1 shows the selection criteria for the calculators.
Figure 1.
Selection criteria for colon cancer survival rate calculators. RPCCC, Roswell Park Comprehensive Cancer Center; MSKCC, Memorial Sloan Kettering Cancer Center; MDACC, MD Anderson Cancer Center.
Surgical resection with confirmation of pathological stage II or III colon cancer was included among the primary inclusion criteria. Patients were identified using the International Statistical Classification of Disease (ICD CM-10) codes C18.0 to C18.9. Clinical or pathological stage I or IV patients were excluded. Patients with histology different from adenocarcinoma, carcinoma in situ, more than 1 recorded malignancy were excluded as well. Figure 2 summarizes the inclusion and exclusion criteria adopted for this study.
Figure 2.
Flow chart showing the inclusion and exclusion criteria used to select patient for the comparison of the calculators.
The following patient-specific characteristics were collected: age, gender, race, and Charlson-Deyo comorbidity score as a measure of comorbidity status. Treatment and outcome-related variables included the surgical procedure (partial colectomy, sub-colectomy/hemicolectomy, total colectomy, total proctocolectomy), margins status, number of nodes examined and positive nodes at pathological examination, post-operative hospital length of stay (days), length of follow-up (months), unplanned readmission, and administration of adjuvant chemotherapy and/or adjuvant radiotherapy. The tumor-related characteristics included preoperative carcinoembryonic antigen (CEA) levels, tumor grade, size, pathological T and N stage (7th edition AJCC staging). Table 1 summarizes the calculator-specific collected variables as they pertain to the selected OS calculators.
Table 1. Variables needed to calculate survival rates for each calculator.
| Variables | MSKCC (institutional) | MDACC (institutional) | RPCCC (NCDB) |
|---|---|---|---|
| Any stage | Any stage | Stage II or III | |
| Post-operative | Not post-operative | Post-operative | |
| 5-year OS | 5-year CS | 5-year OS | |
| Age | ● | ● | ● |
| Gender | ● | ● | ● |
| Race | ● | ● | |
| Charlson-Deyoa | ● | ||
| Grade | ● | ● | ● |
| Size | ● | ||
| CEA level | ● | ||
| Surgery (procedure) | ● | ||
| Margins | ● | ||
| Number nodes examined | ● | ● | |
| Number nodes positive | ● | ● | |
| Pathological T stage | ● | ||
| Pathological N stage | ● | ||
| Pathological stage | ● | ||
| Clinical stage | ● | ||
| Post-op length of stay | ● | ||
| Unplanned readmission | ● | ||
| Adjuvant chemotherapy | ● | ||
| Adjuvant radiation | ● |
a, Charlson-Deyo comorbidity score is an estimate of comorbid conditions based on ICD-9 diagnosis codes. A score of 0 indicates no comorbidities. Point values are assigned to comorbid conditions based on severity. The NCDB truncates possible scores to 0, 1 and 2 due to the small proportion of cases exceeding a score of 2. MSKCC, Memorial Sloan Kettering Cancer Center; MDACC, MD Anderson Cancer Center; RPCCC, Roswell Park Comprehensive Cancer Center; NCDB, National Cancer Data Base; CEA, carcinoembryonic antigen; OS, overall survival; CS, conditional survival; ICD, International Classification of Diseases.
Statistical analysis
Patient characteristics were summarized using the mean, median, and standard deviation for continuous variables and using frequencies and relative frequencies for categorical variables. OS was summarized using standard Kaplan-Meier methods, where estimates of median and 5-year OS were obtained with 95% confidence intervals. The 5-year OS survival scores were summarized for each calculator using the mean, median, and standard deviation. The association between scores from different calculators was examined using a scatter plot matrix, while their agreement was assessed using the intra-class correlation coefficient (ICC). The performance of the calculators was assessed using time-dependent receiver operating characteristic (ROC) curves and corresponding area under the curve (AUC) values. All analyses were conducted in SAS v9.4 (Cary, NC, USA) at a significance level of 0.05.
Results
A total of 504 patients were extracted from our multi-site institutional database. Following the application of the inclusion and exclusion criteria, a total of 97 patients were suitable for the analysis. The overall patient and tumor characteristics used to calculate survival rates are shown in Table 2. Figure 3 shows the Kaplan-Meier curve for OS. The survival data were available for all 97 patients, of which there were 23 events. The median follow-up was 57.6 months. The 5-year OS rate was 0.74 (95% CI: 0.64–0.82).
Table 2. Patients characteristics used to compare survival calculators.
| Variables | Overall |
|---|---|
| Overall | |
| N (%) | 97 [100] |
| Age | |
| Mean (SD) (years) | 70.7 (14.9) |
| Gender (%) | |
| Male | 53 (54.6) |
| Female | 44 (45.4) |
| Race (%) | |
| White | 94 (96.9) |
| Black | 2 (2.1) |
| Unknown | 1 (1.0) |
| Charlson indexa (%) | |
| 0 | 6 (6.2) |
| 1 | 17 (17.5) |
| 2 | 74 (76.3) |
| Grade (%) | |
| 1 | 5 (5.2) |
| 2 | 48 (49.5) |
| 3 | 42 (43.3) |
| Unknown | 2 (2.1) |
| Tumor size (cm) (%) | |
| 1–4 | 24 (24.7) |
| 4–9 | 61 (62.9) |
| >9 | 12 (12.4) |
| CEA (%) | |
| Normal | 40 (41.2) |
| Elevated | 27 (27.8) |
| Unknown | 30 (30.9) |
| Surgery (%) | |
| Partial colectomy | 27 (27.8) |
| Sub-colectomy/hemicolectomy | 59 (60.8) |
| Total colectomy | 9 (9.3) |
| Total proctocolectomy | 2 (2.1) |
| Margins (%) | |
| Negative | 96 (99.0) |
| Positive | 1 (1.0) |
| Nodes examined | |
| Mean (SD) | 31.2 (16.7) |
| Nodes positive | |
| Mean (SD) | 1.5 (3.2) |
| Stage (%) | |
| 2 | 52 (53.6) |
| 3 | 45 (46.4) |
| Length of stay (days) | |
| Mean (SD) | 5.6 (4.9) |
| Unplanned readmission (%) | |
| No | 93 (95.9) |
| Yes | 4 (4.1) |
| Adjuvant chemotherapy (%) | |
| No | 65 (67.0) |
| Yes | 32 (33.0) |
| Adjuvant radiation (%) | |
| No | 94 (96.9) |
| Yes | 3 (3.1) |
a, Charlson-Deyo comorbidity score is an estimate of comorbid conditions based on ICD-9 diagnosis codes. A score of 0 indicates no comorbidities. Point values are assigned to comorbid conditions based on severity. The NCDB truncates possible scores to 0, 1 and 2 due to the small proportion of cases exceeding a score of 2. SD, standard deviation; CEA, carcinoembryonic antigen; ICD, International Classification of Diseases; NCDB, National Cancer Data Base.
Figure 3.
Kaplan-Meier curve showing 5-year OS. OS, overall survival.
The scatter plot matrix used to examine the agreement between all calculators is shown in Figure 4. Results indicate that the different calculators tended to rank patients similarly. Thus, if a patient obtained the highest survival score using the RPCCC calculator, then that patient was likely to obtain a high score for the other two calculators. However, there was variation on what the predicted survival score should be, indicated by the overall ICC of 0.017. When comparing how the MDACC performed with the other calculators, a negative ICC was obtained (−0.294 and −0.541), which is due to the survival scores of MD Anderson being generally more optimistic. Conversely, there was a reasonable level of agreement between the RPCCC and MSKCC calculators (ICC =0.723).
Figure 4.
Agreement plot between Prediction Tools. RPCCC and MSKCC calculators show a reasonable level of agreement. ICC, intra-class correlation coefficient; RPCCC, Roswell Park Comprehensive Cancer Center; MSKCC, Memorial Sloan Kettering Cancer Center.
When evaluating individual calculator performance with the observed OS, the RPCCC calculator had the highest AUC (0.913), followed by the MSKCC calculator (AUC =0.803), and the MDACC calculator (AUC =0.644). Therefore, given the AUC >0.8, RPCCC and MSKCC calculators were similarly able to differentiate between good and poor outcome patients. Figure 5 shows the performance evaluation and the AUCs for each calculator.
Figure 5.
Evaluating Prediction Tool Performance. The AUC for both RPCCC and MSKCC calculators is greater than 0.8; indicating that these calculators can differentiate between good and poor outcome patients. AUC, area under the curve; RPCCC, Roswell Park Comprehensive Cancer Center; MSKCC, Memorial Sloan Kettering Cancer Center; MDACC, MD Anderson Cancer Center.
Discussion
One of the most significant values of survival prediction models lies in the ability to estimate survival outcomes for individual patients, and then to quantify the potential benefit from adjuvant treatment. In this study, a comparison between existing calculators was performed, hypothesizing that a more comprehensive inclusion of patient, treatment, and tumor specific variables would lead to a more accurate prediction of OS.
The present analysis involved patients from three sites in different areas of the United States, which increased the heterogeneity of the studied population. Moreover, the multi-site institutional database was queried in order to perform an external validation with complete data for all RPCCC, MSKCC, and MDACC calculators. The median length of follow up (57.6 months) was sufficient to compare 5-year survival rates for each calculator.
The RPCCC calculator achieved the best performance (AUC =0.913) in estimating 5-year survival rates. Thus, the inclusion of more variables in the equation likely allows a higher level of input and analysis for each patient, resulting in a more accurate OS prediction. Variables which contribute to OS, including comorbid status (Charlson-Deyo index) and post-operative outcomes, such as length of stay and potential readmissions as surrogates for post-operative complications, increase the completeness of the prediction. These variables have been demonstrated to have a strong correlation with OS. As additional benefit, they can also be used as a measure of quality for surgery outcomes (7,14,15). At present, the RPCCC calculator is the only one incorporating these variables in its model (8-11). The MSKCC and MDACC calculators showed a lower performance (AUC =0.803 and 0.644, respectively), although these still show a good to very good ability for OS prediction. The lower AUC scores may be explained by the relatively limited number of variables included in their models. Furthermore, the MDACC calculator does not generate post-operative outcomes but rather predicts conditional 5-year survival. Thus, its performance is extrapolated to this patient population, which had surgery as part of their treatment course.
In the recent years, the development of calculators and tools estimating outcomes has increased and their use is becoming part of the assessment and decision-making process (16-18). Even though equations and nomograms, which these tools are based on, can be quite complex, they result practicable and easily accessible aids for patients and physicians (12,13). The RPCCC calculator equation is quite complex but was developed into a free, user friendly, mobile application (7) that will allow for more feasible use of the calculator. In addition, all of the other calculators are available online through web-based interactive pages. However, in spite of these promising advantages, the use of these tools is not intended as a replacement of the clinical judgement. Instead, these tools should be used an instrument that can refine the prognosis appraisal and treatment decision making, and potentially act as a bridge between individual patient and large-scale statistics.
Limitations of this comparison include those involved in retrospective studies, which introduces bias in patient selection. Moreover, patients were accrued from an institutional database, and thus this may affect the reproducibility of results on the whole population. Indeed, the OS of the cohort used in this study was higher than that reported in the cohort used to generate the RPCCC calculator. Thus, this may have influenced the accuracy of the calculators tested as a more population with more favorable observed outcomes may be better suited for certain calculators, whereas a different population with poorer observed outcomes may potentially be better suited for different calculators. The available data were also limited to a modest number of patients (n=97), and, as a consequence, this may influence the strength of the comparison. In this analysis, it was not possible to compare all existing calculators that estimate survival rates for colon cancer. A significant number of patients were not eligible to be used in the Mayo Clinic ACCENT calculator because of incompatible data. In addition, the Adjuvant! Online tool was unavailable online due to current updating and maintenance (19). The inclusion of these two calculators in the comparison would have granted a more comprehensive overview on the impact of all calculators in the clinical practice. Lastly, calculators estimate 5-year OS or CS instead of disease-free (DFS) or disease-specific (DSS) survival, which act as a valid surrogate for OS (20) and provide more specific information about mortality rates directly related to cancer.
Nonetheless, this study reports a valuable comparison between three important existing survival rate calculators for colon cancer. Based on the hypothesis that a more comprehensive list of variables, spanning patient, tumor, and treatment related characteristics would result in the most optimal post-operative OS survival calculator, the RPCCC calculator was shown to be the most accurate tool. A study utilizing a larger population, and perhaps in a multi-institutional cohort, would enhance the comparison between the different calculators, and is the focus of further studies.
Acknowledgements
Funding: This work was supported by Roswell Park Cancer Institute and National Cancer Institute (NCI) grant P30CA016056, and also by the Mayo Clinic Robert D. and Patricia E. Kern Center for the Science of Health Care Delivery. We acknowledge and thank the American College of Surgeons Committee on Cancer for providing access to the Participant User File from the National Cancer Data Base.
Disclaimers: The American College of Surgeons Committee on Cancer provided the Participant User File from the National Cancer Data Base, but has not reviewed or validated the results or conclusions of our study.
Ethical Statement: Mayo Clinic Institutional Review Board (IRB) approval was obtained for this study (No. 17-008952); informed consent was waived by our IRB as this retrospective study was deemed minimal risk to patients.
Footnotes
Conflicts of Interest: The authors have no conflicts of interest to declare.
References
- 1.Colorectal Cancer Statistics. Centers for Disease Control and Prevention. Available online: https://www.cdc.gov/cancer/colorectal/statistics/
- 2.Siegel RL, Miller KD, Jemal A. Cancer statistics, 2018. CA Cancer J Clin 2018;68:7-30. 10.3322/caac.21442 [DOI] [PubMed] [Google Scholar]
- 3.Babaei M, Balavarca Y, Jansen L, et al. Administration of adjuvant chemotherapy for stage II-III colon cancer patients: An European population-based study. Int J Cancer 2018;142:1480-9. 10.1002/ijc.31168 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Gill S, Loprinzi CL, Sargent DJ, et al. Pooled analysis of fluorouracil-based adjuvant therapy for stage II and III colon cancer: who benefits and by how much? J Clin Oncol 2004;22:1797-806. 10.1200/JCO.2004.09.059 [DOI] [PubMed] [Google Scholar]
- 5.Lan YT, Yang SH, Chang SC, et al. Analysis of the seventh edition of American Joint Committee on colon cancer staging. Int J Colorectal Dis 2012;27:657-63. [DOI] [PubMed] [Google Scholar]
- 6.Edge SB, Compton CC. The American Joint Committee on Cancer: the 7th edition of the AJCC cancer staging manual and the future of TNM. Ann Surg Oncol 2010;17:1471-4. [DOI] [PubMed] [Google Scholar]
- 7.Gabriel E, Attwood K, Thirunavukarasu P, et al. Predicting Individualized Postoperative Survival for Stage II/III Colon Cancer Using a Mobile Application Derived from the National Cancer Data Base. J Am Coll Surg 2016;222:232-44. 10.1016/j.jamcollsurg.2015.12.019 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Weiser MR, Gonen M, Chou JF, et al. Predicting survival after curative colectomy for cancer: individualizing colon cancer staging. J Clin Oncol 2011;29:4796-802. 10.1200/JCO.2011.36.5080 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Chang GJ, Hu CY, Eng C, et al. Practical application of a calculator for conditional survival in colon cancer. J Clin Oncol 2009;27:5938-43. 10.1200/JCO.2009.23.1860 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Renfro LA, Grothey A, Xue Y, et al. ACCENT-based web calculators to predict recurrence and overall survival in stage III colon cancer. J Natl Cancer Inst 2014;106. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Ravdin PM, Siminoff LA, Davis GJ, et al. Computer program to assist in making decisions about adjuvant therapy for women with early breast cancer. J Clin Oncol 2001;19:980-91. 10.1200/JCO.2001.19.4.980 [DOI] [PubMed] [Google Scholar]
- 12.Papamichael D, Renfro LA, Matthaiou C, et al. Validity of Adjuvant! Online in older patients with stage III colon cancer based on 2967 patients from the ACCENT database. J Geriatr Oncol 2016;7:422-9. 10.1016/j.jgo.2016.07.002 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Gill S, Loprinzi C, Kennecke H, et al. Prognostic web-based models for stage II and III colon cancer: A population and clinical trials-based validation of numeracy and adjuvant! online. Cancer 2011;117:4155-65. 10.1002/cncr.26003 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Chen VW, Hsieh MC, Charlton ME, et al. Analysis of stage and clinical/prognostic factors for colon and rectal cancer from SEER registries: AJCC and collaborative stage data collection system. Cancer 2014;120 Suppl 23:3793-806. 10.1002/cncr.29056 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Hari DM, Leung AM, Lee JH, et al. AJCC Cancer Staging Manual 7th edition criteria for colon cancer: do the complex modifications improve prognostic assessment? J Am Coll Surg 2013;217:181-90. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Bardia A, Loprinzi C, Grothey A, et al. Adjuvant chemotherapy for resected stage II and III colon cancer: comparison of two widely used prognostic calculators. Semin Oncol 2010;37:39-46. 10.1053/j.seminoncol.2009.12.005 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Gabriel E, Attwood K, Shah R, et al. Novel Calculator to Estimate Overall Survival Benefit from Neoadjuvant Chemoradiation in Patients with Esophageal Adenocarcinoma. J Am Coll Surg 2017;224:884-94.e1. 10.1016/j.jamcollsurg.2017.01.043 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Sjoquist KM, Renfro LA, Simes RJ, et al. Personalizing Survival Predictions in Advanced Colorectal Cancer: The ARCAD Nomogram Project. J Natl Cancer Inst 2018;110:638-48. 10.1093/jnci/djx253 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Adjuvant! Online. Adjuvant! Inc. Available online: http://www.adjuvantonline.com/indexjsp
- 20.Abrams J. Disease-free survival versus overall survival as a primary end point for adjuvant colon cancer studies: a commentary. J Clin Oncol 2005;23:8564-5. 10.1200/JCO.2005.03.6186 [DOI] [PubMed] [Google Scholar]