Abstract
Background
The Memorial Sloan Kettering Cancer Center (MSK) colon cancer recurrence nomogram is a risk calculator that provides patients and clinicians with individualized prediction of recurrence following curative resection of colon cancer. Although validated on multiple separate cohorts, the nomogram requires periodic updating as patient care changes over time. The aim of this study was to evaluate the nomogram’s accuracy in a contemporary cohort and modify the tool to reflect improvements in outcome related to advances in colon cancer therapy.
Methods
A contemporary patient cohort was compiled, including consecutive colon cancer patients undergoing curative resection for stage I–III colon adenocarcinoma at MSK from 2007 to 2014. The nomogram’s predictive accuracy was assessed by concordance index and calibration plots of predicted vs actual freedom from recurrence at 5 years after surgery.
Results
Data from a total of 999 eligible patients with complete records were used for validation. Median follow-up among survivors was 37 months. The concordance index was 0.756 (95% confidence interval = 0.707 to 0.805), indicating continued discriminating power, but the calibration plot revealed that the nomogram overestimated recurrence risk. Recalibration of the nomogram by estimating a new baseline freedom-from-recurrence function restored the nomogram’s accuracy.
Conclusion
The updated nomogram retains the original nomogram’s variables but includes a lower baseline estimation of recurrence risk, reflecting improvements in outcomes for all stages of colon cancer, likely resulting from advances in imaging and integration of multiple treatment modalities.
Predictive models such as nomograms have become popular to assess risk of cancer outcomes because they provide a more refined estimate than the TNM staging system of the American Joint Commission on Cancer (AJCC) and the International Union Against Cancer (1). Although TNM staging is easy to implement because it relies only on tumor depth, number of positive lymph nodes, and presence of distant metastasis, its predictive accuracy is limited. Outcomes of patients with cancer of the same TNM stage vary considerably because TNM staging does not account for all prognostic variables (2).
Nomograms provide better predictive accuracy of outcomes, enabling individualized treatment and planning of postoperative surveillance and refining the design of clinical trials (3). One such model is the Memorial Sloan Kettering Cancer Center (MSK) colon cancer recurrence nomogram (published in 2008), which allows clinicians to more precisely predict an individual’s risk of colon cancer recurrence following curative resection (4). Based on a single-institutional cohort treated from 1990 to 2000, the nomogram estimates 5-year and 10-year freedom from recurrence (FFR) by integrating information on nine clinicopathologic features including age, tumor size, preoperative carcinoembryonic antigen, use of adjuvant chemotherapy, and several indicators of tumor invasiveness and spread. The nomogram was intended for easy adoption in clinical practice because it was based on commonly measured variables and is available online (https://www.mskcc.org/nomograms/colorectal). The nomogram was externally validated in cohorts from Australia (2000–2005, n = 134), the United Kingdom (1998–2003, n = 138), and China (1996–2008, n = 985), confirming its broad applicability (5–7).
Throughout the past decade, the management of colon cancer has advanced in terms of clinical staging, surgery, and adjuvant chemotherapy. High-quality imaging, including routine use of 64-slice, contrast-enhanced, helical computed tomography and magnetic resonance imaging, has improved staging by preoperative identification of small metastatic deposits (8). Adoption of a standardized, anatomically based surgical technique has increased the thoroughness of tumor resection and lymphadenectomy (9–12). Following prospective randomized trials demonstrating improved outcome, oxaliplatin-based adjuvant chemotherapy became standard for stage III colon cancer in 2004 (13). A better understanding of stage II risk factors has also led to an expanded use of chemotherapy in node-negative patients (14).
Considering the evolution of colon cancer care, the aim of this study was to assess the validity of the MSK colon cancer recurrence nomogram for a contemporary cohort and update the model to ensure continued predictive accuracy.
Methods
Patient Cohort
After approval from the institutional review board, prospectively maintained institutional databases were queried for patients undergoing curative resection for stage I, II, or III colon adenocarcinoma from 2007 through the end of 2014. All lesions were located from the cecum to the rectosigmoid (>12 cm from the anal verge). Exclusion criteria included history of treatment for malignant tumors within the last 5 years, recurrence prior to surgery, residual tumor following surgery, metastatic disease, and preoperative chemotherapy or radiotherapy.
Demographics, clinicopathologic variables, and follow-up were retrieved and manually reviewed via the electronic medical record, similarly to the development of the original nomogram.
Staging, Surgical Procedure, and Surveillance Protocol
Preoperative staging included contrast-enhanced computed tomography (CT) of the chest, abdomen, and pelvis and colonoscopy. Resection method was complete mesocolic excision with central vascular ligation, performed by specialized colorectal surgeons (9). Surgical approach (open, laparoscopic, or robotic) was determined at the surgeon’s discretion. Adjuvant chemotherapy was administered and surveillance performed in accordance with national guidelines (14). Surveillance included physical examination, interval history, and serum carcinoembryonic antigen testing at 3- to 6-month intervals for the first 3 years and then at 6-month intervals thereafter for a total of 5 years. Imaging, most frequently CT of the chest, abdomen, and pelvis with oral and intravenous contrast, was performed at least annually. Colonoscopy was typically performed at 1 year after surgery and then repeated every 3–5 years based on endoscopic findings. Recurrence was determined based on radiographic evidence (with or without biopsy), colonoscopy, and serum carcinoembryonic antigen.
Nomogram Concordance and Calibration
Recurrence-free time was defined from the date of surgery until the date of recurrence or last follow-up, and patients without recurrence were censored at last follow-up. The risk score (ie, linear predictor) based on the existing nomogram was calculated for each patient in the current dataset. Predictive accuracy was assessed by concordance index (15), which represents the probability that given two randomly selected patients, the patient who recurred first had a higher predicted probability of recurrence. Values are interpreted similarly to the area under the receiver operating characteristics curve, with 0.5 equaling random chance and 1.0 representing correct predictions for all patients (16). The nomogram was calibrated by plotting the nomogram-predicted recurrence-free probability vs actual recurrence-free rates based on Kaplan–Meier estimates.
Recalibration
The nomogram was recalibrated by refitting the nomogram model on the contemporary dataset and estimating a new baseline recurrence-free probability function at 5 years while keeping the linear predictor unchanged (17). The new 5-year predicted probability was verified by comparison with observed Kaplan–Meier estimates in a new calibration plot.
Other Statistical Analysis
Characteristics of patients in the 1990–2000 and 2007–2016 cohorts were compared using the Mann-Whitney U and χ2 tests for continuous and categorical variables, respectively. Recurrence-free time was estimated by Kaplan-Meier method and the statistical significance of between-group differences was assessed by log-rank test. All statistical analyses were performed using SAS version 9.4 (SAS Institute Inc, Cary, NC) and R version 3.2.4 (www.R-project.org). All tests were two sided, and P values less than .05 were considered statistically significant.
Results
The database query identified 1095 consecutive patients treated for stage I–III colon adenocarcinoma from 2007 through 2014 who met the inclusion criteria. Clinicopathologic features of patients in the 1990–2000 and 2007–2014 datasets are shown in Table 1. Compared with the 1990–2000 cohort, patients in the 2007–2014 dataset were slightly younger and had tumors with a higher proportion of moderate to poor (vs well) differentiation, lymphovascular invasion, perineural invasion, T3–4 (vs T1–2) disease, and N1–2 (vs N0) disease. The median number of lymph nodes evaluated was higher in the 2007–2014 cohort compared with the 1990–2000 cohort (22, interquartile range [IQR] = 17–30, vs 14, IQR = 9–21, P < .0001). A similar proportion of stage III patients received adjuvant chemotherapy in both cohorts, whereas a greater proportion of stage II patients received chemotherapy in the 2007–2014 cohort (25% vs 14%, P < .0001). Details of adjuvant chemotherapy administered to patients in the 2007–2014 cohort are shown in Supplementary Table 1 (available online). In the 2007–2014 cohort, 62.6% of stage II and 87.3% of stage III patients receiving adjuvant chemotherapy were given regimens including oxaliplatin.
Table 1.
Clinicopathologic features of patients in the 1990–2000 and 2007–2014 datasets*
| Characteristic | 1990–2000 | 2007–2014 | P |
|---|---|---|---|
| (n = 1320) | (n = 1095) | ||
| Age, y | 69 (59–76) | 65 (54–75) | <.0001 |
| Sex | .90 | ||
| Female | 664 (50%) | 554 (51%) | |
| Male | 656 (50%) | 541 (49%) | |
| Location | <.0001 | ||
| Right | 506 (38%) | 434 (40%) | |
| Transverse | 131 (10%) | 132 (12%) | |
| Left | 153 (12%) | 83 (8%) | |
| Sigmoid | 401 (30%) | 290 (26%) | |
| Rectosigmoid | 129 (10%) | 156 (14%) | |
| Differentiation | <.0001 | ||
| Well | 139 (11%) | 29 (3%) | |
| Moderate | 1011 (77%) | 884 (81%) | |
| Poor | 165 (13%) | 179 (16%) | |
| Lymphovascular invasion | 212 (16%) | 493 (46%) | <.0001 |
| Perineural invasion | 79 (6%) | 282 (26%) | <.0001 |
| T-stage | .019 | ||
| T1 | 230 (17%) | 159 (15%) | |
| T2 | 247 (19%) | 190 (17%) | |
| T3 | 743 (56%) | 630 (58%) | |
| T4 | 100 (8%) | 116 (11%) | |
| N-stage | .003 | ||
| N0 | 940 (71%) | 711 (65%) | |
| N1 | 271 (21%) | 265 (24%) | |
| N2 | 109 (8%) | 119 (11%) | |
| AJCC stage, fifth edition | .001 | ||
| I | 421 (32%) | 286 (26%) | |
| II | 520 (39%) | 425 (39%) | |
| III | 379 (29%) | 384 (35%) | |
| Preoperative CEA, ng/mL | 3.1 (1.7 – 6.7) | 3.1 (2 – 5.9) | .33 |
| range = 0.2– 798 | range = 0.4–210.6 | ||
| No. of positive nodes (N1/2) | 2 (1–4) | 2 (1–4) | .576 |
| No. of negative nodes | 13 (8–20) | 21 (16–29) | <.0001 |
| range = 0–146 | range = 2–78 | ||
| No. of lymph nodes evaluated | 14 (9 – 21) | 22 (17 – 30) | <.0001 |
| >12 lymph nodes evaluated | 807 (61%) | 1064 (97%) | <.0001 |
| Stage I | 187 (44%) | 271 (95%) | <.0001 |
| Stage II | 343 (66%) | 417 (98%) | <.0001 |
| Stage III | 277 (73%) | 376 (98%) | <.0001 |
| Postoperative chemotherapy | |||
| Stage I | 8 (2%) | 0 (0%) | .019 |
| Stage II | 72 (14%) | 107 (25%) | <.0001 |
| Stage III | 320 (85%) | 331 (89%) | .14 |
*Continuous data are presented as n (interquartile range), with ranges below when they differ, and categorical data as n (%). AJCC = American Joint Commission on Cancer; CEA = carcinoembryonic antigen.
Kaplan-Meier estimates for FFR stratified by AJCC stage (eighth edition) are shown in Figure 1; 120 experienced recurrence and 19 died without recurrence. Median follow-up among survivors was 37 months (IQR = 19.2–55.0). A stage-by-stage comparison of FFR between patients in the 1990–2000 and 2007–2014 cohorts is shown in Figure 2. The contemporary cohort had statistically significantly higher FFR in stage II (HR = 0.52, 95% CI = 0.35 to 0.77, P = .0011) and a trend toward higher FFR in stage III (HR = 0.77, 95% CI = 0.58 to 1.02, P = .064).
Figure 1.
Postsurgery freedom from recurrence in 1095 patients undergoing complete resection of nonmetastatic colon cancer from 2007 to 2014 according to American Joint Commission on Cancer substage (eighth edition).
Figure 2.
Postsurgery freedom from recurrence in patients in the 1990–2000 and 2007–2014 cohorts according to American Joint Commission on Cancer stage. CI = confidence interval.
A complete set of data required for nomogram validation was available in 999 of the 1095 patients in the 2007–2014 dataset (Supplementary Table 2, available online). FFR in the validation cohort was similar to that in the entire cohort (Supplementary Table 3, available online); 113 had recurrence and 16 died without recurrence.
The concordance index of the nomogram for predicting 5-year FFR in the contemporary dataset was 0.756 (95% CI = 0.707 to 0.805), indicating good discriminating power to distinguish risk of recurrence among patients in the cohort (18,19). Although the concordance index was high, the calibration curve comparing actual with nomogram-predicted 5-year FFR demonstrated that the nomogram overestimated the risk of recurrence at all risk levels in the 2007–2014 cohort (Figure 3A). We adjusted the nomogram by estimating a new baseline FFR function using the current data but keeping the same linear predictors (Figure 4). After recalibration, the nomogram accurately predicted 5-year FFR (Figure 3B).
Figure 3.
Calibration curve for (A) the original nomogram and (B) the updated nomogram for predicting 5-year freedom from recurrence in the 2007–2014 cohort (n = 999).
Figure 4.
Colon cancer nomogram for freedom from recurrence. Locate the patient’s preoperative level of carcinoembryonic antigen (in nanograms per milliliter) on the CEA axis. Draw a straight line up to the Points axis to determine how many points toward recurrence the patient should receive. Repeat this process for each of the remaining axes, drawing a straight line each time to the Points axis. Sum the points received from each prognostic variable and locate this number on the Total points axis. Draw a straight line from the total points down to the 3- or 5-year freedom from recurrence axis to ascertain the patient’s specific risk of remaining free from recurrence for either 3 or 5 years. CEA = carcinoembryonic antigen; L = left colon; R = right colon; RS = rectosigmoid colon; Sig = sigmoid colon; TC = transverse colon.
Discussion
We have updated the MSK nomogram for predicting recurrence of colon cancer after complete resection using data from a contemporary cohort of patients. Because the original nomogram remained prognostic based on concordance index but overestimated the risk of recurrence, we recalibrated by decreasing the baseline recurrence risk function.
The improvement in outcomes we observed relative to the 1990–2000 period likely results from multiple advances. Enhanced imaging capable of identifying small metastatic deposits improves the accuracy of staging for advanced disease. Multidetector helical CT increased the sensitivity of CT for detecting liver metastasis, from 62%–74% in the 1990s using nonhelical CT (20–22) to 80%–90% (23–25). The addition of liver magnetic resonance imaging to evaluate questionable lesions seen on CT further improves the sensitivity to 95%–97% (23,26), which is particularly helpful for evaluating subcentimeter lesions (27).
Another likely factor contributing to improved outcome is adoption of complete mesocolic excision with central vascular ligation. As described by Hohenberger et al. (9), the procedure is an anatomic-based resection of tumor and locoregional lymphatic drainage that does not breach the visceral fascia, avoiding tumor spread within the peritoneal cavity and ensuring complete lymphadenectomy (28). The Erlangen group reported complete mesocolic excision to be most beneficial for stage III patients, improving 5-year cancer-related survival from 61.7% to 80.9% (29). This is consistent with the present study in that absolute improvement was the most evident in the highest-risk patients. Nonetheless, the Danish Colorectal Cancer Group found that mesocolic excision was independently associated with improved outcome irrespective of stage (10).
A third potential factor is increased lymph node yield, from a median of 14 in 1990–2000 to 22 in 2007–2014, related to increased adoption of more radical lymphadenectomy associated with complete mesocolic excision as well as increased lymph node harvest by pathologists (12,30). Evaluating more lymph nodes increases the likelihood of detecting nodal metastases (31) and may partially explain the higher proportion of stage III patients in the current cohort (35% vs 29% in the 1990–2000 dataset; P = .001). More accurate detection of metastatic lymph nodes also likely resulted in some stage migration from II and III and therefore higher rates of freedom from recurrence both for stage II patients and stage III patients.
Treatment advances also include the addition of oxaliplatin to 5-fluorouracil adjuvant chemotherapy. Most recent reports demonstrate an 11% relative improvement in 5-year disease-free survival in stage III patients treated with FOLFOX4 vs LV5FU (HR = 0.78, 95% CI = 0.65 to 0.93, P = .005) (32). The majority of patients with stage III disease in the 2007–2014 cohort received oxaliplatin-based adjuvant chemotherapy.
Finally, the increase in the proportion of patients with stage II disease receiving adjuvant chemotherapy (from 14% to 25%) may have improved outcomes, although its benefit for these patients is controversial. This change likely reflects increased identification of high-risk features such as lymphovascular invasion and perineural invasion; assessment of these features has become standard since 2000 (30). Both the QUASAR study and a large meta-analysis found only small reductions in recurrence in stage II patients treated with adjuvant chemotherapy (33,34).
An important advantage of nomograms over AJCC staging is that they predict risk along a continuous scale rather than through categories. Despite expansion in the number of AJCC substages to 10 in the current eighth edition, heterogeneity still remains within each substage, as demonstrated by a histogram of estimated risk of recurrence by stage (Supplementary Figure 1, available online). Further, the order of AJCC substages does not correspond to relative risk; patients with stage IIIA colon cancer have better outcomes than those with IIB (Figure 1) (35). The nomogram’s more precise prediction of outcomes can help improve treatment decision making in clinical practice.
The risk recurrence calculator updated herein meets all methodological criteria outlined by the AJCC Precision Medicine Core (36). This is especially notable given that the majority of prognostic tools in colorectal cancer were shown in a recent review to be methodologically deficient (3). Another highly rated tool predicts recurrence in stage III patients and is based on the Adjuvant Colon Cancer End Points (ACCENT) database (37). We compared the concordance index of our nomogram to that of the ACCENT calculator using data from the current cohort. Although evaluated in a small number of patients, the nomograms had similar concordance indices (0.693 for ACCENT, 95% CI = 0.623 to 0.763; and 0.674 for MSK, 95% CI = 0.609 to 0.740) (Supplementary Table 4, available online).
The strengths of our study include the relatively large cohort of patients who underwent a standardized resection procedure. Other strengths include comprehensive histologic assessment by specialized pathologists and the availability of granular clinical and demographic information, including use of chemotherapy. However, the study is subject to the selection bias inherent in observational retrospective studies. Risk models developed in a specialty institution have the potential of limited applicability. However, the MSK Colon Cancer Recurrence Nomogram has been previously validated in a diverse set of cohorts (5–7) and likely remains generalizable. Assessing the model’s performance based on the calibration plot may be overly optimistic, as the same data were used to estimate the new baseline recurrence-free probability function. Further external validation of the recalibrated nomogram with contemporary data (including patients aged older than 75 years) is needed.
In summary, evaluating the MSK recurrence nomogram using a contemporary dataset demonstrated continued discriminating power and allowed an update to the baseline recurrence function. Improvements in outcome are encouraging and likely reflect advancements along the spectrum of care including staging, surgery, and adjuvant chemotherapy. As staging moves to prognostic models and risk calculators rather than classifiers and ordered risk strata (1,3,36), periodic updates are required to maintain predictive accuracy and model validity.
Funding
This work was supported by the National Institutes of Health (P30 CA008748).
Notes
Affiliations of authors: Department of Surgery (TK, YS, IHW, EP, JJS, GMN, JGG, PBP, JGA, MRW) and Department of Epidemiology and Biostatistics (MH, MG) and Department of Medicine (AC, RY, ZKS, NHS, AV, LBS) and Department of Pathology (JS, EV), Memorial Sloan Kettering Cancer Center, New York, NY; Department of Gastroenterological Surgery, Cancer Institute Hospital, Japanese Foundation for Cancer Research, Tokyo, Japan (TK); Division of Digestive and General Surgery, Niigata University Graduate School of Medical and Dental Sciences, Niigata, Japan (YS).
J. Joshua Smith has received travel support from Intuitive Surgical and is an advisor for Endogenesis, Inc. José G. Guillem is a member of the Roche Speaker Bureau. Andrea Cercek has received research funding from AbbVie, Amgen, and Seattle Genetics and serves on the advisory board of Bayer. Neil H. Segal has received research funding from Roche/Genentech, Pfizer, Merck, Bristol-Myers Squibb, MedImmune/AstraZeneca, and Incyte and serves on the advisory boards of Roche/Genentech, Merck, Bristol-Myers Squibb, MedImmune/AstraZeneca, Boehringer Ingelheim, Pfizer, Pieris, PsiOxus, Synlogic, Aduro, Kyn Therapeutics, PureTech Ventures, Horizon Pharma, EMD Serono, Gritstone Oncology, Chugai, TRM Oncology, and IFM Therapeutics. Anna Varghese has received research funding from BioMed Valley, Verastem, Lilly, Bristol-Myers Squibb, and Silenseed. Leonard B. Saltz has received research funding from Taiho Pharmaceuticals.
We gratefully acknowledge Jessica Moore and Arthur Gelmis for editing the manuscript.
Supplementary Material
References
- 1.In: Brierley JD, Gospodarowicz MK, Wittekind C, eds. TNM Classification of Malignant Tumours, 8th ed.Hoboken, NJ: John Wiley & Sons, Inc; 2016. [Google Scholar]
- 2. Weiser MR, Gonen M, Chou JF, et al. Predicting survival after curative colectomy for cancer: individualizing colon cancer staging. J Clin Oncol. 2011;29(36):4796–4802. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3. Mahar AL, Compton C, Halabi S, et al. Personalizing prognosis in colorectal cancer: a systematic review of the quality and nature of clinical prognostic tools for survival outcomes. J Surg Oncol. 2017; 116(8):969–982. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4. Weiser MR, Landmann RG, Kattan MW, et al. Individualized prediction of colon cancer recurrence using a nomogram. J Clin Oncol. 2008;26(3):380–385. [DOI] [PubMed] [Google Scholar]
- 5. Collins IM, Kelleher F, Stuart C, et al. Clinical decision aids in colon cancer: a comparison of two predictive nomograms. Clin Colorectal Cancer. 2012;11(2):138–142. [DOI] [PubMed] [Google Scholar]
- 6. Kazem MA, Khan AU, Selvasekar CR.. Validation of nomogram for disease free survival for colon cancer in UK population: a prospective cohort study. Int J Surg. 2016;27(March):58–65. [DOI] [PubMed] [Google Scholar]
- 7. Liu M, Qu H, Bu Z, et al. Validation of the Memorial Sloan-Kettering Cancer Center Nomogram to predict overall survival after curative colectomy in a Chinese colon cancer population. Ann Surg Oncol. 2015;22(12):3881–3887. [DOI] [PubMed] [Google Scholar]
- 8. Bonanni L, De'liguori Carino N, Deshpande R, et al. A comparison of diagnostic imaging modalities for colorectal liver metastases. Eur J Surg Oncol. 2014;40(5):545–550. [DOI] [PubMed] [Google Scholar]
- 9. Hohenberger W, Weber K, Matzel K, et al. Standardized surgery for colonic cancer: complete mesocolic excision and central ligation--technical notes and outcome. Colorectal Dis. 2009;11(4):354–364; discussion 364–365. [DOI] [PubMed] [Google Scholar]
- 10. Bertelsen CA, Neuenschwander AU, Jansen JE, et al. Disease-free survival after complete mesocolic excision compared with conventional colon cancer surgery: a retrospective, population-based study. Lancet Oncol. 2015;16(2):161–168. [DOI] [PubMed] [Google Scholar]
- 11. Cho MS, Baek SJ, Hur H, et al. Modified complete mesocolic excision with central vascular ligation for the treatment of right-sided colon cancer: long-term outcomes and prognostic factors. Ann Surg. 2015;261(4):708–715. [DOI] [PubMed] [Google Scholar]
- 12. West NP, Kobayashi H, Takahashi K, et al. Understanding optimal colonic cancer surgery: comparison of Japanese D3 resection and European complete mesocolic excision with central vascular ligation. J Clin Oncol. 2012;30(15):1763–1769. [DOI] [PubMed] [Google Scholar]
- 13. André T, Boni C, Mounedji-Boudiaf L, et al. Oxaliplatin, fluorouracil, and leucovorin as adjuvant treatment for colon cancer. N Engl J Med. 2004;350(23):2343–2351. [DOI] [PubMed] [Google Scholar]
- 14.National Comprehensive Cancer Network. NCCN Guidelines for Patients: Colon Cancer, Version 2.2017; 2017. www.nccn.org/professionals/physician_gls/pdf/colon.pdf. Accessed January 27, 2019.
- 15. Harrell FE Jr, Lee KL, Mark DB.. Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Stat Med. 1996;15(4):361–387. [DOI] [PubMed] [Google Scholar]
- 16. Hanley JA, McNeil BJ.. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology. 1982;143(1):29–36. [DOI] [PubMed] [Google Scholar]
- 17. van Houwelingen HC. Validation, calibration, revision and combination of prognostic survival models. Stat Med. 2000;19(24):3401–3415. [DOI] [PubMed] [Google Scholar]
- 18. Hosmer D, Lemeshow S.. Applied Logistic Regression. New York, NY: John Wiley & Sons; 1989. [Google Scholar]
- 19. Hosmer D, Lemeshow S.. Applied Logistic Regression. 2nd ed.New York, NY: John Wiley & Sons; 2000. [Google Scholar]
- 20. Valls C, Lopez E, Gumà A, et al. Helical CT versus CT arterial portography in the detection of hepatic metastasis of colorectal carcinoma. AJR Am J Roentgenol. 1998;170(5):1341–1347. [DOI] [PubMed] [Google Scholar]
- 21. Wernecke K, Rummeny E, Bongartz G, et al. Detection of hepatic masses in patients with carcinoma: comparative sensitivities of sonography, CT, and MR imaging. AJR Am J Roentgenol. 1991;157(4):731–739. [DOI] [PubMed] [Google Scholar]
- 22. Zerhouni EA, Rutter C, Hamilton SR, et al. CT and MR imaging in the staging of colorectal carcinoma: report of the Radiology Diagnostic Oncology Group II. Radiology. 1996;200(2):443–451. [DOI] [PubMed] [Google Scholar]
- 23. Kim YK, Ko SW, Hwang SB, et al. Detection and characterization of liver metastases: 16-slice multidetector computed tomography versus superparamagnetic iron oxide-enhanced magnetic resonance imaging. Eur Radiol. 2006;16(6):1337–1345. [DOI] [PubMed] [Google Scholar]
- 24. Portugaller HR, Stacher R, Komaz G, et al. The value of different spiral CT phases in the detection of liver metastases [in German]. Rofo. 2002;174(4):452–458. [DOI] [PubMed] [Google Scholar]
- 25. Valls C, Andía E, Sánchez A, et al. Hepatic metastases from colorectal cancer: preoperative detection and assessment of resectability with helical CT. Radiology. 2001;218(1):55–60. [DOI] [PubMed] [Google Scholar]
- 26. Sahani DV, Kalva SP, Fischman AJ, et al. Detection of liver metastases from adenocarcinoma of the colon and pancreas: comparison of mangafodipir trisodium-enhanced liver MRI and whole-body FDG PET. AJR Am J Roentgenol. 2005;185(1):239–246. [DOI] [PubMed] [Google Scholar]
- 27. Niekel MC, Bipat S, Stoker J.. Diagnostic imaging of colorectal liver metastases with CT, MR imaging, FDG PET, and/or FDG PET/CT: a meta-analysis of prospective studies including patients who have not previously undergone treatment. Radiology. 2010;257(3):674–684. [DOI] [PubMed] [Google Scholar]
- 28. Heald RJ, Ryall RD.. Recurrence and survival after total mesorectal excision for rectal cancer. Lancet. 1986;327(8496):1479–1482. [DOI] [PubMed] [Google Scholar]
- 29. Merkel S, Weber K, Matzel KE, et al. Prognosis of patients with colonic carcinoma before, during and after implementation of complete mesocolic excision. Br J Surg. 2016;103(9):1220–1229. [DOI] [PubMed] [Google Scholar]
- 30. Compton CC, Fielding LP, Burgart LJ, et al. Prognostic factors in colorectal cancer. College of American Pathologists Consensus Statement 1999. Arch Pathol Lab Med. 2000;124(7):979–994. [DOI] [PubMed] [Google Scholar]
- 31. Gönen M, Schrag D, Weiser MR.. Nodal staging score: a tool to assess adequate staging of node-negative colon cancer. J Clin Oncol. 2009;27(36):6166–6171. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32. Andre T, Boni C, Navarro M, et al. Improved overall survival with oxaliplatin, fluorouracil, and leucovorin as adjuvant treatment in stage II or III colon cancer in the MOSAIC trial. J Clin Oncol. 2009;27(19):3109–3116. [DOI] [PubMed] [Google Scholar]
- 33. Gray R, Barnwell J, McConkey C, et al. Adjuvant chemotherapy versus observation in patients with colorectal cancer: a randomised study. Lancet. 2007;370(9604):2020–2029. [DOI] [PubMed] [Google Scholar]
- 34. Gill S, Loprinzi CL, Sargent DJ, et al. Pooled analysis of fluorouracil-based adjuvant therapy for stage II and III colon cancer: who benefits and by how much? J Clin Oncol. 2004;22(10):1797–1806. [DOI] [PubMed] [Google Scholar]
- 35. O'Connell JB, Maggard MA, Ko CY.. Colon cancer survival rates with the new American Joint Committee on Cancer sixth edition staging. J Natl Cancer Inst. 2004;96(19):1420–1425. [DOI] [PubMed] [Google Scholar]
- 36. Kattan MW, Hess KR, Amin MB, et al. American Joint Committee on Cancer acceptance criteria for inclusion of risk models for individualized prognosis in the practice of precision medicine. CA Cancer J Clin. 2016;66(5):370–374. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 37. Renfro LA, Grothey A, Xue Y, et al. ACCENT-based web calculators to predict recurrence and overall survival in stage III colon cancer. J Natl Cancer Inst. 2014;106(12):dju333. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.