Predicting Overall Survival in Patients with Metastatic Rectal Cancer: A Machine Learning Approach

Abstract

BACKGROUND:

A significant proportion of patients with rectal cancer will present with synchronous metastasis at the time of diagnosis. Rates of OS for these patients are highly variable and previous attempts to build predictive models often have low predictive power, with concordance indexes (c-index) less than 0.70.

METHODS:

Using the National Cancer Database (2010–2014), we identified patients with synchronous metastatic rectal cancer. The data was split into a training dataset (diagnosis years 2010–2012), which was used to build the machine-learning model, and a testing dataset (diagnosis years 2013–2014), which was used to externally validate the model. A nomogram predicting 3-year overall survival was created using Cox proportional hazard regression with lasso-penalization. Predictors were selected based on clinical significance and availability in NCDB. Performance of the machine-learning model was assessed by c-index.

RESULTS:

A total of 4,098 and 3,107 patients were used to construct and validate the nomogram, respectively. Internally validated c-indexes at 1, 2, and 3 years were 0.816 (95% CI 0.813 – 0.818), 0.789 (95% CI 0.786 – 0.790), and 0.778 (95% CI 0.775 – 0.780), respectively. External validated c-indexes at 1, 2, and 3 years were 0.811, 0.779, and 0.778, respectively.

CONCLUSIONS:

There is wide variability in the OS for patients with metastatic rectal cancer, making accurate predictions difficult. However, using machine learning techniques, more accurate models can be built. This will aid patients and clinicians in setting expectations and making clinical decisions in this group of challenging patients.

INTRODUCTION

Colon and rectal cancer is the fourth most common cancer diagnosis in the United States, with an estimated 140,000 new diagnoses and 50,000 deaths annually [1]. The most common metastatic site for patients with rectal cancer is the liver, and approximately 20–25% of patients will present with synchronous hepatic metastases [2, 3]. Other metastatic sites include lung, bone, peritoneum, and the nervous system [4]. Classically, metastatic disease has been considered incurable and is associated with poor overall survival [1]. However, with advances in chemotherapy and shifting paradigms in the definition of a resectable metastases, overall survival rates have improved in recent years [5–8].

In order to assist clinicians and patients with prognostic information and care-planning, predictive models have been built for this population of patients. When evaluating predictive models, the most commonly-used measure is the concordance index (c-index), which estimates the probability of concordance between predicted and observed outcomes [9]. A perfect concordance is 1.0, while a c-index of 0.50 is equatable to a coin-flip. Previous predictive models for metastatic colorectal cancer are often limited to a single metastatic organ (i.e. liver-only, lung-only) and/or have been hampered by c-indexes <0.70 [10–18]. This may be because outcomes in patients with metastatic disease are highly variable due to the wide variety of clinical conditions [19], making building accurate predictive models difficult. In addition, metastatic colon and rectal cancer are often grouped together as a single entity. However, there is mounting evidence that colon and rectal metastases are distinct entities, with different metastatic patterns and outcomes [4, 20]. Therefore, it may be prudent to build a rectum-specific predictive model for patients with metastatic disease.

Previously, models have been built on simple multivariable regression techniques, using either logistic regression or Cox proportional hazard modeling. However, advanced predictive modeling using machine learning can be used to build models that are more accurate, robust, and generalizable. Therefore, our aim was to utilize machine learning techniques to accurately predict 3-year overall survival in patients with metastatic rectal cancer. In addition, in order to make the model more accessible to clinicians and patients alike, we constructed a nomogram representing our predictive model in a graphical format.

MATERIALS AND METHODS

Patient Sample and Variables

The primary goal of this study is to construct accurate predictive nomograms for overall survival for patients with metastatic rectal cancer. Patients were identified from the National Cancer Database (NCDB), a national oncology outcomes database that is jointly sponsored by the American College of Surgeons’ Commissions on Cancer (CoC) and the American Cancer Society. The NCDB contains clinical oncology data sourced from over 1,500 CoC-accredited centers. Using the NCDB, all patients with metastatic rectal adenocarcinoma diagnosed from 2010 to 2014 were identified. Because the NCDB only contains de-identified patient information, this study was exempt from institution review board approval.

The primary tumor was identified as adenocarcinoma by International Classification of Disease for Oncology histology codes (8140–8145, 8210, 8211, 8220, 8221, 8255, 8261–8263, 8310, 8323, 8330–8332, 8480, 8481, 8490, 8525, 8530, 8570–8574). Survival time was defined as the number of months from diagnosis to an event (alive or dead). Because the model required non-zero survival times, survival times reported as 0 were transformed into 0.01 (equating to 0.3 days). Variables were selected due to clinical significance and availability within NCDB. Patient age was defined as the age of the patient at the time of diagnosis. In the NCDB, patient comorbidity is represented by the Charlson-Deyo Comorbidity Score (CDCS), which consists of 15 comorbidities that are assigned various point values [21]. Because of the small proportion of patients with CDCS of greater than 3, the CDCS variables is truncated to scores of 0, 1, 2, or ≥3 in the NCDB. Tumor grade was reported as well-differentiated, moderately-differentiated, poorly-differentiated, and undifferentiated/anaplastic. The CEA level, as defined in NCDB as the highest pre-treatment CEA level, was split into quartiles (≤6ng/mL, 6.1–28.9ng/mL, 29–97.9ng/mL, and ≥98ng/mL). Metastasis at the time of diagnosis to the liver, lung, brain, bone, and peritoneum were identified and reported as binary variables (i.e. yes/no). Resection of the primary tumor site and of a metastatic site (excluding resection of only distant lymph nodes) were also dichotomized. The number of positive lymph nodes was reported as a continuous variable, with aspiration of positive lymph nodes and unknown number of positive lymph nodes considered as one positive lymph node. Patients who had no lymph nodes examined was classified as having no positive lymph nodes. Lastly, any chemotherapy and/or radiation therapy received by the patient were included as separate binary variables in the model.

Nomogram Construction and Validation

Patients were split into a training set (diagnosis year 2010–2012) and testing set (2013–2014). Patients with any missing data were removed from analysis. Differences between the training and testing sets were evaluated using the chi-square test for categorical variables and the Mann-Whitney U test for continuous variables. Kaplan-Meier analysis with log-rank testing was used to compare the overall survival between the sets.

The predictive model was created using a 10-fold cross-validated Cox proportional hazard regression with lasso-penalization. The lasso (least absolute shrinkage and selection operator) is a machine-learning technique that can lead to superior performance over traditional multivariable regression because it performs both variable selection and penalization [22]. Variable selection reduces the number of predictors so that non-significant predictors are removed from the final model, while penalization (also known as regularization) decreases the predictors’ ability to affect the predicted outcome. By performing both variable selection and penalization, the lasso is able to build accurate models without under-fitting or over-fitting the training data. For our analysis, we combined properties of the lasso (i.e. variable selection and penalization) with Cox proportional hazard analysis, in which predictors for overall survival was subject to selection and penalization. The model was designed to predict the 3-year probability of overall survival for these patients. The machine-learning model formed the basis of the nomogram, with each predictor assigned points that can be summed to determine the 3-year probability of overall survival.

In evaluating the performance of our model, we employed both calibration (external) and validation (internal and external) assessments [23]. In external calibration, we determined how well our predictive model fits observed data from the testing dataset. We did this by stratifying patients in the testing dataset into 5 risk groups and reporting the predicted and actual probability of survival at 3 years for each risk group. In internal validation, we tested the congruence between model predictions and observed data in the training dataset by 100-repetition bootstrap resampling to determine the median time-dependent c-indexes at 1, 2, and 3 years with 95% confidence intervals. For external validation, we applied our predictive model to the observed data from the testing dataset to determine the time-dependent c-indexes at 1, 2, and 3 years.

All statistical and machine-learning analyses were performed using R (Version 3.3.2, R Foundation, Vienna, Austria) and R package hdnom [24]. The level of significance was set at 0.05 and all comparisons are two-tailed.

RESULTS

After exclusion, a total of 4,098 patients with rectal cancer were included in the training dataset, and a total of 3,107 patients with rectal cancer were included in the testing dataset. A comparison of the training and testing cohorts is shown in Table 1. The Kaplan-Meier analysis of the training and testing datasets is shown in Figure 1. The training group (21.7 months, 95% CI 20.9 – 22.8) had significantly shorter median OS than the testing group (24.6 months, 95% CI 23.5 – 25.7, p=0.002).

Table 1.

Comparison of the training and validation datasets for rectal cancer patients.

	Rectum
	Training	Validation	p-value
Total Number of Patients	4059	3069	-
Median Age (range)	60 (18 – ≥90)	60 (19 – ≥90)	0.437
Charlson-Deyo Score			0.984
0	3176 (78.2%)	2404 (78.3%)
1	681 (16.8%)	511 (16.7%)
2	130 (3.2%)	102 (3.3%)
≥3	72 (1.8%)	52 (1.7%)
Grade Differentiation			0.172
Well	269 (6.6%)	212 (6.9%)
Moderate	2949 (72.7%)	2286 (74.5%)
Poor	776 (19.1%)	525 (17.1%)
Undifferentiated	65 (1.6%)	47 (1.5%)
Highest CEA Level		46	0.282
≤6ng/mL	1096 (27.0%)	834 (27.2%)
6.1–28.9ng/mL	1177 (29.0%)	839 (27.3%)
29–97.9ng/mL	768 (18.9%)	573 (18.7%)
≥98ng/mL	1018 (25.1%)	823 (26.8%)
Liver Metastasis	3299 (81.3%)	2503 (81.6%)	0.763
Lung Metastasis	1506 (37.1%)	1156 (37.7%)	0.626
Bone Metastasis	344 (8.5%)	229 (7.5%)	0.119
Brain Metastasis	68 (1.7%)	48 (1.6%)	0.713
Peritoneal Metastasis	1050 (25.9%)	844 (27.5%)	0.122
Primary Site Resected	1565 (38.6%)	1158 (37.7%)	0.478
Non-Primary Site Resected	692 (17.0%)	567 (18.5%)	0.118
Median # of Positive LNs (range)	0 (0 – 50)	0 (0 – 37)	0.081
Any Chemotherapy	3372 (83.1%)	2647 (86.2%)	<0.001
Any Radiation Therapy	1732 (42.7%)	1248 (40.7%)	0.089

Figure 1.

Kaplan-Meier curves comparing the training and testing datasets for patients with metastatic rectal cancer.

The nomogram for patients with metastatic rectal cancer is shown in Figures 2. For the nomogram, predictors are assigned a range of points that, when totaled, will equate to a given predicted probability of overall survival at 3 years. The higher the points that a patient receives, the worse their 3-year OS. The categorical variable receiving the most amount of points is omission of any type of chemotherapy (68 points). The categorical variable receiving the least amount of points is the presence of lung metastasis (4 points). In creating the machine-learning model, radiation therapy dropped out of the model as a predictor for 3-year overall survival.

Figure 2.

Nomograms predicting the 3-year overall survival for patients with metastatic rectal cancer.

External calibration, stratified into 5 risk-groups, is shown in Table 2. On external calibration, there was no significant difference in the predicted 3-year OS from the model compared to the actual 3-year OS in the testing dataset in 4 out of 5 risk groups. The predicted OS was outside of the 95% confidence interval of the actual 3-year OS for the highest risk group. The time-dependent internal and external validation at 1, 2, and 3 years are shown in Table 3. On internal validation, the c-indexes at 1, 2, and 3 years were 0.816 (95% CI 0.813 – 0.818), 0.789 (95% CI 0.786 – 0.790), and 0.778 (95% CI 0.775 – 0.780), respectively. On external validation, the c-indexes at 1, 2, and 3 years were 0.811, 0.779, and 0.778, respectively.

Table 2.

External calibration showing probability of 3-year OS for all cohorts

Risk Group	Observed (95% CI)	Predicted
1 (Highest Risk)	0.096 (0.071 – 0.131)	0.013*
2	0.175 (0.138 – 0.220)	0.179
3	0.289 (0.244 – 0.341)	0.293
4	0.495 (0.445 – 0.550)	0.447
5 (Lowest Risk)	0.616 (0.565 – 0.670)	0.644

^*Outside of 95% CI of observed probability

Table 3.

Time-dependent internal and external validation for all cohorts represented by area under the ROC

Year	Internal Validation (95% CI)	External Validation
1	0.816 (0.813 – 0.818)	0.811
2	0.789 (0.786 – 0.790)	0.779
3	0.778 (0.775 – 0.780)	0.778

DISCUSSION

Patients with metastatic rectal cancer have high variability in overall survival due to their disease process, making accurate predictive models challenging in this patient population. However, by using a large nationwide oncology database and harnessing the predictive power of machine learning, we were able to construct nomograms for 3-year OS in metastatic rectal cancer patients with superior accuracy to those previously published [11, 12, 17, 18, 25].

Advances in the treatment of patients with metastatic rectal cancer is highlighted by the significant increase in median OS between the training and testing cohorts. The improvement in OS for these patients may be due to use of newer and more effective systemic agents and regimens [8, 26] and shifting paradigms in the definition of “resectable” metastases [27, 28]. This may have contributed to the significant increase in the proportion of patients who received any type of chemotherapy in the testing cohort. It was somewhat surprising that the proportion of patients who underwent resection of their rectal cancer was only 38.6% and 37.7% in the training and testing datasets, respectively. While there is equivocal evidence regarding benefits to resecting the primary site [29–31], it seems like this strategy is less commonly employed in metastatic rectal cancer compared to metastatic colon cancer [32]. This is likely due to the increased risk of morbidity associated with rectal resection. Likewise, the proportion of patients who had their metastatic site resected was only 17.0% and 18.5% in the training and testing datasets, respectively. While this difference did not reach statistical significance, it does suggest a small increase in the rate of metastasectomy in patients with metastatic rectal cancer.

In order to account for potential factors that are not captured in NCDB, models were built on patients diagnosed from 2010–2012, and were externally calibrated and validated on patients diagnosed from 2013–2014. This allowed us to incorporate time trends without explicitly including the year of diagnosis as a predictor (which would have limited use of these nomograms on future patients). This also prevented the training and testing data sets from being too similar, decreasing the risk of over-fitting the model to the training data. Because the testing dataset included patients diagnosed in 2013, we were not able to use this dataset to externally validate 5-year overall survival, limiting our analysis to 3-year overall survival. This is not uncommon in analysis of metastatic rectal cancer [11, 12, 33]. In the NCDB, 3-year OS for patients with metastatic disease is approximately 30%, highlighting the deadly nature of this disease despite advances in the field.

In our nomogram, the number of points assigned to each predictor is a measure of the predictor’s effect on 3-year OS. The number of points is determined by the lasso regression and produced some interesting results. For example, as seen in our study and in previous literature, the most common metastatic site for patients with rectal cancer is the liver [4]. However, bone metastasis, while much rarer, confers a far worse prognosis for patients [34]. Therefore, in our nomogram, the presence of bone metastasis is given more points than the presence of liver metastasis. The same phenomenon is true for the presence of peritoneal and brain metastasis, which are also associated with worse prognosis compared to liver metastases [35, 36]. However, metastasis to the lung was much less predictive of OS compared to metastasis to liver, peritoneum, bone, and/or brain in patients with metastatic rectal cancer. Though the lung represents the second most common metastatic site for patients with rectal cancer [4], overall survival after pulmonary metastasectomy has improved dramatically in the current era [37]. This may have led to a relative lack of predictive power of pulmonary metastasis, especially when compared to the effect of other metastatic sites on OS [4, 34–36]. This “crowding out” effect of pulmonary metastasis is seen in previous studies, in which the addition of lung metastasis to liver metastasis did not affect OS [38], and lung metastasis-associated variables were not prognostic for OS [18].

Though radiation therapy was included as a possible predictor, our lasso regression “selected” it out of the model. Therefore, because the nomogram is built from the lasso regression model, radiation therapy does not appear on the nomogram. This was unexpected, but may be because the role of radiation therapy in patients with metastatic rectal cancer is not well-defined, with previous studies yielding conflicting results [39–41]. Therefore, it may be possible that radiation therapy did not contribute significantly to the 3-year OS of these patients, leading the variable to be dropped by the lasso regression. In addition, radiation therapy may also have been “crowded out” by other predictors that contribute more to 3-year OS. For example, the predictor with the largest amount of points in the nomogram is treatment with any type of chemotherapy. However, of the 2,980 patients who received any radiation therapy, 2,760 (92.6%) also received chemotherapy. Therefore, it is possible that the effect of chemotherapy on OS simply “crowded out” the effects that radiation therapy had on OS.

Utilization of machine-learning techniques allowed us to build a predictive model with superior c-indexes compared to previous models [10–18]. However, accurate predictions for this cohort of patients remains challenging. When the testing dataset is split into 5 risk groups, our lasso model was able to accurately predict the 3-year OS in 4 out of 5 risk groups. The lasso model significantly under-predicted the 3-year OS for patients in the highest risk group. This may be related to the fact that there are predictors that contribute to survival in patients with high risk of death that are not captured in the NCDB, especially predictors that are not oncologic in nature (i.e. medical comorbidities, surgical complications).

Many consider machine learning as a “black box”, in which predictions are generated by a computer. Unfortunately, most clinicians have limited understanding of the machinations involved to generate these predictions. While medicine remains behind other disciplines in utilizing machine learning, its predictive power has been demonstrated with increasing frequency [42–44]. The lasso regression used in this study is a more approachable form of machine learning because it is based on multivariable regression. However, one advantage of the lasso over traditional multivariable regression is its ability to perform both variable selection and penalization. We also believe that techniques such as the lasso can also make nomograms easier to use. Because nomograms are graphical representations of complex algorithms, they can become too cumbersome and complex if too many predictors are included. The lasso is able to “select out” predictors that do not adequately contribute to predictive accuracy. In this study, radiation therapy was “selected out” by the lasso algorithm. This allows our nomogram to remain accurate but also user-friendly for both clinicians and patients alike, which makes it a useful tool in the shared decision-making process that is key in oncology care [45–47].

Limitations

These nomograms have certain limitations. First, the nomograms are constructed using a retrospective nationwide database. While this provides a large training and testing dataset, it limits the predictors that can be used to construct the dataset. There may be other predictors that are not included in NCDB that may be more predictive of OS, which could potentially make the models even more accurate. For example, the NCDB does not collect data on the specific chemotherapy regimen for each patient (e.g. FOLFOX versus FOLFIRI), nor whether a patient received targeted therapy (e.g. bevacizumab or erlotinib). In addition, emerging treatment options for patients with metastatic rectal cancer, such as immunotherapy, is not yet robustly recorded in the NCDB. However, our model can be updated with new data points as they are added to the NCDB, something that should be done regularly for all predictive models in order to provide accurate predictions. Our predictive outcome is limited to OS, as the NCDB does not capture disease-specific survival or recurrence data. The NCDB only includes metastases to the liver, lung, brain, bone, or peritoneum. While this covers the majority of metastatic sites for rectal tumors, rarer sites are not included in this analysis and the use of this nomogram in these patients may be limited. The NCDB does not collect data on the extent of metastases (e.g. the number of metastatic lesions), nor does it collect the location of lymph node metastases, though the number of positive lymph nodes is included in our analysis. Not all metastatic sites are biopsy-proven but are determined by clinical evidence. Inclusion of only biopsy-proven metastases could potentially limit our sample size and introduce additional bias. In addition, the NCDB only records metastatic sites at the time of diagnosis, so these nomograms are limited to patients who present with synchronous metastatic disease. Because the machine learning model can only be run with complete data, we excluded all patients with missing data. While imputation can be used to replace missing values, our large sample sizes ensured that sufficient power can be achieved with listwise deletion. Though we split our dataset into distinct training and testing datasets, the optimal method to externally validate our models is to use a separate dataset. However, because the NCDB covers such a high proportion of cancer diagnoses in the United States, it may be challenging to find patients that are not already represented in the NCDB. A potential solution is to prospectively collect validation data, though this is time-consuming and potentially unfeasible. Lastly, the NCDB contains data only from CoC-accredited centers. While that includes >1,500 centers across the U.S. and represents >70% of all cancer cases, the outcomes of patients at these institutions may be different than those treated at non-CoC-accredited centers, which may limit the generalizability of these models.

CONCLUSION

To our knowledge, this is the first application of machine learning to construct predictive models in metastatic rectal cancer. These models have superior performance compared to the currently available predictive models, showcasing the predictive power of machine learning. Nomograms created from these models can be of great assistance to both clinicians and patients in the treatment of metastatic rectal cancer.