Can Machine-learning Techniques Be Used for 5-year Survival Prediction of Patients With Chondrosarcoma?

Abstract

Background

Several studies have identified prognostic factors for patients with chondrosarcoma, but there are few studies investigating the accuracy of computationally intensive methods such as machine learning. Machine learning is a type of artificial intelligence that enables computers to learn from data. Studies using machine learning are potentially appealing, because of its possibility to explore complex patterns in data and to improve its models over time.

Questions/purposes

The purposes of this study were (1) to develop machine-learning algorithms for the prediction of 5-year survival in patients with chondrosarcoma; and (2) to deploy the best algorithm as an accessible web-based app for clinical use.

Methods

All patients with a microscopically confirmed diagnosis of conventional or dedifferentiated chondrosarcoma were extracted from the Surveillance, Epidemiology, and End Results (SEER) Registry from 2000 to 2010. SEER covers approximately 30% of the US population and consists of demographic, tumor characteristic, treatment, and outcome data. In total, 1554 patients met the inclusion criteria. Mean age at diagnosis was 52 years (SD 17), ranging from 7 to 102 years; 813 of the 1554 patients were men (55%); and mean tumor size was 8 cm (SD 6), ranging from 0.1 cm to 50 cm. Exact size was missing in 340 of 1544 patients (22%), grade in 88 of 1544 (6%), tumor extension in 41 of 1544 (3%), and race in 16 of 1544 (1%). Data for 1-, 3-, 5-, and 10-year overall survival were available for 1533 (99%), 1512 (98%), 1487 (96%), and 977 (63%) patients, respectively. One-year survival was 92%, 3-year survival was 82%, 5-year survival was 76%, and 10-year survival was 54%. Missing data were imputed using the nonparametric missForest method. Boosted decision tree, support vector machine, Bayes point machine, and neural network models were developed for 5-year survival. These models were chosen as a result of their capability of predicting two outcomes based on prior work on machine-learning models for binary classification. The models were assessed by discrimination, calibration, and overall performance. The c-statistic is a measure of discrimination. It ranges from 0.5 to 1.0 with 1.0 being perfect discrimination and 0.5 that the model is no better than chance at making a prediction. The Brier score measures the squared difference between the predicted probability and the actual outcome. A Brier score of 0 indicates perfect prediction, whereas a Brier score of 1 indicates the poorest prediction. The Brier scores of the models are compared with the null model, which is calculated by assigning each patient a probability equal to the prevalence of the outcome.

Results

Four models for 5-year survival were developed with c-statistics ranging from 0.846 to 0.868 and Brier scores ranging from 0.117 to 0.135 with a null model Brier score of 0.182. The Bayes point machine was incorporated into a freely available web-based application. This application can be accessed through https://sorg-apps.shinyapps.io/chondrosarcoma/.

Conclusions

Although caution is warranted, because the prediction model has not been validated yet, healthcare providers could use the online prediction tool in daily practice when survival prediction of patients with chondrosarcoma is desired. Future studies should seek to validate the developed prediction model.

Level of Evidence

Level III, prognostic study.

Introduction

Chondrosarcomas are one of the most common primary malignancies of bone, second only to osteosarcomas [30]. They range from low-grade tumors that rarely metastasize to high-grade, aggressive tumors with high metastatic potential. Treatment relies primarily on surgical resection, because chondrosarcomas are insensitive to conventional radiation and chemotherapy [11, 24, 31]. Identifying prognostic factors for outcomes such as the development of metastasis, recurrence, or survival is important both for patients and their physicians. These can give insight into the patient’s future prognosis, help in deciding the optimal treatment strategy, and determine which patients might need more extensive and closer followup. Several prognostic factors have been identified such as tumor size, tumor grade, histologic subtype, age, margin status, and metastasis at presentation [1, 6, 15, 20].

Few studies have translated these factors into predictive models that can be used in the clinical setting, and moreover, no study has used machine learning for that purpose [44].

The power of machine learning is rapidly transforming health care as it previously has transformed other industries [10, 39]. Machine learning is an approach that uses modern computer and mathematic algorithms to recognize complex combinations of predictors with the capacity of handling huge amounts of data. It has gained acceptance in a wide range of medical fields, including analyzing genetic sets, classifying diseases, predicting survival, and predicting drug response [4, 32, 33, 37, 40, 51]. In musculoskeletal oncology, a number of studies have used machine learning for pathology diagnosis, classification, and outcome prediction [12, 18, 19, 25, 34, 51]. However, to our knowledge, no study exists that uses machine learning to create algorithms for primary bone tumors such as chondrosarcoma.

Our primary study aim is to develop machine-learning algorithms for the prediction of 5-year survival in patients with chondrosarcoma. Our secondary study aim is to deploy the best algorithm as an accessible web-based app for clinical use.

Patients and Methods

For this retrospective cohort study, the Surveillance, Epidemiology, and End Results (SEER) database was used. SEER is the cancer registry of the National Cancer Institute that covers approximately 30% of the US population, reporting patient demographics, tumor characteristics, treatment modalities, and survival for patients [50]. The last version of this database has been updated until the end of 2015. Several studies on chondrosarcoma have previously used this database [3, 7, 13, 14, 21, 27, 28, 38, 43, 44, 48]. The SEER database has a few shortcomings: it has a limited number of variables available, categories of variables have been predefined and therefore detailed information is lost, and not all outcomes of interest such as local recurrence are present. However, for this specific study, most (outcome) variables of interest are available, and the high number of patients and the fact that the data come from multiple institutions make the database suitable for our purpose of creating a machine-learning model for a general population.

Patients between 2000 and 2010 were included in this study. Patients were identified with an International Classification of Diseases for Oncology, 3rd edition diagnosis of chondrosarcoma of the bone and extracted using SEER*Stat software [49]. The inclusion criteria were (1) microscopic confirmation of a conventional or dedifferentiated chondrosarcoma; and (2) surgically treated chondrosarcoma.

Our outcome of interest was 5-year survival. Demographic data consisted of age (years at diagnosis), race (white, black, Asian, other), sex (male, female), and year of diagnosis. Other explanatory variables were based on previous studies and included histologic subtype (conventional chondrosarcoma, dedifferentiated chondrosarcoma), tumor grade as recorded by SEER (well differentiated, moderately differentiated, poorly differentiated), tumor size (cm), tumor extension (localized, local invasion, distant metastasis), and location (extremities, spine, pelvic bones and sacrum, and rib, sternum, and clavicle) [1–3, 6, 8, 17, 21, 31, 42, 48].

From 2000 to 2010, 1544 patients with a microscopically confirmed diagnosis of a conventional or dedifferentiated chondrosarcoma of the extremities, spine, pelvis, rib, sternum, or clavicle were identified. Of these patients, 340 of 1544 (22%) were missing exact size, 88 of 1544 (6%) were missing grade, 41 of 1544 (3%) were missing tumor extension, and 16 of 1544 (1%) were missing race. Two of the patients had all four variables missing; 12 patients had size, grade, and tumor extension missing; and 36 patients had only size and grade missing. Mean age at diagnosis was 52 years (SD 17), ranging from 7 to 102 years, and 813 of the 1544 patients (53%) were men (Table 1). The mean tumor size was 8 cm (SD 5.7), ranging from 0.1 to 50 cm; 851 of the 1544 patients (57%) had localized disease at diagnosis, 557 of the 1544 (37%) had locally invasive disease, and 95 (6%) had distant metastasis. With regard to tumor location, 915 of the 1544 (59%) were located in the extremities; 289 of the 1544 (18%) in the rib, sternum, or clavicle; 264 of the 1544 (17%) in the pelvic bones or sacrum; and 76 of the 1544 (5%) in the spine.

Table 1.

Baseline characteristics of patients with chondrosarcoma (n = 1554)

Data for 1-, 3-, 5-, and 10-year overall survival were available for 1533 (99%), 1512 (98%), 1487 (96%), and 977 (63%) patients, respectively. One-year survival was 92%, 3-year survival was 82%, 5-year survival was 76%, and 10-year survival was 54%.

Application Development

The best algorithm for predicting 5-year mortality, based on performance metrics described subsequently, was incorporated into an interactive interface and deployed as a freely available and user-friendly app. The purpose of this app was to provide 5-year survival probabilities to clinicians after entering patients’ data in the pretrained machine-learning algorithm. The app was programmed to be accessible and adaptable for use on desktops, tablets, and smartphones.

Statistical Analysis

Baseline characteristics were calculated as frequencies and percentages for categorical variables, whereas mean and SD were used for continuous variables. Missing data for size, grade, tumor extension, and race were imputed using the nonparametric missForest method [45]. Variables used for the development of machine-learning models included age, sex, histologic subtype, tumor size, grade, tumor extension, location, and whether the patient underwent surgery. Boosted decision tree, support vector machine, Bayes point machine, and neural network models were developed to predict 5-year survival and model performance was assessed with 10-fold crossvalidation. These models were chosen based on prior studies of machine-learning algorithms for binary classification [16, 35, 50]. Tenfold crossvalidation was chosen because it has lower bias compared with bootstrapping and previous work demonstrated favorable generalizability of model performance as assessed with crossvalidation [29].

Support vector machines are kernel-based algorithms that transform the feature space into high-order dimensions to define a hyperplane for separation. The multilayer perceptron neural networks used in this study are models that map input features into outputs through input, hidden, and output nodes with connections and weights modified by backpropagation of error to optimize the links between input features and desired outputs. Boosted decision trees are extensions of decision tree models with the creation of multiple simple decision tree models and sequentially improving performance by increasing the weight of misclassified examples [22]. The support vector machine, neural network, and boosted decision tree model hyperparameters were tuned through parameter sweeps and final hyperparameters selected for the models were lambda = 0.0001 for the support vector machine, number of leaves = 5, minimum leaf instances = 19, learning rate = 0.154, number of trees = 479 for the boosted decision tree; and learning rate = 0.031, loss function = squared error, and momentum = 0 for the neural network. Bayes point machine is a kernel-based algorithm that seeks to approximate the Bayes optimal decision curve and has been shown to perform well on generalizability to new data. The algorithm was trained through the expectation propagation method [23, 41].

Model performance was measured separately by measures of discrimination (area under the curve [AUC] and plots for receiver operator, calibration, and by overall performance [Brier score]) [47]. The AUC, or c-statistic, is a score ranging from 0.50 to 1.0 with 1.0 indicating the highest discriminating score and 0.50 indicating the lowest discriminating score. The receiver operating characteristic curve plots the false-positive rate on the x-axis and the true-positive rate on the y-axis. Calibration was assessed with calibration intercept and calibration slope and calibration plots [46, 47]. Calibration refers to the agreement between predicted and observed outcomes. A calibration plot with perfect calibration has an intercept of 0 and a slope of 1. Overall model performance was assessed using the Brier score. A Brier score of 0 indicates perfect prediction, whereas a Brier score of 1 indicates the poorest prediction. For a correct interpretation of the Brier score, the prevalence of the outcome must be taken into account. For that purpose, the Brier score of the null model is calculated, which assigns each patient a probability equivalent to the prevalence of the outcome, which is death in this study [5, 47].

Microsoft Azure (Redmond, WA, USA), the Anaconda Distribution (Continuum Analytics, Austin, TX, USA) with RStudio (Version 1.0.153; Boston, MA, USA), Python Version 3.6 (Python Software Foundation, Wilmington, DE, USA), and StataCorp 2013 (Stata Statistical Software: Release 13; StataCorp LP, College Station, TX, USA) were used for data analysis, model creation, and web application development.

Results

In general, there was slight superiority in our performance metrics favoring the Bayes point machine over the other models for 5-year survival prediction (Table 2). The performance of the decision tree, support vector machine, Bayes point machine, and neural network models as measured by c-statistic ranged from 0.846 to 0.868 (Fig. 1). Model performance as assessed on calibration ranged from calibration plot intercept -0.109 to 0.006 and calibration plot slope 0.326 to 1.025 (Fig. 2). For overall performance, Brier scores ranged from 0.117 to 0.135.

Table 2.

Machine learning model performance for 5-year chondrosarcoma survival prediction (n = 1487)

Fig. 1 A-D.

The receiver operating curves for 5-year survival are shown here for the (A) decision tree; (B) support vector machine; (C) Bayes point machine; and (D) neural network models (n = 1487).

Fig. 2 A-D.

This figure shows the calibration with 95% confidence intervals for 5-year survival for the (A) decision tree; (B) support vector machine; (C) Bayes point machine; and (D) neural network models (n = 1487).

Although the performance metrics of all models were comparable, the Bayes point machine had slightly better performance metrics than the other models and was therefore incorporated into a web-based application and deployed as a freely available tool for clinicians. The web application is accessible on desktops, tablets, and smartphones and can be found on our webpage: https://sorg-apps.shinyapps.io/chondrosarcoma/.

Discussion

Patients with cancer often want to know as much as possible about their disease and prognosis, and good predictive tools can help clinicians provide patients with that information [9, 26, 36]. For chondrosarcoma, the second most common primary malignancy of bone, several studies have focused on identifying prognostic factors for this group of patients, but few have attempted to translate them to a predictive model and provided a web-based tool for clinicians [1, 6, 15, 20]. In this study, we successfully created machine-learning algorithms to predict 5-year overall survival in patients with chondrosarcoma. Not only were we able to demonstrate high performance of these models, as assessed by discrimination, calibration, and overall performance, but in addition, we developed a user-friendly and accessible web-based application to obtain predictions.

We acknowledge that our study has several limitations that should be taken into consideration when interpreting the results of this study. First and foremost, the algorithm developed in this study has yet to be externally validated. External validation is important to assess the real performance of a model and to determine whether the model is indeed applicable to a general population. By making the model publicly accessible, we hope to encourage other institutions to validate the model. Based on performance metrics, the Bayes point machine seemed the most favorable with these data. Validation with external cohorts may prove another model to be better. Second, because the data used for this study were taken from a national database, they are subject to limitations such as a lack of certain variables that have previously been described as important factors. Surgical margin lacks, for instance, which has been described as an important prognostic factor in several studies [20, 31], although it has been refuted in others [1, 17]. Because it may be a modifiable factor, it is unfortunate that we were unable to take it into account in our model. The variable “surgery” has been dichotomized into [yes, no]. Unfortunately, we have no information on the type of surgery performed. Like surgical margin, the type of surgery may be associated with survival and may be modifiable. Future studies could seek to assess whether this factor is important and use it to improve the algorithms. The presence of a pathologic fracture is another missing variable that has been found to be of importance by some studies [1], but not by others [31]. Third, the SEER database has a considerable amount of missing data. For this study, 22% of the included patients missed information on exact size, 6% missed grade, and 3% missed tumor extension. All missing variables were imputed, and there was no difference in survival between patients with and without exact size (p = 0.884 for 5-year survival). Fourth, although the data for this study came from multiple institutions, it remains to be seen if the models are applicable for the general population, especially outside of the United States. Fifth, the performance metrics of the support vector machine, Bayes point machine, and neural network were only marginally different. We chose to deploy the Bayes point machine as a web-based application as a result of its slightly better performance, but future reassessment with new data may favor other models.

The primary aim of this study was to develop machine-learning algorithms for the prediction of 5-year survival. All models were assessed on performance using different metrics and all four models performed well. Because we did not have an external validation cohort at our disposal, it remains to be seen if the final model we chose is the best one. New data can help in further training the models and improving them. In the future, we aim to use our institution’s data to externally validate our cohort, to possibly compare it with estimations of clinicians, and to compare it with other prediction models. Recently, a nomogram was created to predict 3- and 5-year overall survival and cancer-specific survival in patients with Grade II and Grade III chondrosarcoma using the SEER database from 1988 to 2011 [44]. The nomogram was constructed using regression analyses by using 50% of their data for the training set and 50% for the validation set. The authors claim to have validated the nomogram internally using part of the training set and externally using the remaining 50% of the cohort. However, strictly speaking this is not a form of external validation, because it used the same database of patients. They managed to achieve a c-statistic for overall survival of 0.803 in the cohort for “internal” validation and 0.753 in the cohort for “external” validation. In the current study, we achieved c-statistics ranging from 0.846 to 0.868 and model performance was assessed using 10-fold crossvalidation, which has lower bias and variance compared with split-sample validation. Model calibration was graphically evaluated with calibration plots over the full range of predicted probabilities and numerically by providing calibration intercept and slope. Discrimination alone does not provide enough information on the validity of the algorithm, and assessing calibration and overall performance is very important to determine the value of a proposed model before deploying it for clinical use [46].

The best performing model, the Bayes point machine, was successfully deployed as a web-based, freely accessible tool for clinical use (https://sorg-apps.shinyapps.io/chondrosarcoma/). For example, when inputting the features of a 59-year-old male patient with a moderately differentiated, localized conventional chondrosarcoma of the extremity of 13.7 cm in the application, the algorithm outputs a 5-year survival probability of 76.9%.

To the best of our knowledge, this is the first available predictive model for predicting survival in chondrosarcoma on an online platform. By translating the machine-learning algorithms to a freely available online platform, we provide healthcare professionals with an easily accessible prediction tool that is readily available on desktops, tablets, and smartphones. Although caution is warranted, because the prediction model has not yet been validated, healthcare providers could access the online tool and use it in daily practice when survival prediction of patients with chondrosarcoma is wanted. While we are actively working on the validation of the algorithm, by releasing it, we encourage others to validate and possibly improve it with their single- or multiinstitutional data. To do so, the eight variables used for the algorithm should be collected for patients with chondrosarcoma with at least 5-year followup, and performance should be assessed by the same performance metrics as used by the current study.