Abstract
Gastric cancer with liver metastasis (GCLM) often has a poor prognosis. Therefore, it is crucial to identify risk factors affecting their overall survival (OS) and cancer-specific survival (CSS). This study aimed to construct practical machine learning models to predict survival time and help clinicians choose appropriate treatments. We reviewed the clinical and survival data of GCLM patients from 2010 to 2017 in the Surveillance, Epidemiology, and End Results (SEER) databases and divided the patients into training and testing groups. The risk factors affecting OS and CSS were determined by least absolute shrinkage and selector operator (LASSO), univariate cox regression, best subset regression (BSR) and the stepwise backward regression. Then, five machine learning models, including random survival forest (RSF), Gradient Boosting Machine (GBM), the Cox proportional hazard (CPH), Survival Support Vector Machine (survivalSVM), and eXtreme Gradient Boosting (XGBoost), were built using the identified risk factors. The model with the best predictive ability was determined using concordance index (c-index), area under the curve (AUC), brier score, and decision curve analysis (DCA), and externally verified with data from 233 cases diagnosed with liver metastasis of cancer from The Shijiazhuang People’s Hospital, Jinan City People’s Hospital, and The Sixth People’s Hospital of Huizhou from 2017 to 2018. The study involved a total of 1300 GCLM patients. The prognostic risk factors affecting OS and CSS were the same, including grade, histology, T stage, N stage, surgery, and chemotherapy. The XGBoost model was found to have the best predictive ability for OS, with AUC of 0.891 [95% CI 0.841-0.941], brier score of 0.061 [95% CI 0.046-0.076], and c-index of 0.752 [95% CI 0.742-0.761], as well as for CSS, with AUC of 0.895 [95% CI 0.848-0.942], brier score of 0.064 [95% CI 0.050-0.079], and c-index of 0.746 [95% CI 0.736-0.756]. The AUC score, brier score and c-index all illustrated the accuracy of the model, and the validation using the external datasets further confirmed the reliability of the model. Therefore, the XGBoost model demonstrated significant potential in predicting survival times and selecting appropriate treatment plans.
Keywords: Gastric cancer, liver metastasis, overall survival, cancer-specific survival, machining learning, XGBoost
Introduction
Gastric cancer ranks the fifth in incidence and is the third leading cause of cancer-related death worldwide [1]. Patients with gastric cancer are often diagnosed at an advanced stage, which results in a poor prognosis [2,3]. Advanced gastric cancer is particularly prone to metastasis, especially to the liver, peritoneum, and lungs. Liver metastases occur in approximately 18%-20% of gastric cancer patients with distant metastases, leading to a median overall survival (OS) of about 5 months [4,5].
Currently, neoadjuvant chemotherapy is the primary treatment for gastric cancer with liver metastasis (GCLM). Emerging chemotherapy drugs such as S-1, irinotecan, and taxanes [6-8] have shown potential in extending patient survival. However, the role of gastrectomy in this context remains controversial [9-13]. Studies have reported that the OS of patients with and without gastrectomy ranges from 8.0 to 16.3 months and 2.4 to 6.8 months, respectively [9,10]. Conversely, Terashima et al. analyzed 253 gastric cancer patients with synchronous distant metastasis and found that GCLM patients did not gain a survival benefit from gastrectomy [12]. This inconsistency may due to the limitations in sample size and ethnicity, which impact the generalizability of their conclusions. Thus, it is crucial to determine whether surgical intervention is more effective in improving survival time compared to neoadjuvant therapy in each individual patient. Meanwhile, identifying risk factors and discovering clinically beneficial methods to predict prognosis and provide effective interventions for gastric cancer patients with liver metastasis is essential. Machine learning (ML) is increasingly utilized due to its powerful ability to process large amounts of data and achieve higher prediction accuracy [14-16]. ML models can be reliable tools for predicting the prognosis of GCLM. However, to date, there are no ML models specifically predict the prognosis of patients with this condition.
Based on this, we intend to use data from the SEER database of GCLM to analyze prognostic risk factors and construct clinically applicable ML models for predicting the OS and cancer-specific survival (CSS).
Methods
SEER data source and patients
We downloaded the data using the SEER*Stat Software (version 8.4.0.1) and selected the exact data from the “Incidence - SEER Research Plus Data, 17 Registries, Nov 2021 Sub (2000-2019) - Linked To County Attributes - Time Dependent (1990-2019) Income/Rurality, 1969-2020 Counties, National Cancer Institute, DCCPS, Surveillance Research Program”, released in April 2022, based on the November 2021 submission.
The exclusion criteria include: (I) patients without liver metastases; (II) lacking positive histology; (III) patients with one primary tumor only who had a prior tumor; (IV) lacking survival months and CSS; (V) lacking complete TNM stage, tumor size, and race data; (VI) lacking surgery information.
Statistical analysis and model establishment
The demographic, clinical and treatment features were collected, including age, race, sex, marital status, histological type, T stage, N stage, surgery, chemotherapy, and radiotherapy. We analyzed all the data using the R version 4.1.3 programming language and environment (http://www.r-project.org/). The patients were divided into a training group and a testing group at a ratio of 7:3. Totally 233 GCLM patients were considered as an external validation group, which included 126 patients from The Shijiazhuang People’s Hospital, 67 patients from The Jinan City People’s Hospital, and 40 patients from The Sixth People’s Hospital of Huizhou. These cases were retrospectively collected from 2017 to 2018. The exclusion criteria include: (I) patients with metastases outside of liver; (II) lacking positive histology; (III) patients with one primary tumor only and were not the first time to get a tumor; (IV) lacking survival months and CSS; (V) lacking complete clinical information; (VI) lacking treatment information. The filtering process is shown in Figure 1.
Figure 1.
The filtering process of external validation cohort.
For statistical analysis, we first used the chi-square test to examine the distribution characteristics of categorical variables. Next, we employed Least Absolute Shrinkage and Selection Operator (LASSO), Univariate Cox regression, and Best Subset Regression (BSR) to identify the risk features. Before constructing the models, we calculated the minimum Akaike Information Criterion (AIC) values through stepwise backward regression to determine the optimal feature combination [17]. Finally, we used these features to develop ML models, including Random Survival Forest (RSF) [18], Gradient Boosting Machine (GBM), Cox Proportional Hazards (CPH), Survival Support Vector Machine (survivalSVM), and eXtreme Gradient Boosting (XGBoost) [19], to predict OS and CSS in GCLM patients. Our models were iteratively tested and tuned, and parameters were optimized to obtain the best performance. This study was approved by Shijiazhuang People’s Hospital, and informed consent was obtained from patients for relevant data analysis. The study was conducted in accordance with the ethical standards set forth in the 1964 Declaration of Helsinki and its later amendments.
Model validation and visualization
We compared the accuracy of the 5 models by using the c-index and the areas under the receiver operating characteristic (ROC) curves (AUCs) [20]. The Brier scores were used to compare the deviation between the model’s predicted value and the actual value [21]. Based on this, we chose the best models to predict the OS and CSS, respectively. After that, we used decision curve analyses (DCAs) [22] to assess the clinical utility value. We also verified the accuracy and sensitivity of the model on the testing set by using the c-index, AUC and DCA. Finally, we exhibited the feature importance value by using the SHAP (Shapley additive explanations) [23] and constructed a websites calculator to demonstrate the models for predicting OS and CSS, respectively. All processes are shown in Figure 2.
Figure 2.
Flowchart of developing the machine learning models for GCLM patients. GCLM: gastric cancer with liver metastasis.
Results
Characteristics of GCLM patients
According to the inclusion and exclusion criteria, a total of 1300 GCLM patients were included in our study. Table 1 shows the demographic and clinicopathological features of all the study cohort. There were more male patients (914 [70.31%]) than female ones (386 [29.69%]). Up to 39.31% of patients had lesions in the cardia, but only 1.69% had lesions in the pylorus. The majority of GCLM patients were Grade III stage (624 [48.00%]), suffered adenocarcinoma (896 [68.92%]), and had tumors larger than 2 cm (1208 [92.92%]).
Table 1.
Baseline of the demographic and related clinical characteristics in GCLM patients [n, (%)]
| Subject characteristics | GCLM patients (2010-2017) | P value | ||
|---|---|---|---|---|
|
| ||||
| All patients (N=1300) | Training group (N=910) | Testing group (N=390) | ||
| Age (%) | 0.85 | |||
| <50 | 182 (14.00) | 125 (13.74) | 57 (14.62) | |
| 50-59 | 281 (21.62) | 202 (22.20) | 79 (20.26) | |
| 60-69 | 381 (29.31) | 270 (29.67) | 111 (28.46) | |
| 70-79 | 306 (23.54) | 208 (22.86) | 98 (25.13) | |
| ≥80 | 150 (11.54) | 105 (11.54) | 45 (11.54) | |
| Sex (%) | 0.25 | |||
| Male | 914 (70.31) | 649 (71.32) | 265 (67.95) | |
| Female | 386 (29.69) | 261 (28.68) | 125 (32.05) | |
| Race (%) | 0.10 | |||
| White | 890 (68.46) | 613 (67.36) | 277 (71.03) | |
| Black | 227 (17.46) | 155 (17.03) | 72 (18.46) | |
| Asian or Pacific Islander | 169 (13.00) | 132 (14.51) | 37 (9.49) | |
| American Indian/Alaska Native | 14 (1.08) | 10 (1.10) | 4 (1.03) | |
| Marital (%) | 0.99 | |||
| Married | 777 (59.77) | 544 (59.78) | 233 (59.74) | |
| Unmarried | 482 (37.08) | 337 (37.03) | 145 (37.18) | |
| Unknown | 41 (3.15) | 29 (3.19) | 12 (3.08) | |
| Primary site (%) | 0.68 | |||
| Cardia | 511 (39.31) | 346 (38.02) | 165 (42.31) | |
| Fundus | 80 (6.15) | 54 (5.93) | 26 (6.67) | |
| Body | 121 (9.31) | 86 (9.45) | 35 (8.97) | |
| Antrum | 205 (15.77) | 151 (16.59) | 54 (13.85) | |
| Pylorus | 22 (1.69) | 14 (1.54) | 8 (2.05) | |
| Lesser | 91 (7.00) | 69 (7.58) | 22 (5.64) | |
| Greater | 56 (4.31) | 39 (4.29) | 17 (4.36) | |
| Overlapping/NOS | 214 (16.46) | 151 (16.59) | 63 (16.15) | |
| Grade (%) | 0.23 | |||
| I | 59 (4.54) | 43 (4.73) | 16 (4.10) | |
| II | 304 (23.38) | 228 (25.05) | 76 (19.49) | |
| III | 624 (48.00) | 427 (46.92) | 197 (50.51) | |
| IV | 48 (3.69) | 31 (3.41) | 17 (4.36) | |
| Unknown | 265 (20.38) | 181 (19.89) | 84 (21.54) | |
| Histology (%) | 0.83 | |||
| Adenocarcinoma | 896 (68.92) | 634 (69.67) | 262 (67.18) | |
| Signet ring cell carcinoma | 69 (5.31) | 48 (5.27) | 21 (5.38) | |
| Mucinous adenocarcinoma | 13 (1.00) | 9 (0.99) | 4 (1.03) | |
| Others | 322 (24.77) | 219 (24.07) | 103 (26.41) | |
| T stage (%) | 0.97 | |||
| T1 | 341 (26.23) | 241 (26.48) | 100 (25.64) | |
| T2 | 103 (7.92) | 72 (7.91) | 31 (7.95) | |
| T3 | 392 (30.15) | 276 (30.33) | 116 (29.74) | |
| T4 | 464 (35.69) | 321 (35.27) | 143 (36.67) | |
| N stage (%) | 0.57 | |||
| N0 | 507 (39.00) | 352 (38.68) | 155 (39.74) | |
| N1 | 532 (40.92) | 367 (40.33) | 165 (42.31) | |
| N2 | 128 (9.85) | 96 (10.55) | 32 (8.21) | |
| N3 | 133 (10.23) | 95 (10.44) | 38 (9.74) | |
| Surgery (%) | 0.39 | |||
| No surgery of primary site | 978 (75.23) | 678 (74.51) | 300 (76.92) | |
| Surgery performed | 322 (24.77) | 232 (25.49) | 90 (23.08) | |
| Radiation (%) | 0.57 | |||
| None/Unknown | 1067 (82.08) | 751 (82.53) | 316 (81.03) | |
| Yes | 233 (17.92) | 159 (17.47) | 74 (18.97) | |
| Chemotherapy (%) | 0.77 | |||
| No/Unknown | 381 (29.31) | 264 (29.01) | 117 (30.00) | |
| Yes | 919 (70.69) | 646 (70.99) | 273 (70.00) | |
| Tumor size (%) | 0.11 | |||
| ≤2 cm | 92 (7.08) | 71 (7.80) | 21 (5.38) | |
| 2.1-5 cm | 547 (42.08) | 369 (40.55) | 178 (45.64) | |
| >5 cm | 661 (50.85) | 470 (51.65) | 191 (48.97) | |
GCLM: gastric cancer with liver metastasis.
Features’ selection for predicting OS and CSS
For OS, 10 features were identified using Univariate Cox Regression analysis, including age, race, primary site, grade, histology, T stage, N stage, surgery, radiation, and chemotherapy (Figure 3A). The Best Subset Regression (BSR) found 6 features with the maximum R2: grade, histology, T stage, N stage, surgery, and chemotherapy (Figure 3E). The LASSO regression analysis selected 5 features: age, histology, N stage, surgery, and chemotherapy (Figure 3C, 3F). Stepwise backward regression revealed that a model including grade, histology, T stage, N stage, surgery, and chemotherapy had the lowest Akaike Information Criterion (AIC) value of 9316.46. Multivariate Cox regression showed that grade IV (HR=1.925, 95% CI=1.102-3.362, P=0.021), T3 stage (HR=0.711, 95% CI=0.581-0.871, P<0.001), N3 stage (HR=1.447, 95% CI=1.110-1.890, P=0.006), surgery (HR=0.424, 95% CI=0.347-0.517, P<0.001), and chemotherapy (HR=0.401, 95% CI=0.340-0.473, P<0.001) were significantly associated with OS (Table 2). The forest plot indicated that each risk factor was independently related (Figure 3B). Heat maps highlighted the key values of risk features (Figure 3D, 3G). Therefore, the 6 features with the smallest AIC were selected to construct predictive models for OS.
Figure 3.
Baseline characteristics and risk features identification for OS. Univariate Cox regression forest plot (A) and multivariate Cox regression forest plot in GCLM patients. (B) Partial-likelihood deviance curve for feature selection (C), and Best subset regression (E), and LASSO coefficient profiles of the 13 variables in the training set. (F) Risk factor association plots for the training set (D) and testing set (G), respectively. GCLM: gastric cancer with liver metastasis; OS: overall survival; LASSO: Least Absolute Shrinkage and Selection Operator.
Table 2.
Univariate and multivariate cox regression analyses of overall survival (OS) and cancer-specific survival (CSS) in GCLM patients
| Subject characteristics | OS | CSS | ||||||
|---|---|---|---|---|---|---|---|---|
|
|
|
|||||||
| Univariate | Multivariate | Univariate | Multivariate | |||||
|
|
|
|
|
|||||
| HR (95% CI) | P value | HR (95% CI) | P value | HR (95% CI) | P value | HR (95% CI) | P value | |
| Age (%) | ||||||||
| <50 | 1.00 (Reference) | 1.000 | 1.00 (Reference) | 1.000 | 1.00 (Reference) | 1.000 | ||
| 50-59 | 1.167 | 0.219 | 1.146 | 0.291 | 1.203 | 0.146 | ||
| 60-69 | 1.303 | 0.026 | 1.218 | 0.106 | 1.293 | 0.034 | ||
| 70-79 | 1.428 | 0.004 | 1.299 | 0.044 | 1.401 | 0.008 | ||
| ≥80 | 1.877 | <0.001 | 1.312 | 0.073 | 1.749 | <0.001 | ||
| Sex (%) | ||||||||
| Male | 1.00 (Reference) | 1.000 | 1.00 (Reference) | 1.000 | ||||
| Female | 0.908 | 0.223 | 0.919 | 0.292 | ||||
| Race (%) | ||||||||
| White | 1.00 (Reference) | 1.000 | 1.00 (Reference) | 1.000 | ||||
| Black | 0.782 | 0.012 | 0.757 | 0.006 | ||||
| Asian or Pacific Islander | 0.957 | 0.672 | 0.998 | 0.984 | ||||
| American Indian/Alaska Native | 1.152 | 0.674 | 1.197 | 0.593 | ||||
| Marital (%) | ||||||||
| Married | 1.00 (Reference) | 1.000 | 1.00 (Reference) | 1.000 | ||||
| Unmarried | 1.012 | 0.872 | 1.011 | 0.881 | ||||
| Unknown | 1.171 | 0.426 | 1.165 | 0.448 | ||||
| Primary site (%) | ||||||||
| Cardia | 1.00 (Reference) | 1.000 | 1.00 (Reference) | 1.000 | ||||
| Fundus | 0.449 | <0.001 | 0.433 | <0.001 | ||||
| Body | 0.67 | 0.003 | 0.679 | 0.004 | ||||
| Antrum | 0.897 | 0.297 | 0.916 | 0.404 | ||||
| Pylorus | 0.818 | 0.478 | 0.853 | 0.574 | ||||
| Lesser | 0.842 | 0.213 | 0.831 | 0.191 | ||||
| Greater | 0.812 | 0.249 | 0.819 | 0.277 | ||||
| Overlapping/NOS | 0.759 | 0.009 | 0.752 | 0.008 | ||||
| Grade (%) | ||||||||
| I | 1.00 (Reference) | 1.000 | 1.00 (Reference) | 1.000 | 1.00 (Reference) | 1.000 | 1.00 (Reference) | 1.000 |
| II | 2.068 | <0.001 | 1.698 | 0.014 | 2.151 | <0.001 | 1.766 | 0.011 |
| III | 2.888 | <0.001 | 2.335 | <0.001 | 3.012 | <0.001 | 2.423 | <0.001 |
| IV | 1.42 | 0.212 | 1.925 | 0.021 | 1.548 | 0.126 | 2.107 | 0.01 |
| Unknown | 1.393 | 0.115 | 1.495 | 0.062 | 1.44 | 0.095 | 1.533 | 0.055 |
| Histology (%) | ||||||||
| Adenocarcinoma | 1.00 (Reference) | 1.000 | 1.00 (Reference) | 1.000 | 1.00 (Reference) | 1.000 | 1.00 (Reference) | 1.000 |
| Signet ring cell carcinoma | 1.28 | 0.103 | 1.243 | 0.165 | 1.331 | 0.06 | 1.288 | 0.108 |
| Mucinous adenocarcinoma | 1.817 | 0.076 | 1.808 | 0.082 | 1.892 | 0.058 | 1.906 | 0.059 |
| Others | 0.397 | <0.001 | 0.532 | <0.001 | 0.394 | <0.001 | 0.528 | <0.001 |
| T stage (%) | ||||||||
| T1 | 1.00 (Reference) | 1.000 | 1.00 (Reference) | 1.000 | 1.00 (Reference) | 1.000 | 1.00 (Reference) | 1.000 |
| T2 | 0.463 | <0.001 | 0.686 | 0.011 | 0.485 | <0.001 | 0.723 | 0.032 |
| T3 | 0.523 | <0.001 | 0.712 | <0.001 | 0.549 | <0.001 | 0.752 | 0.007 |
| T4 | 0.667 | <0.001 | 1.039 | 0.687 | 0.686 | <0.001 | 1.075 | 0.125 |
| N stage (%) | ||||||||
| N0 | 1.00 (Reference) | 1.000 | 1.00 (Reference) | 1.000 | 1.00 (Reference) | 1.000 | 1.00 (Reference) | 1.000 |
| N1 | 1.484 | <0.001 | 1.179 | 0.050 | 1.511 | <0.001 | 1.195 | 0.038 |
| N2 | 1.453 | 0.002 | 1.201 | 0.168 | 1.477 | 0.002 | 1.208 | 0.161 |
| N3 | 1.462 | 0.002 | 1.447 | 0.006 | 1.519 | 0.001 | 1.496 | 0.003 |
| Surgery (%) | ||||||||
| No surgery of primary site | 1.00 (Reference) | 1.000 | 1.00 (Reference) | 1.000 | 1.00 (Reference) | 1.000 | 1.00 (Reference) | 1.000 |
| Surgery performed | 0.503 | <0.001 | 0.424 | <0.001 | 0.496 | <0.001 | 0.407 | <0.001 |
| Radiation (%) | ||||||||
| None/Unknown | 1.00 (Reference) | 1.000 | 1.00 (Reference) | 1.000 | ||||
| Yes | 1.277 | 0.007 | 1.259 | 0.013 | ||||
| Chemotherapy (%) | ||||||||
| No/Unknown | 1.00 (Reference) | 1.000 | 1.00 (Reference) | 1.000 | 1.00 (Reference) | 1.000 | 1.00 (Reference) | 1.000 |
| Yes | 0.579 | <0.001 | 0.401 | <0.001 | 0.575 | <0.001 | 0.393 | <0.001 |
| Tumor size (%) | ||||||||
| ≤2 cm | 1.00 (Reference) | 1.000 | 1.00 (Reference) | 1.000 | ||||
| 2.1-5 cm | 1.057 | 0.683 | 1.113 | 0.449 | ||||
| >5 cm | 0.819 | 0.14 | 0.859 | 0.278 | ||||
For CSS, risk features identified by univariate Cox regression, BSR, LASSO, and stepwise backward regression included grade, histology, T stage, N stage, surgery, and chemotherapy, yielding the lowest AIC value of 8977.337 (Figure 4A-C, 4E, 4F). Multivariate Cox regression demonstrated that grade IV (HR=2.107, 95% CI=1.194-3.718, P=0.01), T3 stage (HR=0.752, 95% CI=0.612-0.924, P=0.007), N3 stage (HR=1.496, 95% CI=1.144-1.956, P=0.003), surgery (HR=0.407, 95% CI=0.332-0.499, P<0.001), and chemotherapy (HR=0.393, 95% CI=0.332-0.464, P<0.001) were significantly associated with CSS (Table 2). The forest plot indicated that each risk factor was independently related (Figure 4B). Heat maps highlighted the key values of risk features (Figure 4D, 4G). Consequently, the 6 features with the smallest AIC were selected to construct predictive models for CSS.
Figure 4.
Baseline characteristics and risk features identification for CSS. Univariate Cox regression forest plot (A) and multivariate Cox regression forest plot in GCLM patients. (B) Partial-likelihood deviance curve for feature selection (C), and Best subset regression (E), and LASSO coefficient profiles of the 13 variables in the training set. (F) Risk factor association plots for the training set (D) and testing set (G), respectively. GCLM: gastric cancer with liver metastasis; CSS: cancer specific survival; LASSO: Least Absolute Shrinkage and Selection Operator.
Model selection according to their performance for predicting OS and CSS
After confirming the risk features, we fed these features into 5 different models, including RSF, GBM, CPH, survivalSVM, and XGBoost, and identified the best model to predict the OS. The AUC of all the 5 models were greater than 0.7 at 1, 3 and 5 years (Figure 5A-F). The AUCs of the XGBoost model were 0.81, 0.89 and 0.09 at the three time points, respectively, and were all higher than those of the other four models in the training set. This pattern was also confirmed in the testing set, with AUC values of 0.78, 0.87 and 0.87 at 1, 3 and 5 years, respectively. The brier score could reflect the calibration of the prediction results of the model, and the closer the score was to 0, the more accurate the prediction was. The XGBoost demonstrated brier scores of 0.169, 0.080, and 0.061 at 1, 3 and 5 years, respectively, which were the lowest among the models (Table 3). Meanwhile, the XGBoost had the highest c-index value of 0.752, which illustrates the accuracy of a model (Table 3). Therefore, we selected the XGBoost as the most accurate model to predict the OS. After that, we performed the DCA to illustrated the degree of clinical benefit and found that almost all the blue lines were above the black lines, suggesting that the XGBoost model had satisfactory utility in predicting OS probability over 1, 3, and 5 years for both the training set and the testing set (Figure 5G-L). Furthermore, we calculated the cutoff value (0.35) using the risk score of each patient and divided the patients into a high-risk group and a low-risk group. The KM survival curves showed that the OS of the high-risk group was much shorter than that of the low-risk group in both the training and testing sets (Figure 5M, 5N).
Figure 5.
Model selection and validation in the training set. Receiver operating characteristic curves of all models regarding 1-year, 3-year and 5-year OS in the training set (A, C, E) and testing set (B, D, F). Decision curve analysis of the best model for the training set (G, I, K) and testing set (H, J, L) regarding 1-year, 3-year and 5-year OS. Kaplan-Meier survival curves for the training set (M) and testing set (N). OS: overall survival.
Table 3.
The models’ performance in the training set
| Model | Overall Survival (OS) | Cancer-specific survival (CSS) | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
|
|
|||||||||||||
| AUC | Brier Score | C-index | AUC | Brier Score | C-index | |||||||||
|
|
|
|
|
|||||||||||
| 1-Year | 3-Year | 5-Year | 1-Year | 3-Year | 5-Year | 1-Year | 3-Year | 5-Year | 1-Year | 3-Year | 5-Year | |||
| RSF | 0.786 | 0.889 | 0.887 | 0.177 | 0.085 | 0.065 | 0.705 | 0.794 | 0.895 | 0.887 | 0.176 | 0.087 | 0.068 | 0.709 |
| GBM | 0.791 | 0.860 | 0.851 | 0.175 | 0.088 | 0.065 | 0.716 | 0.790 | 0.861 | 0.854 | 0.176 | 0.090 | 0.067 | 0.717 |
| CPH | 0.759 | 0.833 | 0.838 | 0.187 | 0.096 | 0.070 | 0.695 | 0.757 | 0.831 | 0.836 | 0.189 | 0.101 | 0.073 | 0.696 |
| Survivalsvm | 0.727 | 0.818 | 0.848 | 0.196 | 0.100 | 0.070 | 0.675 | 0.726 | 0.815 | 0.841 | 0.199 | 0.106 | 0.075 | 0.667 |
| XGBoost | 0.809 | 0.887 | 0.891 | 0.169 | 0.080 | 0.061 | 0.752 | 0.814 | 0.898 | 0.895 | 0.169 | 0.084 | 0.064 | 0.764 |
GCLM: gastric cancer with liver metastasis.
Consistent with the OS, the XGBoost showed the highest the c-index (0.764) for predicting CSS in GCLM patients, with best accuracy (AUC: 0.81, 0.90 and 0.90 at 1, 3 and 5 years respectively) and reliability (Brier score: 0.17, 0.08 and 0.06 at 1, 3 and 5 years respectively; Figure 6A-L; Table 3). Therefore, XGBoost was chosen to be the best model to predict the CSS as well. The DCA also showed that the net benefits happened across almost a range 0.1-1.0 of threshold probabilities at 1, 3 and 5 years, which demonstrated the clinical utility of the XGBoost (Figure 6G-L). For CSS, the cutoff value for high-risk and low-risk groups was 0.57. The KM survival curves showed that the CSS of the high-risk group was much lower than that of the low-risk group in both the training and testing sets (Figure 6M, 6N).
Figure 6.
Model selection and validation in the testing set. Receiver operating characteristic curves of all models for the training set (A, C, E) and testing set (B, D, F) regarding 1-year, 3-year and 5-year CSS. Decision curve analysis of the best model for the training set (G, I, K) and testing set (H, J, L) regarding 1-year, 3-year and 5-year CSS. Kaplan-Meier survival curves for the training (M) and testing set (N). CSS: cancer specific survival.
Model validation and visualization
We used 233 GCLM patients from three hospitals (The Shijiazhuang People’s Hospital, Jinan City People’s Hospital, and The Sixth People’s Hospital of Huizhou) as an external validation set to verify the model’s practicability. The AUC values for OS were 0.80, 0.75, and 0.78 at 1, 3, and 5 years, respectively (Figure 7A). For CSS, the AUC values were 0.81, 0.75, and 0.80 at 1, 3, and 5 years, respectively (Figure 7B). The c-index was 0.707 for OS and 0.705 for CSS. DCA curves indicated consistent clinical predictive value with the training and testing sets for both OS and CSS (Figure 7C-H). Patients were divided into high-risk and low-risk groups based on previous cutoff values, revealing significant survival differences between the two groups (log-rank P<0.0001), with the high-risk group experiencing poorer OS and CSS (Figure 7I, 7J).
Figure 7.
Model validation in the external set. Receiver operating characteristic curves for the OS (A) and CSS (B) at 1, 3 and 5 years. Decision curve analysis for the OS (C-E) and CSS (F-H) at 1, 3 and 5 years. Kaplan-Meier survival curves for the OS (I) and CSS (J). OS: overall survival; CSS: cancer specific survival.
Discussion
This study successfully identified the risk factors affecting OS and CSS in GCLM patients using the SEER database. We developed novel ML models for OS and CSS based on these risk factors and validated the accuracy of the models using clinical data from 3 centers. We also created a web-based tool to help clinicians formulate clinical management plans easily and efficiently.
Previous studies have shown that prognostic risk factors for patients with gastric cancer include age, pathological type, surgery, and chemotherapy. In our study, we screened as many potential factors as possible and found that for GCLM patients, the risk factors affecting OS and CSS were the same: grade, histology, T stage, N stage, surgery, and chemotherapy.
The choice of treatment for GCLM patients has always been controversial. Chemotherapy, particularly neoadjuvant therapy, is currently the main treatment option. Existing chemotherapy regimens include ramucirumab plus paclitaxel [24], epirubicin + oxaliplatin + capecitabine [25], S-1 + cisplatin [26], cisplatin + 5-fluorouracil [27], docetaxel/irinotecan [28], ramucirumab [29], docetaxel [30], and ramucirumab + paclitaxel [31], with median OS times reported to be 9.2 years, 11.2 years, 13 years, 11.3 years, 5.3 years, 5.2 years, 5.2 years, and 9.6 years, respectively. Previous studies have proven that patients who received chemotherapy had longer median survival times than those who did not [6-8,32]. Our study results are consistent with these findings, showing a median survival time of 11 months for patients who received chemotherapy and 3 months for those who did not. Therefore, chemotherapy can significantly prolong survival time for GCLM patients.
Currently, there is debate over the necessity of surgery for gastric cancer patients with metastasis. Some studies have found that palliative gastrectomy can increase survival rates, especially in individuals younger than 70 years old with a single metastatic site [33-35]. A meta-analysis by Sun et al. reported significantly longer survival time for metastatic gastric cancer patients who underwent palliative surgery compared to those who did not [36]. However, the National Comprehensive Cancer Network and three European guidelines do not recommend palliative gastrectomy [37,38]. In our study, GCLM patients who underwent surgery for the primary site had a lower risk of death compared to those who did not. This suggests that combining surgery with chemotherapy may significantly improve patient survival. Additionally, we found that patients with N3 stage and Grade III tumors have the highest risk of death. Moreover, compared to adenocarcinoma, patients with certain pathological types, other than signet ring cell carcinoma and mucinous adenocarcinoma, such as non-small cell carcinoma and stromal tumors, had poorer prognoses.
Therefore, we constructed 5 ML models using these features, and found that XGBoost had the best predictive ability. To our knowledge, there were no previous study applied ML models to evaluate the prognosis of GCLM patients. Compared with Dong et al.’s [33] nomogram, the AUC of our model was nearly 0.9. Meanwhile, brier score and DCA also showed higher accuracy and clinical predictive value of the model. We believed the model could provide more accurate OS. This is not only conducive to the doctor-patient relationship, but also helpful for formulating the optimal treatment plan for patients. Therefore, we believe that the web calculator (https://hbwszhaoqun.shinyapps.io/xgboost-model/) can provide greater convenience for the users.
Although we successfully identified the risk factors affecting patient prognosis and constructed two models with satisfied performance to predict the OS and CSS of GCLM patients, our models also have deficiencies. First of all, the data in the SEER database, including chemotherapy method and course of treatment, as well as surgical methods, were incomplete, which may lead to the deviation of prediction. Secondly, the database did not provide hematology-related indicators, including leukocytes, neutrophils and other information. If we added this information to construct the models, our prediction model may perform better. In addition, there are geographical limitations to external validation of our model, and our model has not been verified by large samples, so a multicenter, large-sample prospective study is needed to verify the accuracy of the model.
Conclusion
Using different algorithms, we identified the risk factors affecting patient prognosis and successfully constructed two models to predict OS and CSS. Compared to existing models, our models demonstrated superior predictive performance. We showcased their potential in helping clinicians and patients predict survival time and choose optimal treatment plans.
Acknowledgements
The authors thank SEER for its research design and data sharing, as well as all investigators and participants.
Written informed consent was obtained from all participants.
Disclosure of conflict of interest
None.
References
- 1.Smyth EC, Nilsson M, Grabsch HI, van Grieken NC, Lordick F. Gastric cancer. Lancet. 2020;396:635–648. doi: 10.1016/S0140-6736(20)31288-5. [DOI] [PubMed] [Google Scholar]
- 2.Cui JK, Liu M, Shang XK. Hepatectomy for liver metastasis of gastric cancer: a meta-analysis. Surg Innov. 2019;26:692–697. doi: 10.1177/1553350619856491. [DOI] [PubMed] [Google Scholar]
- 3.Guan WL, He Y, Xu RH. Gastric cancer treatment: recent progress and future perspectives. J Hematol Oncol. 2023;16:57. doi: 10.1186/s13045-023-01451-3. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Zhang Y, Lin Y, Duan J, Xu K, Mao M, Wang X. A population-based analysis of distant metastasis in stage IV gastric cancer. Med Sci Monit. 2020;26:e923867. doi: 10.12659/MSM.923867. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Qiu MZ, Shi SM, Chen ZH, Yu HE, Sheng H, Jin Y, Wang DS, Wang FH, Li YH, Xie D, Zhou ZW, Yang DJ, Xu RH. Frequency and clinicopathological features of metastasis to liver, lung, bone, and brain from gastric cancer: a SEER-based study. Cancer Med. 2018;7:3662–3672. doi: 10.1002/cam4.1661. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Kurokawa Y, Doki Y, Kitabayashi R, Yoshikawa T, Nomura T, Tsuji K, Goto M, Cho H, Hihara J, Hiki N, Nunobe S, Mizusawa J, Boku N, Terashima M. Short-term outcomes of preoperative chemotherapy with docetaxel, oxaliplatin, and S-1 for gastric cancer with extensive lymph node metastasis (JCOG1704) Gastric Cancer. 2024;27:366–374. doi: 10.1007/s10120-023-01453-7. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Kakeji Y, Yoshida K, Kodera Y, Kochi M, Sano T, Ichikawa W, Lee SW, Shibahara K, Shikano T, Kataoka M, Ishiguro A, Ojima H, Sakai Y, Musha N, Takase T, Kimura T, Takeuchi M, Fujii M. Three-year outcomes of a randomized phase III trial comparing adjuvant chemotherapy with S-1 plus docetaxel versus S-1 alone in stage III gastric cancer: JACCRO GC-07. Gastric Cancer. 2022;25:188–196. doi: 10.1007/s10120-021-01224-2. [DOI] [PubMed] [Google Scholar]
- 8.Gupta J, Ahmed AT, Tayyib NA, Zabibah RS, Shomurodov Q, Kadheim MN, Alsaikhan F, Ramaiah P, Chinnasamy L, Samarghandian S. A state-of-art of underlying molecular mechanisms and pharmacological interventions/nanotherapeutics for cisplatin resistance in gastric cancer. Biomed Pharmacother. 2023;166:115337. doi: 10.1016/j.biopha.2023.115337. [DOI] [PubMed] [Google Scholar]
- 9.Solaini L, Ministrini S, Bencivenga M, D’Ignazio A, Marino E, Cipollari C, Molteni B, Mura G, Marrelli D, Graziosi L, Roviello F, De Manzoni G, Tiberio GAM, Morgagni P. Conversion gastrectomy for stage IV unresectable gastric cancer: a GIRCG retrospective cohort study. Gastric Cancer. 2019;22:1285–1293. doi: 10.1007/s10120-019-00968-2. [DOI] [PubMed] [Google Scholar]
- 10.Yu J, Huang C, Sun Y, Su X, Cao H, Hu J, Wang K, Suo J, Tao K, He X, Wei H, Ying M, Hu W, Du X, Hu Y, Liu H, Zheng C, Li P, Xie J, Liu F, Li Z, Zhao G, Yang K, Liu C, Li H, Chen P, Ji J, Li G Chinese Laparoscopic Gastrointestinal Surgery Study (CLASS) Group. Effect of laparoscopic vs open distal gastrectomy on 3-year disease-free survival in patients with locally advanced gastric cancer: the CLASS-01 randomized clinical trial. JAMA. 2019;321:1983–1992. doi: 10.1001/jama.2019.5359. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Terashima M, Fujitani K, Ando M, Sakamaki K, Kawabata R, Ito Y, Yoshikawa T, Kondo M, Kodera Y, Kaji M, Oka Y, Imamura H, Kawada J, Takagane A, Shimada H, Tanizawa Y, Yamanaka T, Morita S, Ninomiya M, Yoshida K. Survival analysis of a prospective multicenter observational study on surgical palliation among patients receiving treatment for malignant gastric outlet obstruction caused by incurable advanced gastric cancer. Gastric Cancer. 2021;24:224–231. doi: 10.1007/s10120-020-01114-z. [DOI] [PubMed] [Google Scholar]
- 12.Terashima M, Fujitani K, Ando M, Sakamaki K, Kawabata R, Ito Y, Yoshikawa T, Kondo M, Kodera Y, Kaji M, Oka Y, Imamura H, Kawada J, Takagane A, Shimada H, Tanizawa Y, Yamanaka T, Morita S, Ninomiya M, Yoshida K. Survival analysis of a prospective multicenter observational study on surgical palliation among patients receiving treatment for malignant gastric outlet obstruction caused by incurable advanced gastric cancer. Gastric Cancer. 2021;24:224–231. doi: 10.1007/s10120-020-01114-z. [DOI] [PubMed] [Google Scholar]
- 13.Zhuo W, Liu Y, Li S, Guo D, Sun Q, Jin J, Rao X, Li M, Sun M, Jiang M, Xu Y, Teng L, Jin Y, Si J, Liu W, Kang Y, Zhou T. Long noncoding RNA GMAN, up-regulated in gastric cancer tissues, is associated with metastasis in patients and promotes translation of ephrin A1 by competitively binding GMAN-AS. Gastroenterology. 2019;156:676–691. e11. doi: 10.1053/j.gastro.2018.10.054. [DOI] [PubMed] [Google Scholar]
- 14.Wei L, Huang Y, Chen Z, Li J, Huang G, Qin X, Cui L, Zhuo Y. A novel machine learning algorithm combined with multivariate analysis for the prognosis of renal collecting duct carcinoma. Front Oncol. 2022;11:777735. doi: 10.3389/fonc.2021.777735. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Lin J, Yin M, Liu L, Gao J, Yu C, Liu X, Xu C, Zhu J. The development of a prediction model based on random survival forest for the postoperative prognosis of pancreatic cancer: a SEER-based study. Cancers (Basel) 2022;14:4667. doi: 10.3390/cancers14194667. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Zhou SN, Jv DW, Meng XF, Zhang JJ, Liu C, Wu ZY, Hong N, Lu YY, Zhang N. Feasibility of machine learning-based modeling and prediction using multiple centers data to assess intrahepatic cholangiocarcinoma outcomes. Ann Med. 2023;55:215–223. doi: 10.1080/07853890.2022.2160008. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Li W, Wang G, Wu R, Dong S, Wang H, Xu C, Wang B, Li W, Hu Z, Chen Q, Yin C. Dynamic predictive models with visualized machine learning for assessing chondrosarcoma overall survival. Front Oncol. 2022;12:880305. doi: 10.3389/fonc.2022.880305. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Zeng J, Li K, Cao F, Zheng Y. The development of a prediction model based on deep learning for prognosis prediction of gastrointestinal stromal tumor: a SEER-based study. Sci Rep. 2024;14:6609. doi: 10.1038/s41598-024-56701-2. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Liu P, Fu B, Yang SX, Deng L, Zhong X, Zheng H. Optimizing survival analysis of XGBoost for ties to predict disease progression of breast cancer. IEEE Trans Biomed Eng. 2021;68:148–160. doi: 10.1109/TBME.2020.2993278. [DOI] [PubMed] [Google Scholar]
- 20.Janssens ACJW, Martens FK. Reflection on modern methods: revisiting the area under the ROC curve. Int J Epidemiol. 2020;49:1397–1403. doi: 10.1093/ije/dyz274. [DOI] [PubMed] [Google Scholar]
- 21.Lee C, Light A, Alaa A, Thurtle D, van der Schaar M, Gnanapragasam VJ. Application of a novel machine learning framework for predicting non-metastatic prostate cancer-specific mortality in men using the Surveillance, Epidemiology, and End Results (SEER) database. Lancet Digit Health. 2021;3:e158–e165. doi: 10.1016/S2589-7500(20)30314-9. [DOI] [PubMed] [Google Scholar]
- 22.Chen B, Zeng Y, Liu B, Lu G, Xiang Z, Chen J, Yu Y, Zuo Z, Lin Y, Ma J. Risk factors, prognostic factors, and nomograms for distant metastasis in patients with newly diagnosed osteosarcoma: a population-based study. Front Endocrinol (Lausanne) 2021;12:672024. doi: 10.3389/fendo.2021.672024. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Zhou SN, Jv DW, Meng XF, Zhang JJ, Liu C, Wu ZY, Hong N, Lu YY, Zhang N. Feasibility of machine learning-based modeling and prediction using multiple centers data to assess intrahepatic cholangiocarcinoma outcomes. Ann Med. 2023;55:215–223. doi: 10.1080/07853890.2022.2160008. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Lordick F, Nilsson M, Leong T. Adjuvant radiotherapy for gastric cancer-end of the road? Ann Oncol. 2021;32:287–289. doi: 10.1016/j.annonc.2020.12.006. [DOI] [PubMed] [Google Scholar]
- 25.Shitara K, Rha SY, Wyrwicz LS, Oshima T, Karaseva N, Osipov M, Yasui H, Yabusaki H, Afanasyev S, Park YK, Al-Batran SE, Yoshikawa T, Yanez P, Dib Bartolomeo M, Lonardi S, Tabernero J, Van Cutsem E, Janjigian YY, Oh DY, Xu J, Fang X, Shih CS, Bhagia P, Bang YJ KEYNOTE-585 investigators. Neoadjuvant and adjuvant pembrolizumab plus chemotherapy in locally advanced gastric or gastro-oesophageal cancer (KEYNOTE-585): an interim analysis of the multicentre, double-blind, randomised phase 3 study. Lancet Oncol. 2024;25:212–224. doi: 10.1016/S1470-2045(23)00541-7. [DOI] [PubMed] [Google Scholar]
- 26.Yamada Y, Boku N, Mizusawa J, Iwasa S, Kadowaki S, Nakayama N, Azuma M, Sakamoto T, Shitara K, Tamura T, Chin K, Hata H, Nakamori M, Hara H, Yasui H, Katayama H, Fukuda H, Yoshikawa T, Sasako M, Terashima M. Docetaxel plus cisplatin and S-1 versus cisplatin and S-1 in patients with advanced gastric cancer (JCOG1013): an open-label, phase 3, randomised controlled trial. Lancet Gastroenterol Hepatol. 2019;4:501–510. doi: 10.1016/S2468-1253(19)30083-4. [DOI] [PubMed] [Google Scholar]
- 27.Janjigian YY, Kawazoe A, Bai Y, Xu J, Lonardi S, Metges JP, Yanez P, Wyrwicz LS, Shen L, Ostapenko Y, Bilici M, Chung HC, Shitara K, Qin SK, Van Cutsem E, Tabernero J, Li K, Shih CS, Bhagia P, Rha SY KEYNOTE-811 Investigators. Pembrolizumab plus trastuzumab and chemotherapy for HER2-positive gastric or gastro-oesophageal junction adenocarcinoma: interim analyses from the phase 3 KEYNOTE-811 randomised placebo-controlled trial. Lancet. 2023;402:2197–2208. doi: 10.1016/S0140-6736(23)02033-0. [DOI] [PubMed] [Google Scholar]
- 28.Shitara K, Iwata H, Takahashi S, Tamura K, Park H, Modi S, Tsurutani J, Kadowaki S, Yamaguchi K, Iwasa S, Saito K, Fujisaki Y, Sugihara M, Shahidi J, Doi T. Trastuzumab deruxtecan (DS-8201a) in patients with advanced HER2-positive gastric cancer: a dose-expansion, phase 1 study. Lancet Oncol. 2019;20:827–836. doi: 10.1016/S1470-2045(19)30088-9. [DOI] [PubMed] [Google Scholar]
- 29.Saeed A, Park R, Sun W. The integration of immune checkpoint inhibitors with VEGF targeted agents in advanced gastric and gastroesophageal adenocarcinoma: a review on the rationale and results of early phase trials. J Hematol Oncol. 2021;14:13. doi: 10.1186/s13045-021-01034-0. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30.Jain A. Perioperative chemotherapy for gastric cancer in FLOT4. Lancet. 2020;395:e2. doi: 10.1016/S0140-6736(19)32505-X. [DOI] [PubMed] [Google Scholar]
- 31.Kim CG, Jung M, Kim HS, Lee CK, Jeung HC, Koo DH, Bae WK, Zang DY, Kim BJ, Kim H, Yun UJ, Che J, Park S, Kim TS, Kwon WS, Park J, Cho SW, Nam CM, Chung HC, Rha SY. Trastuzumab combined with ramucirumab and paclitaxel in patients with previously treated human epidermal growth factor receptor 2-positive advanced gastric or gastroesophageal junction cancer. J. Clin. Oncol. 2023;41:4394–4405. doi: 10.1200/JCO.22.02122. [DOI] [PubMed] [Google Scholar]
- 32.Lin Z, Wang R, Zhou Y, Wang Q, Yang CY, Hao BC, Ke CF. Prediction of distant metastasis and survival prediction of gastric cancer patients with metastasis to the liver, lung, bone, and brain: research based on the SEER database. Ann Transl Med. 2022;10:16. doi: 10.21037/atm-21-6295. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33.Dong Z, Zhang Y, Geng H, Ni B, Xia X, Zhu C, Liu J, Zhang Z. Development and validation of two nomograms for predicting overall survival and cancer-specific survival in gastric cancer patients with liver metastases: a retrospective cohort study from SEER database. Transl Oncol. 2022;24:101480. doi: 10.1016/j.tranon.2022.101480. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 34.Muro K, Van Cutsem E, Narita Y, Pentheroudakis G, Baba E, Li J, Ryu MH, Zamaniah WIW, Yong WP, Yeh KH, Kato K, Lu Z, Cho BC, Nor IM, Ng M, Chen LT, Nakajima TE, Shitara K, Kawakami H, Tsushima T, Yoshino T, Lordick F, Martinelli E, Smyth EC, Arnold D, Minami H, Tabernero J, Douillard JY. Pan-Asian adapted ESMO Clinical Practice Guidelines for the management of patients with metastatic gastric cancer: a JSMO-ESMO initiative endorsed by CSCO, KSMO, MOS, SSO and TOS. Ann Oncol. 2019;30:19–33. doi: 10.1093/annonc/mdy502. [DOI] [PubMed] [Google Scholar]
- 35.Honda M, Yasunaga H, Michihata N, Miyakawa T, Kumazawa R, Matsui H, Imaizumi T. Impact of guideline recommendation for novel surgical procedures on surgeons’ decisions: a time series analysis of gastric cancer surgeries from a nationwide cohort study. Int J Surg. 2023;109:316–322. doi: 10.1097/JS9.0000000000000179. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 36.Joharatnam-Hogan N, Shiu KK, Khan K. Challenges in the treatment of gastric cancer in the older patient. Cancer Treat Rev. 2020;85:101980. doi: 10.1016/j.ctrv.2020.101980. [DOI] [PubMed] [Google Scholar]
- 37.Lordick F, Carneiro F, Cascinu S, Fleitas T, Haustermans K, Piessen G, Vogel A, Smyth EC ESMO Guidelines Committee. Electronic address: clinicalguidelines@esmo.org. Gastric cancer: ESMO Clinical Practice Guideline for diagnosis, treatment and follow-up. Ann Oncol. 2022;33:1005–1020. doi: 10.1016/j.annonc.2022.07.004. [DOI] [PubMed] [Google Scholar]
- 38.Shah MA, Kennedy EB, Alarcon-Rozas AE, Alcindor T, Bartley AN, Malowany AB, Bhadkamkar NA, Deighton DC, Janjigian Y, Karippot A, Khan U, King DA, Klute K, Lacy J, Lee JJ, Mehta R, Mukherjee S, Nagarajan A, Park H, Saeed A, Semrad TJ, Shitara K, Smyth E, Uboha NV, Vincelli M, Wainberg Z, Rajdev L. Immunotherapy and targeted therapy for advanced gastroesophageal cancer: ASCO guideline. J. Clin. Oncol. 2023;41:1470–1491. doi: 10.1200/JCO.22.02331. [DOI] [PubMed] [Google Scholar]