Machine Learning-based Prediction of Unplanned Acute Care in Outpatients Receiving S-1 Chemotherapy

Abstract

Background/Aim

The increasing use of oral anticancer agents in outpatient settings has led to a growing need for unplanned acute care (UAC) due to treatment-related adverse events. Early identification of high-risk patients is therefore clinically important. This study aimed to develop a machine learning-based model to predict UAC in outpatients receiving S-1 chemotherapy and to compare its performance with conventional logistic regression.

Patients and Methods

This retrospective single-center study included 579 outpatients who newly underwent S-1 therapy. UAC was defined as chemotherapy-related unplanned hospitalization or urgent outpatient visits during the first treatment course. Predictive models were developed using logistic regression and machine-learning algorithms, including a support vector machine (SVM). Model performance was evaluated using recall-oriented metrics, with the F2 score adopted as the primary performance measure. Shapley additive explanations (SHAP) were applied for feature selection and model interpretation.

Results

Among the 579 patients, 45 experienced UAC. In the independent test dataset (n=173), the SHAP-selected SVM model demonstrated superior performance compared with logistic regression, achieving higher recall (0.769 vs. 0.615) and F2 score (0.368 vs. 0.325).

Conclusion

An SVM-based machine-learning model improved the prediction of UAC among outpatients receiving S-1 chemotherapy by reducing false-negative predictions and may support early risk stratification to enhance the safety of outpatient chemotherapy.

Introduction

In recent years, advances in cancer pharmacotherapy have led to an increasing number of patients receiving oral anticancer agents in outpatient settings. Compared with intravenous chemotherapy, oral treatment reduces hospital visits and allows patients to continue therapy at home, thereby contributing to improved quality of life (1). However, a subset of patients experiences serious adverse reactions requiring unplanned acute care (UAC), such as unplanned hospitalization or urgent outpatient visits (2).

UAC imposes substantial physical and psychological burdens on patients and increases the workload of healthcare professionals, as well as healthcare resource utilization (3). Therefore, preventing UAC is a critical clinical issue in improving the safety of outpatient chemotherapy. Effective prevention requires not only appropriate management after the onset of an adverse event but also early identification of patients at high risk for UAC before or in the early phase of treatment initiation.

S-1, an oral fluoropyrimidine anticancer agent, is widely used in the treatment of gastrointestinal cancer and plays an important role in outpatient chemotherapy. However, S-1 is associated with adverse events, such as gastrointestinal toxicity, myelosuppression, and dehydration, which may lead to UAC (4). Accurate risk assessment and early identification of high-risk patients receiving S-1 are therefore clinically important.

Previously, we identified factors associated with UAC in patients receiving oral anticancer drugs using logistic regression analysis (5, 6). Although several studies have explored factors related to adverse events or UAC, studies focusing on individualized prediction of UAC using integrated clinical information remain limited.

Recently, machine-learning techniques have rapidly evolved in the medical field and have shown utility in diagnosis support, prognosis prediction, and treatment response estimation (7-10). Machine-learning methods can capture complex relationships and interactions among multiple variables (11). Leveraging routinely collected clinical data may therefore enable the development of more practical and accurate risk-based predictive models.

Accordingly, this study aimed to develop a machine learning-based predictive model for UAC in outpatients receiving S-1 chemotherapy and to compare its predictive performance with that of a conventional logistic regression model. Given the serious clinical consequences of missed UAC events, particular emphasis was placed on minimizing false-negative (FN) predictions to enhance clinical applicability.

Patients and Methods

Study design and patient population. This retrospective single-center study was conducted at the National Cancer Center Hospital East. Eligible patients were outpatients newly undergoing S-1 chemotherapy (tegafur/gimeracil/oteracil) between April 1, 2020 and March 31, 2021. Patients with hematological malignancies were excluded. In addition, patients with missing data for performance status (PS) were excluded because PS was considered an essential clinical variable for risk assessment.

Definition of UAC. UAC was defined as chemotherapy-related unplanned hospitalization or urgent outpatient visits occurring within the first treatment course after the initiation of S-1 therapy. UAC events were required to be attributable to serious adverse reactions associated with anticancer treatment.

The causal relationship between adverse events and chemotherapy was assessed according to institutional guidelines using a five-level scale (“unrelated”, “unlikely”, “possible”, “probable”, and “definite”). Events classified as “possible”, “probable” or “definite” were considered chemotherapy-related and were included as UAC events.

Candidate variables for model development. Candidates were restricted to variables for which data were routinely available before treatment initiation, reflecting information that might be used for risk stratification in real-world outpatient practice. These included age, sex, cancer type, treatment intent (progression/recurrence or curative setting), treatment line (first-line or second line or greater), body mass index, PS, concomitant use of injectable anticancer agents, and baseline hematological and biochemical laboratory values obtained prior to the first treatment course. A total of 41 baseline variables were considered as candidates.

Data preprocessing. Handling of missing values: Several laboratory variables had missing data, at a rate ranging from 0.17% to 13.99%. Because none of the laboratory variables followed a normal distribution and the proportion of missing values was generally low, missing values were imputed using median values to minimize the influence of outliers and preserve robustness.

Correlation analysis and multicollinearity: Spearman’s rank correlation coefficients were calculated to assess correlations among candidate variables. Strong correlations were defined as correlation coefficients greater than 0.8. For strongly correlated variable pairs, one variable was excluded based on clinical interpretability and generalizability to avoid multicollinearity.

Feature scaling: For algorithms sensitive to feature scale, including support vector machines (SVM), all continuous variables were standardized using Z-score normalization (mean of 0, standard deviation of 1).

Dataset splitting and validation strategy. The full dataset was divided into training (70%) and test (30%) datasets using stratified hold-out validation to ensure identical proportions of UAC events in both datasets. The training dataset was used for model development, cross-validation, and hyperparameter tuning, whereas the independent test dataset was reserved exclusively for final model evaluation.

Baseline characteristics between the training and test datasets were assessed using standardized mean differences, with values of below 0.1 indicating adequate balance.

Model development. Algorithms evaluated: A logistic regression model was used as the reference model. In addition, three machine-learning algorithms, namely adaptive boosting (AdaBoost), random under-sampling boosting (RUSBoost), and a SVM with a Gaussian kernel, were evaluated.

To address class imbalance and prioritize the detection of UAC events, higher misclassification costs were assigned to FN predictions in the AdaBoost and SVM models.

Cross-validation and hyperparameter tuning: Model optimization was performed using repeated cross-validation, consisting of two-fold cross-validation repeated five times. This approach was adopted to reduce variability associated with data partitioning and to obtain more stable performance estimates.

Hyperparameter tuning was conducted in two stages. Firstly, a grid search was used to define the initial search space. Subsequently, Bayesian optimization was applied to identify optimal hyperparameter combinations. Hyperparameter search ranges for each algorithm are summarized in Table I.

Table I. Hyperparameter search ranges and optimization strategies for each machine-learning algorithm. AdaBoost: Adaptive boosting; RUSBoost: random under-sampling boosting; SVM: Support vector machine.

Shapley additive explanations (SHAP)-based selection of variables: To reduce model complexity and mitigate overfitting, SHAP values were calculated for trained models (12). Features with low SHAP contributions were sequentially removed, and models were retrained at each step. Model performance was evaluated after each iteration, and the feature set yielding the best performance was selected.

In addition to feature reduction, SHAP analysis was used to interpret the contributions of variables to model predictions.

Evaluation of model performance. Model performance was evaluated using confusion matrices. Given the clinical importance of avoiding missed UAC events, recall-oriented metrics were prioritized. The F2 score, which places greater emphasis on recall than precision, was adopted as the primary performance metric (9, 13).

Precision, recall, and the F2 score were calculated as follows:

$$\text{Precision}=\frac{\text{TP}}{\text{TP}+\text{FP}}$$

$$\text{Recall}=\frac{\text{TP}}{\text{TP}+\text{FN}}$$

$${\text{F}}_2=\frac{(1+2^2)\times \text{Precision}\times \text{Recall}}{(2^2\times \text{Precision})+\text{Recall}}$$

where TP and FP denote true positives and false positives, respectively.

In addition to the F2 score, the accuracy, area under the receiver operating characteristics curve (ROC-AUC), and area under the precision-recall curve (PR-AUC) were calculated as secondary performance metrics.

The model demonstrating the highest F2 score during cross-validation was selected as the final model and was evaluated using the independent test dataset alongside the logistic regression model.

Software. Correlation analyses (Spearman’s rank) were performed using IBM SPSS Statistics version 23.0 (IBM Corp., Armonk, NY, USA). Data preprocessing, logistic regression, and machine-learning model development were conducted using MATLAB (versions 2024a-2026a; The MathWorks, Inc., Natick, MA, USA).

Ethical considerations. This study was conducted in accordance with the Declaration of Helsinki and was approved by the Institutional Review Boards of the National Cancer Center Hospital East (no. 2022-113) and Tokyo University of Science (No. 23034).

Results

Study population and dataset composition. A total of 579 outpatients newly undergoing S-1 chemotherapy met the eligibility criteria and were included in the analysis. Using stratified hold-out validation, 406 patients were assigned to the training dataset and 173 patients to the independent test dataset. UAC occurred in 32 patients in the training dataset and in 13 patients in the test dataset.

Baseline demographic and clinical characteristics were generally well balanced between the training and test datasets. Standardized mean differences for most variables were below 0.1, indicating adequate balance between datasets (Table II and Table III).

Table II. Baseline demographic and clinical characteristics of patients in the training and test datasets.

BMI: Body mass index; PS: performance status; SMD: standardized mean difference.

Table III. Baseline laboratory characteristics of patients in the training and test datasets.

ALP: Alkaline phosphatase; ALT: alanine aminotransferase; AST: aspartate aminotransferase; BUN: blood urea nitrogen; CRP: C-reactive protein; eGFR: estimated glomerular filtration rate; γ-GTP: γ-glutamyl transpeptidase; Hb: hemoglobin; LDH: lactate dehydrogenase; RBC: red blood cells; SMD: standardized mean difference; T-Bil: total bilirubin; WBC: white blood cells.

Feature correlation analysis and preprocessing. Spearman’s rank correlation analysis identified strong correlations (ρ>0.8) between several laboratory variables, including estimated glomerular filtration rate, creatinine, hemoglobin, and red blood cell, white blood cell and neutrophil counts. To mitigate multicollinearity while preserving clinical interpretability, creatinine, hemoglobin, and neutrophil count were retained, whereas the corresponding correlated variables were excluded.

After correlation-based preprocessing, a total of 38 features were retained for subsequent model development.

Model development and cross-validation performance. Predictive models were developed using logistic regression, AdaBoost, RUSBoost, and SVM algorithms, with performance evaluated for both full-feature and SHAP-selected models.

Logistic regression model: For the logistic regression model, candidate variables were screened using univariate analyses in the training dataset. Cancer type and several laboratory variables (alanine aminotransferase, aspartate amino-transferase, creatinine, hemoglobin, and platelet count) were selected and included in the multivariate model. In repeated cross-validation, the logistic regression model achieved a mean F2 score of 0.351 (Table IV and Table V).

Table IV. Univariate logistic regression analysis of patient background characteristics associated with unplanned acute care (UAC) during S-1 chemotherapy.

BMI: Body mass index; CI: confidence interval; OR: odds ratio; PS: performance status; SD: standard deviation. *A value of p<0.1 was used as the threshold for variable selection in the multivariate logistic regression model to reduce the risk of overfitting given the limited number of outcome events. Variables meeting this threshold are shown in bold.

Table V. Univariate logistic regression analysis of baseline laboratory variables associated with unplanned acute care (UAC) during S-1 chemotherapy.

ALP: Alkaline phosphatase; ALT: alanine aminotransferase; AST: aspartate aminotransferase; BUN: blood urea nitrogen; CRP: C-reactive protein; Hb: hemoglobin; LDH: lactate dehydrogenase; γ-GTP: γ-glutamyl transpeptidase; T-Bil: total bilirubin. Crude odds ratios for continuous variables indicate association with a 1-unit increase. *A value of p<0.1 was used as the threshold for variable selection in the multivariate logistic regression model to reduce the risk of overfitting given the limited number of outcome events. Variables meeting this threshold are shown in bold.

Machine-learning models: Among the machine-learning algorithms, the SVM model demonstrated the highest F2 score during repeated cross-validation. In addition, SHAP-based selection of variables improved the cross-validated F2 score of the SVM model compared with the full-feature SVM model, suggesting reduced model complexity and improved generalizability. Based on these results, the SHAP-based SVM model was chosen as the final candidate model for evaluation using the independent test dataset (Table VI). The variables selected by the SHAP-based process for each algorithm are shown in Figure 1.

Table VI. Comparison of cross-validated F2 scores across prediction models with full and Shapley additive explanation-selected feature sets.

AdaBoost: Adaptive boosting; RUSBoost: random under-sampling boosting; SVM: support vector machine.

Figure 1.

Shapley additive explanations (SHAP) summary plots showing the contributions of variables to predictions in the machine-learning models. The magnitude of the SHAP value represents the extent to which each factor contributes to the model’s prediction. Positive SHAP values indicate that the variable contributes to an increased probability of the target event, whereas negative SHAP values indicate that a factor reduces the probability of the target event. Variables marked with an asterisk (*) indicate features selected through SHAP-based feature selection for the final model. Please see Table II for the definition of terms.

Performance of the final model on the independent test dataset. The performance of the SHAP-selected SVM model and the logistic regression model was evaluated using the independent test dataset (n=173). The SVM model achieved a higher F2 score (0.368) and recall (0.769) than the logistic regression model (F2 score: 0.325; recall: 0.615), indicating improved detection of patients who experienced UAC.

In addition, the SVM model demonstrated superior discrimination performance, with higher PR-AUC (0.128 vs. 0.091) and ROC-AUC (0.670 vs. 0.587) values compared with logistic regression (Table VII, Figure 2). Confusion matrix analysis further showed that the SVM model produced fewer FN predictions, reflecting a reduced risk of missing UAC events (Figure 3).

Table VII. Performance of logistic regression and Shapley additive explanations (SHAP)-selected support vector machine (SVM) models in the independent test dataset.

PR-AUC: Precision-recall area under the curve; ROC-AUC: receiver operating characteristic area under the curve. Sensitivity corresponds to recall, and specificity corresponds to true negative rate.

Figure 2.

Receiver operating characteristic (ROC) curves and precision-recall (PR) curves for the developed predictive models. The panels show the ROC (left) and PR (right) curves for the logistic regression model (A) and the support vector machine (SVM) reduced-variable model (B). The ROC curves illustrate the trade-off between sensitivity and specificity, whereas the PR curves illustrate the relationship between precision and recall, which is particularly informative for imbalanced datasets. For analytical results, please see Table VII.

Figure 3.

Confusion matrices illustrating the performance of logistic regression and Shapley additive explanations (SHAP)-selected support vector machine (SVM) models in detecting unplanned acute care (UAC) in the independent test dataset. The panels show the structure of the confusion matrix (A), and the confusion matrices for the logistic regression model (B) and the SHAP-selected SVM model (C), evaluated using the independent test dataset. Each matrix displays the number of true-positive (TP), false-positive (FP), true-negative (TN), and false-negative (FN) results for UAC prediction. Compared with the logistic regression model, the SHAP-selected SVM model yielded fewer FN predictions, indicating improved detection of patients who experienced UAC.

Model interpretation using SHAP analysis. SHAP analysis was performed for each machine learning model to visualize feature contributions to model predictions (Figure 1).

In the final SVM model, treatment-related variables, including treatment setting and concomitant use of injectable anticancer agents, ranked among the top contributors. Cancer type and selected laboratory variables, such as platelet count and total protein, were also identified as influential features.

In the AdaBoost model, laboratory variables including platelet count, hemoglobin, total protein, aspartate aminotransferase, glucose, lactate dehydrogenase, electrolyte-related parameters, and C-reactive protein were consistently ranked among the top contributors, with treatment setting also included among the influential features.

In the RUSBoost model, creatinine, platelet count and body mass index were the most highly ranked variables, followed by additional laboratory variables, including hemoglobin, monocyte count, neutrophil count, electrolyte-related parameters, and C-reactive protein.

Overall, the composition of top-ranked variables differed across models, with treatment-related variables more frequently appearing among the highest-ranked in the SVM model, whereas laboratory variables predominated in the AdaBoost and RUSBoost models.

Discussion

In this study, we developed and evaluated machine learning-based models to predict UAC among outpatients receiving S-1 chemotherapy. Compared with a conventional logistic regression model, the SVM model demonstrated superior performance in recall-oriented metrics, indicating improved detection of patients who experienced UAC. Notably, the SVM model consistently reduced FN predictions in the independent test dataset, highlighting its potential utility for identifying high-risk patients in outpatient oral anticancer therapy.

The primary contribution of this study lies in its explicit prioritization of FN reduction for UAC prediction. UAC is a relatively infrequent but clinically significant event, and failure to identify high-risk patients may result in delayed intervention and adverse outcomes (14, 15). In datasets characterized by class imbalance, overall accuracy is known to be strongly influenced by the majority class and may inadequately reflect performance for minority events such as UAC (16). In this context, recall-oriented evaluation is particularly important. Metrics such as the F2 score, which places greater emphasis on recall than precision, and the PR-AUC have been shown to provide more informative assessments than accuracy or ROC-AUC in imbalanced classification settings (16, 17). On the basis of these considerations, recall, the F2 score, and PR-AUC were selected as the primary evaluation metrics in this study, enabling model development and assessment with a specific focus on minimizing missed UAC events.

The superior performance of the SVM model relative to logistic regression may be partly explained by its ability to capture complex relationships among heterogeneous clinical features. Logistic regression models, which are typically based on linear associations between predictors and outcomes, may therefore be insufficient to fully represent such complexity. These characteristics may have contributed to the improved UAC detection observed in the present study.

An additional methodological feature of this study was the use of SHAP analysis for both model interpretation and feature selection. SHAP-based selection of variables improved the cross-validated performance of the SVM model, suggesting reduced model complexity and improved generalizability. Furthermore, SHAP analysis revealed differences in the composition of influential variables across models. In the final SVM model, treatment-related variables, such as treatment setting and concomitant use of injectable anticancer agents, frequently appeared among the top-ranked features, whereas laboratory variables predominated in the AdaBoost and RUSBoost models. These observations indicate that different modeling approaches may emphasize distinct aspects of the available clinical information. Importantly, SHAP analysis was used to enhance model transparency and interpretability rather than to infer causal relationships, and the identified feature patterns should be interpreted accordingly (12).

From a clinical perspective, the proposed predictive model relies exclusively on baseline laboratory values and patient background information available prior to treatment initiation. This characteristic enhances its feasibility as a screening tool in outpatient settings, where rapid risk stratification without additional clinical burden is desirable. Identification of patients at elevated risk of UAC may facilitate intensified monitoring, early supportive care, or tailored follow-up strategies, potentially improving the safety of outpatient chemotherapy (14, 15). However, whether implementation of such a model leads to reductions in UAC incidence or improvements in clinical outcomes requires further investigation.

Study limitations. Firstly, this was a retrospective single-center study, and patient characteristics and treatment practices may have varied across institutions, limiting generalizability. Secondly, the number of UAC events was relatively small, reflecting real-world incidence but raising the possibility of model instability, particularly for more complex algorithms. Thirdly, the analysis focused primarily on pre-treatment variables and did not incorporate dynamic factors such as dose modifications, supportive care interventions, or medication adherence during treatment. In addition, model performance was evaluated retrospectively, and the clinical impact of model-guided interventions remains untested. Future studies should therefore include external validation using multi-center datasets, incorporation of longitudinal clinical information, and prospective evaluation of clinical utility, including optimization of decision thresholds for real-world implementation.

Conclusion

This study demonstrates that an SVM-based predictive model emphasizing recall-oriented evaluation metrics can improve the detection of UAC risk among outpatients receiving S-1 chemotherapy. By prioritizing the avoidance of FN predictions and utilizing routinely available baseline clinical data, this approach may support safer outpatient oral anticancer treatment. Further external validation and prospective studies are warranted to establish its clinical utility.

Conflicts of Interest

The Authors declare there are no conflicts of interest in regard to this study.

Authors’ Contributions

MT, TN, MN and YK conceived the study. MT and TN analyzed the data and performed the statistical analysis. MT and YK drafted the manuscript. KK, MF, AM, YM, TN and MN contributed to discussions and reviewed the manuscript. YK supervised the study and takes responsibility for the integrity of the data and the accuracy of the analysis. All authors approved the final manuscript.

Artificial Intelligence (AI) Disclosure

During the preparation of this manuscript, language editing and stylistic refinement were performed with the assistance of a large language model (ChatGPT, OpenAI). All scientific content, including study design, data analysis, and interpretation, was developed by the authors. The final manuscript was critically reviewed and approved by all Authors. No figures or visual data were generated or modified using AI tools.