Machine learning prediction model for distant metastasis at diagnosis in pediatric neuroblastoma: development, interpretation, and temporal validation

Abstract

Background

Distant metastasis (DM) at diagnosis is a key determinant of risk stratification, treatment intensity and prognosis in pediatric neuroblastoma (NB). However, an interpretable risk-estimation tool based only on routinely available variables at diagnosis or during baseline diagnostic/pathological assessment and assessed through temporal validation remains limited. This study aimed to develop and temporally validate an interpretable machine-learning model for estimating DM risk at diagnosis.

Methods

We constructed a population-based pediatric cancer registry cohort of patients aged 0–19 years with NB or ganglioneuroblastoma (GNB). The primary outcome was DM at diagnosis. Candidate predictors were restricted to demographic and tumor-related variables available at diagnosis or during baseline diagnostic/pathological assessment, while surgery, radiotherapy and chemotherapy were excluded to reduce information leakage. Feature selection integrated univariable logistic regression, least absolute shrinkage and selection operator (LASSO) regression, Boruta and prespecified clinical retention rules. Patients diagnosed during 2000–2017 formed the development cohort, which was stratified 7:3 into training and internal validation sets; patients diagnosed during 2018–2023 formed the temporal validation cohort. Seven algorithms were compared. Shapley Additive Explanations (SHAP) analysis and decision curve analysis (DCA) were used to assess model interpretability and net benefit.

Results

The primary cohort included 3,837 patients, of whom 1,998 (52.1%) had DM at diagnosis. Light gradient boosting machine (LightGBM) was selected as the final model. It achieved an area under the receiver operating characteristic curve (AUC) of 0.775 [95% confidence interval (CI): 0.743–0.804] in internal validation and 0.802 (95% CI: 0.774–0.829) in temporal validation with Brier scores of 0.193 and 0.181, respectively. In temporal validation, sensitivity and specificity were 0.750 and 0.729. DCA suggested net benefit across a range of threshold probabilities. SHAP analysis identified laterality, age at diagnosis, histologic subtype, tumor size and regional lymph nodes examined as leading model-contributing features. A prototype web-based calculator was developed for research-oriented visualization of model predictions. In an exploratory secondary analysis, lung metastasis was associated with higher 36-month mortality odds among patients with DM.

Conclusions

A LightGBM model based on routinely available variables at diagnosis or during baseline diagnostic/pathological assessment showed stable discrimination and calibration for estimating DM risk in pediatric NB. This tool may support research-oriented early risk stratification, but independent external and prospective validation is required before clinical use.

Highlight box.

Key findings

• This population-based registry study developed and temporally validated an interpretable machine-learning model for estimating distant metastasis (DM) at diagnosis in pediatric neuroblastoma (NB) and ganglioneuroblastoma.

• The final light gradient boosting machine (LightGBM) model showed stable discrimination, with an area under the receiver operating characteristic curve of 0.775 in internal validation and 0.802 in temporal validation.

• Shapley Additive Explanations (SHAP) analysis identified tumor laterality, age at diagnosis, histologic subtype, tumor size, and regional lymph nodes examined as leading contributors to model output.

What is known and what is new?

• DM at diagnosis is a key determinant of risk stratification, treatment intensity, and prognosis in pediatric NB. Existing prediction studies often lack temporal validation, model interpretability, or assessment of clinical utility.

• This study used routinely available variables at diagnosis or during baseline diagnostic/pathological assessment, excluded treatment-related variables to reduce information leakage, and combined temporal validation, SHAP interpretation, decision curve analysis, and a local prototype web-based calculator.

What is the implication, and what should change now?

• The proposed model may provide a research-oriented tool for early risk estimation, cohort stratification, and standardized reporting in pediatric NB.

• The calculator should not replace systematic staging evaluation, imaging assessment, pathological confirmation, or multidisciplinary decision-making.

• Independent external validation and prospective evaluation are required before clinical implementation.

Introduction

Neuroblastoma (NB) is the most common extracranial solid tumor in children and one of the prototypical malignancies of infancy and early childhood (1-4). Using data from the Global Burden of Disease (GBD) study, Nong et al. characterized the global trends in incidence, mortality, and disability-adjusted life years (DALYs) for pediatric NB from 1990 to 2021, reporting that NB accounts for approximately 8–10% of childhood malignancies and remains one of the most frequent extracranial solid tumors in this population (5). NB is marked by pronounced biological and clinical heterogeneity. Some children with low-risk disease may experience spontaneous tumor regression or require only limited treatment, whereas others present at diagnosis with highly aggressive disease and extensive dissemination (6-9). Within this heterogeneous disease spectrum, the presence of distant metastasis (DM) at diagnosis represents a critical determinant of risk classification, treatment intensity, and long-term prognosis. Children with DM are generally assigned to high-risk categories, require intensive multimodal therapy, and have substantially poorer overall survival outcomes (10-12). Therefore, metastatic status at diagnosis is not only a key component of initial clinical assessment but also an essential variable for subsequent risk stratification and standardized research reporting.

In clinical practice, confirmation of DM relies on a comprehensive staging work-up, including functional imaging, cross-sectional imaging, bone marrow assessment, pathological confirmation, and molecular biomarker testing (13-15). These examinations remain indispensable for accurate disease staging; however, their availability and timing may vary across healthcare settings, resource levels, and the early diagnostic phase. Therefore, an early quantitative estimate of the probability of DM based on demographic and tumor-related characteristics available at diagnosis or during baseline diagnostic/pathological assessment may support risk communication, research stratification, and standardized description of cohort profiles. Importantly, the use of routine variables available during diagnosis and baseline assessment is not intended to replace complete staging evaluation. Rather, it represents a deliberate modeling strategy designed to reduce dependence on post-diagnostic treatment information, complex molecular testing, or resource-intensive examinations, thereby improving feasibility and transferability across routine research and clinical data environments.

Machine learning (ML) methods can capture potential nonlinear relationships and interactions among variables and have been increasingly applied to cancer risk assessment and outcome prediction (16,17). Zhang et al. reviewed the applications of artificial intelligence (AI) and ML in cancer risk assessment, early diagnosis, prognostic prediction, and treatment selection (18). Nevertheless, several common limitations continue to impede the clinical translation of existing prediction models. First, some studies have relied primarily on a single random split for internal evaluation, without temporal validation or independent cohort validation; consequently, the stability of these models across different diagnostic periods remains uncertain. Second, complex models are often perceived as “black boxes”, and insufficient interpretability may limit clinicians’ understanding of, and confidence in, model-derived predictions. Model interpretability has therefore been increasingly recognized as a prerequisite for implementing prediction models in real-world clinical settings (19-23). In addition, many studies have focused mainly on discrimination metrics, while less attention has been given to calibration performance, clinical net benefit, and the practical usability of model-based tools (24-27). For estimating the risk of DM at diagnosis in children with NB, a prediction framework that integrates temporal validation, model interpretability, clinical utility assessment, and practical applicability is still needed.

Therefore, using a population-based cohort of children with NB, we developed and temporally validated an interpretable ML model to estimate the risk of DM at diagnosis using demographic and tumor-related variables available at diagnosis or during baseline diagnostic/pathological assessment. Seven ML algorithms were compared, and light gradient boosting machine (LightGBM) was selected as the final primary model. Its discrimination, calibration, and clinical net benefit were systematically evaluated in both the internal validation cohort and the temporal validation cohort. To avoid using post-diagnostic information to predict metastatic status at diagnosis, treatment-related variables, including surgery, radiotherapy, and chemotherapy, were not incorporated into the primary model. We further applied Shapley Additive Explanations (SHAP) to interpret feature contributions and developed a local prototype web-based calculator for visualization of model predictions and model interpretation in research settings. As a supplementary exploratory analysis, we also assessed clinical factors associated with 36-month mortality among children with DM. Overall, this study aimed to establish an interpretable and temporally validated risk prediction model based on routinely available information from diagnosis and baseline assessment, providing a reference for early risk stratification and standardized research reporting in pediatric NB. We present this article in accordance with the TRIPOD+AI reporting checklist (available at https://tp.amegroups.com/article/view/10.21037/tp-2026-0650/rc).

Methods

Study design, data source, and cohort construction

This study was designed as a population-based retrospective prediction modeling study. Data were obtained from the United States Surveillance, Epidemiology, and End Results (SEER) Program using the SEER 17 Registries, November 2025 submission, which includes cases diagnosed from 2000 through 2023. The study was conducted in accordance with the Declaration of Helsinki and its subsequent amendments. Because the data are publicly available and de-identified, this study was exempt from Institutional Review Board approval, and the requirement for informed consent was waived.

The study population was identified using the SEER variable “International Classification of Childhood Cancer (ICCC) site recode 3rd edition/International Agency for Research on Cancer (IARC) 2017”, with the exact category “IVa Neuroblastoma and ganglioneuroblastoma” used to identify NB and ganglioneuroblastoma (GNB). A total of 97,851 pediatric tumor records were initially identified, of which 4,117 met the ICCC IVa criterion. The analytic cohort was constructed at the patient level. For patients with more than one eligible record sharing the same patient ID, records were ordered by year of diagnosis and sequence number, and the first eligible record was retained. The 44 excluded records were not exact duplicate database rows; rather, they were additional tumor records from 44 patients with more than one eligible tumor record. Under this one-record-per-patient rule, 44 additional tumor records were excluded. A further 236 records with missing or indeterminate DM status were excluded, leaving 3,837 patients in the primary analytic cohort. The sample size was determined by the availability of all eligible cases in the registry during the study period, and no formal a priori sample-size calculation was performed.

Outcome definition

The primary outcome was the presence of DM at diagnosis. DM status was determined using three SEER staging variables in a prespecified hierarchical order: (I) Combined Summary Stage with Expanded Regional Codes [2004+]; (II) Summary Stage 2000 [1998–2017]; and (III) SEER Historic Stage A (1973–2015). When Combined Summary Stage with Expanded Regional Codes [2004+] had a nonblank value, “Distant site(s)/node(s) involved” was classified as DM, whereas “Localized only”, “Regional by direct extension only”, “Regional lymph nodes involved only”, and “Regional by both direct extension and lymph node involvement” were classified as non-DM. “Unknown/unstaged/unspecified/DCO” and other nonclassifiable nonblank values were treated as indeterminate and excluded. Only when this higher-priority variable was blank or unavailable was Summary Stage 2000 [1998–2017] used; “Distant” was classified as DM, “Localized” or “Regional” as non-DM, and “Unknown/unstaged” or other nonclassifiable nonblank values as indeterminate. If both preceding variables were unavailable, SEER Historic Stage A [1973–2015] was used, with “Distant” classified as DM, “Localized” or “Regional” as non-DM, and “Unstaged” or other nonclassifiable nonblank values as indeterminate. When multiple staging variables contained usable information, the highest-priority variable determined the final classification. A nonblank but indeterminate value in a higher-priority field was not overridden by a lower-priority staging variable.

The secondary exploratory outcome was 36-month mortality among patients with DM. In children with DM, death within 36 months after diagnosis was defined as an event, whereas patients who survived beyond 36 months were classified as non-events. Patients diagnosed after 2020 or those for whom 36-month mortality status could not be determined were excluded from this exploratory analysis.

Candidate predictors

Initial candidate variables were prespecified according to the study objective and information available in the SEER database at diagnosis or during baseline diagnostic/pathological assessment. These variables included age at diagnosis, sex, race/ethnicity, year of diagnosis, primary site, histologic subtype, tumor laterality, method of diagnostic confirmation, tumor size, regional lymph nodes examined, positive regional lymph nodes, county-level median household income, and rural-urban residence. Tumor size was recorded in millimeters, and regional nodal variables were derived from SEER regional lymph node fields. Year of diagnosis was categorized as 2000–2007, 2008–2015, and 2016–2023 to represent different diagnostic periods. The number of regional lymph nodes examined is an assessment-related variable obtained during diagnostic/pathological evaluation rather than an intrinsic tumor characteristic. Accordingly, the intended use point of the primary model is the baseline stage after routine diagnostic and pathological assessment information has become available, rather than the initial presentation before any pathological assessment.

Treatment-related variables, including surgery, radiotherapy, and chemotherapy, were not included in the primary prediction model. These variables generally reflect post-diagnostic treatment decisions and disease-management pathways; using them to predict DM status at diagnosis could introduce post-diagnostic information and lead to information leakage.

Data preprocessing and missing-value imputation

To minimize the risk of information leakage, all missing-value imputation rules were fitted exclusively in the training cohort and then applied unchanged to the internal and temporal validation cohorts. Continuous variables were imputed using the training-cohort median, whereas categorical variables were imputed using the training-cohort mode. Variables with more than 40% conventional missingness in the training cohort and variables with zero variance were removed during preprocessing. Detailed variable definitions, coding schemes, missingness or special-code status, and preprocessing decisions are provided in Tables S1,S2 and Appendix 1.

Positive regional lymph nodes did not simply have 67.4% conventional missingness. Re-examination of the original SEER codes showed that, among the 3,837 patients, 1,249 (32.55%) had numeric values from 0 to 90; codes 95, 97, 98, and 99 occurred in 28 (0.73%), 134 (3.49%), 2,142 (55.82%), and 284 (7.40%) patients, respectively, and code 96 was not observed. All 2,142 records with code 98 corresponded exactly to Regional Nodes Examined code 00, and the converse was also true, indicating a structural non-assessment pattern rather than conventional missingness. Codes 95, 97, and 99 were treated as other special ascertainment codes. Because only 1,249 patients had directly usable numeric values and the availability of this variable was conditional on regional nodal assessment, Positive regional lymph nodes were excluded before supervised predictor selection and were not entered into univariable logistic regression, least absolute shrinkage and selection operator (LASSO) regression, Boruta, or any prediction model. Primary site was also removed during preprocessing because no usable variation remained after recoding.

Predictor selection

To prevent validation data from influencing model development, the development cohort was split into the training and internal validation cohorts before any outcome-informed predictor selection. All supervised predictor-selection procedures, including univariable logistic regression, LASSO regression, and the Boruta algorithm, were performed exclusively within the training cohort; outcome information from the internal and temporal validation cohorts was not used for predictor selection or retention decisions.

After preprocessing, 11 predictors remained eligible for supervised predictor selection. Candidate predictor selection used a prespecified combined strategy in which the three statistical procedures were treated as complementary screening approaches. Univariable logistic regression was used to assess the association of each candidate predictor with the outcome, LASSO regression was used for variable shrinkage and selection in a multivariable framework, and Boruta was used to identify outcome-related variables based on random-forest importance. In addition, age at diagnosis, sex, race/ethnicity, and histologic subtype were prespecified as clinically retained variables. The results were integrated using a union rule to avoid excluding potentially informative predictors solely because they were not selected by a single statistical procedure. A candidate predictor was retained for model development if it was selected by LASSO or Boruta, had an overall P value <0.05 in univariable logistic regression, or belonged to the prespecified clinically retained set.

Based on these rules, nine variables available at diagnosis or during baseline diagnostic/pathological assessment were retained for model development: age at diagnosis, sex, race/ethnicity, year of diagnosis, histologic subtype, tumor laterality, method of diagnostic confirmation, tumor size, and the number of examined regional lymph nodes. The supporting results of univariable analysis, LASSO, and Boruta are presented in Table S3.

Model development

Patients diagnosed between 2000 and 2017 constituted the development cohort, which included 2,884 patients. The development cohort was further divided by stratified random sampling according to DM status, with 70% assigned to the training cohort and 30% to the internal validation cohort. The final training cohort included 2,019 patients, and the internal validation cohort included 865 patients. Patients diagnosed between 2018 and 2023 constituted the temporal validation cohort, comprising 953 patients, and were used to evaluate model stability in a more recent diagnostic period. LASSO was implemented using cv.glmnet for a binomial outcome with alpha =1 and 10-fold cross-validation; the penalty parameter was selected according to the 1-standard-error rule (lambda.1se). Boruta was run with a maximum of 100 iterations, a P value threshold of 0.01, multiple-comparison adjustment enabled, and random seed 123; tentative features were resolved using TentativeRoughFix, and only confirmed features were retained. The prediction random forest was implemented using scikit-learn’s RandomForestClassifier; the selected configuration was n_estimators =500, max_features = ‘sqrt’, and min_samples_leaf =5, with hyperparameter tuning performed by fivefold stratified cross-validation in the training cohort. The development and temporal validation cohorts were derived from the same SEER registry system and used the same eligibility criteria, outcome definition, predictor definitions, and preprocessing rules. The main planned difference between the two cohorts was the calendar period of diagnosis.

Seven ML algorithms were compared in this study, including logistic regression, decision tree, random forest, eXtreme Gradient Boosting (XGBoost), LightGBM, Support Vector Machine (SVM), and multilayer perceptron (MLP)/artificial neural network (ANN). All models were trained in the training cohort, and hyperparameters were tuned using five-fold cross-validation, with the area under the receiver operating characteristic curve (AUC) used as the primary optimization metric. The optimal hyperparameters are provided in Table S4. LightGBM was selected as the final primary model. The model was implemented as a complete ML pipeline consisting of variable preprocessing, categorical variable encoding, continuous variable standardization, and the LightGBM classifier. The model was designed to output individualized predicted probabilities rather than formal risk categories. No formal risk groups were created for model evaluation. Classification metrics were calculated using a prespecified probability threshold of 0.5. No model updating or recalibration was performed after temporal validation. Because the event proportion for the primary outcome was approximately 52%, indicating a largely balanced class distribution, Synthetic Minority Over-sampling Technique (SMOTE) or other resampling methods were not applied during model development.

Model evaluation, interpretation, and prototype web-based calculator

For internal and temporal validation, the complete preprocessing and LightGBM pipeline fitted in the training cohort was applied to each validation cohort without refitting. Individual predicted probabilities of DM were then generated and compared with observed metastatic status. Model performance was evaluated in both the internal validation cohort and the temporal validation cohort. Discrimination was assessed using the AUC, and 95% confidence intervals (CIs) were estimated using 1,000 bootstrap resamples with the percentile method. Calibration was evaluated using the Brier score and calibration curves. Classification performance metrics included accuracy, sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and F1 score. Clinical utility was assessed using decision curve analysis (DCA) to estimate the net benefit of the model across a range of threshold probabilities. The complete validation performance of all seven algorithms is presented in Tables S5,S6.

To enhance the interpretability of the final model, SHAP analysis was used to quantify the relative contribution of each predictor to the model output, and the 10 most influential encoded features were visualized. SHAP values were used solely for model-level interpretation and were not intended to infer causal relationships between predictors and DM. Based on the final LightGBM model, we developed a prototype web-based calculator for research-oriented visualization of individualized predicted probabilities and model interpretation. The prototype uses the same nine predictors and the fitted LightGBM pipeline as the primary model. It is currently maintained in a local research environment and has not been deployed to a persistent public server; therefore, it should not be considered a publicly accessible clinical tool.

Because the timing of regional lymph nodes examined may vary across diagnostic workflows, we additionally performed a sensitivity analysis excluding this variable. With the modeling procedure, training-cohort split, and hyperparameter-tuning strategy otherwise unchanged, an 8-predictor LightGBM model was rebuilt and retuned, and its discrimination and calibration were evaluated in the internal and temporal validation cohorts to assess the dependence of the primary model on this variable.

Secondary exploratory prognostic analysis

As a secondary exploratory analysis, rather than as an independent ML prediction model, we further evaluated clinical factors associated with 36-month mortality among children with DM. The full metastatic cohort included patients with DM who were diagnosed between 2000 and 2020 and had determinable 36-month mortality status. A total of 1,704 patients were included, of whom 491 died within 36 months after diagnosis. This analysis was performed using multivariable logistic regression and incorporated demographic variables, tumor-related variables, and variables related to treatment receipt. Missing tumor size values were imputed using the median value from the full metastatic cohort, with an additional missingness indicator included in the model. Missing surgical information was treated as an “Unknown” category. Baseline characteristics, univariable models, the multivariable base model, and the treatment-extended model are presented in Tables S7-S10.

Given the substantial structural missingness of metastatic-site information in earlier diagnostic years, analyses involving specific metastatic sites were conducted only as exploratory analyses in a modern subcohort. The modern metastatic-site cohort was defined as patients with DM diagnosed between 2016 and 2020. The initial sample included 369 patients, of whom 84 died within 36 months. After excluding eight patients with missing records for any metastatic site, the final complete-case analysis included 361 patients, including 80 deaths. After adjustment for age at diagnosis, sex, race/ethnicity, and histologic subtype, this analysis explored the associations between different metastatic sites and 36-month mortality. Complete model estimates are provided in Table S11. Treatment-related variables were interpreted only as indicators related to treatment receipt and disease severity, and no causal inference was made.

Statistical analysis

Baseline characteristics were summarized according to the presence or absence of DM at diagnosis. Continuous variables are presented as medians with interquartile ranges and were compared between groups using the Wilcoxon rank-sum test. Categorical variables are reported as frequencies and percentages and were compared using the Chi-squared test or Fisher’s exact test, as appropriate. All statistical tests were two-sided.

Results

Study cohort and baseline characteristics

The final primary analysis cohort included 3,837 children with NB or GNB, of whom 1,998 had DM at diagnosis, corresponding to an event rate of 52.1%. The development cohort included 2,884 patients with 1,487 DM events. After stratified random splitting, the training cohort included 2,019 patients with 1,041 events, and the internal validation cohort included 865 patients with 446 events. The temporal validation cohort included 953 patients with 511 events. Compared with patients without DM, those with DM at diagnosis were older, had larger tumors, showed a higher proportion of NB not otherwise specified, differed in the distribution of tumor laterality, and were more frequently confirmed by histopathological diagnosis (all P<0.001). In contrast, year of diagnosis, county-level median household income, rural-urban residence, and the number of examined regional lymph nodes were broadly comparable between the two groups (Table 1).

Table 1. Baseline characteristics of the study population according to distant metastasis status.

Characteristic	Overall (n=3,837)	No distant metastasis (n=1,839)	Distant metastasis (n=1,998)	P value
Age at diagnosis, years	1 [0–3]	1 [0–3]	2 [0–4]	<0.001
Tumor size, mm	65 [43–99]	55 [35–83]	77 [52–110]	<0.001
Regional lymph nodes examined	0 [0–1]	0 [0–1]	0 [0–1]	0.690
Positive regional lymph nodes	1 [0–3] (n=1,249)	1 [0–2] (n=619)	2 [1–4] (n=630)	<0.001
Sex				<0.001
Female	1,811 (47.2)	922 (50.1)	889 (44.5)
Male	2,026 (52.8)	917 (49.9)	1,109 (55.5)
Race/ethnicity				0.001
Non-Hispanic White	2,097 (54.7)	1,036 (56.3)	1,061 (53.1)
Hispanic	905 (23.6)	450 (24.5)	455 (22.8)
Other race/ethnicity	835 (21.8)	353 (19.2)	482 (24.1)
Year of diagnosis				0.79
2000–2007	1,270 (33.1)	603 (32.8)	667 (33.4)
2008–2015	1,329 (34.6)	647 (35.2)	682 (34.1)
2016–2023	1,238 (32.3)	589 (32.0)	649 (32.5)
Histologic subtype				<0.001
Neuroblastoma, NOS	3,223 (84.0)	1,357 (73.8)	1,866 (93.4)
Ganglioneuroblastoma	614 (16.0)	482 (26.2)	132 (6.6)
Laterality				<0.001
Not paired/bilateral/other	1,790 (46.7)	1,126 (61.2)	664 (33.2)
Right	942 (24.6)	323 (17.6)	619 (31.0)
Left	1,105 (28.8)	390 (21.2)	715 (35.8)
Diagnostic confirmation				<0.001
Positive histology	3,728 (97.2)	1,750 (95.2)	1,978 (99.0)
Radiography/clinical only	98 (2.6)	85 (4.6)	13 (0.7)
Other/unknown confirmation	11 (0.3)	4 (0.2)	7 (0.4)
County-level median household income				0.87
<$75,000	1,083 (28.2)	517 (28.1)	566 (28.3)
$75,000–$99,999	1,779 (46.4)	860 (46.8)	919 (46.0)
≥ $100,000	974 (25.4)	461 (25.1)	513 (25.7)
Rural-urban residence				0.96
Metro area with ≥1 million population	2,351 (61.3)	1,123 (61.1)	1,228 (61.5)
Other metro area	1,476 (38.5)	711 (38.7)	765 (38.3)
Non-metro/rural	10 (0.3)	5 (0.3)	5 (0.3)

Data are median [interquartile range] or n (%). P values were calculated using Wilcoxon rank-sum, Chi-squared, or Fisher’s exact tests, as appropriate. Positive regional lymph nodes were summarized descriptively only among patients with numeric values from 0 to 90 (overall n=1,249; no distant metastasis n=619; distant metastasis n=630). SEER special codes 95, 97, 98, and 99 were not included in the median and interquartile-range calculations. This variable was excluded before supervised predictor selection and was not used for model development. Primary site was omitted because no usable variability remained after recoding and preprocessing. NOS, not otherwise specified; SEER, Surveillance, Epidemiology, and End Results.

Predictor selection and candidate predictors

After integrating evidence from univariable logistic regression, LASSO regression, the Boruta algorithm, and prespecified clinical retention criteria, nine variables available at diagnosis or during baseline diagnostic/pathological assessment were retained for model development: age at diagnosis, sex, race/ethnicity, year of diagnosis, histologic subtype, tumor laterality, method of diagnostic confirmation, tumor size, and the number of examined regional lymph nodes. The supporting evidence for predictor selection is presented in Table S3.

Model development and performance comparison

The discriminatory performance of the seven ML algorithms in the internal validation cohort and temporal validation cohort is shown in Table 2 and Figure 1. In the internal validation cohort, LightGBM achieved the highest AUC of 0.775 (95% CI: 0.743–0.804), followed by random forest (AUC =0.771), XGBoost (AUC =0.768), MLP/ANN (AUC =0.761), SVM (AUC =0.750), decision tree (AUC =0.750), and logistic regression (AUC =0.724).

Table 2. Discrimination and calibration performance of machine learning models.

Model	Internal validation AUC (95% CI)	Temporal validation AUC (95% CI)	Internal validation Brier score	Temporal validation Brier score
LightGBM	0.775 (0.743–0.804)	0.802 (0.774–0.829)	0.193	0.181
Random forest	0.771 (0.739–0.801)	0.801 (0.772–0.828)	0.194	0.184
XGBoost	0.768 (0.736–0.799)	0.807 (0.778–0.835)	0.195	0.180
MLP/ANN	0.761 (0.727–0.792)	0.797 (0.768–0.824)	0.199	0.185
SVM	0.750 (0.718–0.782)	0.799 (0.771–0.824)	0.204	0.182
Decision tree	0.750 (0.717–0.784)	0.794 (0.764–0.821)	0.204	0.180
Logistic regression	0.724 (0.689–0.758)	0.782 (0.753–0.812)	0.212	0.192

Internal validation refers to the held-out validation subset from the development cohort; temporal validation refers to patients diagnosed in 2018–2023. Models are ranked by internal validation AUC. ANN, artificial neural network; AUC, area under the receiver operating characteristic curve; CI, confidence interval; LightGBM, light gradient boosting machine; MLP, multilayer perceptron; SVM, Support Vector Machine; XGBoost, eXtreme Gradient Boosting.

Figure 1.

Receiver operating characteristic curves for the seven machine-learning models. (A) Internal validation set. (B) Temporal validation set. ANN, artificial neural network; AUC, area under the receiver operating characteristic curve; LightGBM, light gradient boosting machine; MLP, multilayer perceptron; ROC, receiver operating characteristic; SVM, Support Vector Machine; XGBoost, eXtreme Gradient Boosting.

In the temporal validation cohort, the AUC values of the models were generally comparable. XGBoost achieved the numerically highest AUC of 0.807, while LightGBM yielded an AUC of 0.802, with only a small absolute difference between the two models. Because LightGBM ranked first in the internal validation cohort and maintained comparable and stable discriminatory performance in the temporal validation cohort, it was selected as the final primary model for subsequent model interpretation and development of the prototype web-based calculator.

Calibration analysis indicated an overall acceptable agreement between predicted probabilities and the observed proportions of DM, with only small differences in Brier scores across models (Table 2, Figure 2).

Figure 2.

Calibration curves for the seven machine-learning models. (A) Internal validation set. (B) Temporal validation set. The diagonal line represents perfect agreement between predicted and observed probabilities. ANN, artificial neural network; LightGBM, light gradient boosting machine; MLP, multilayer perceptron; SVM, Support Vector Machine; XGBoost, eXtreme Gradient Boosting.

DCA suggested that the final model provided a higher net benefit than the treat-all and treat-none strategies across part of the clinically relevant threshold range (Figure 3).

Figure 3.

Decision curve analysis for the seven machine-learning models. (A) Internal validation set. (B) Temporal validation set. Net benefit is shown across threshold probabilities together with the treat-all and treat-none strategies. ANN, artificial neural network; LightGBM, light gradient boosting machine; MLP, multilayer perceptron; SVM, Support Vector Machine; XGBoost, eXtreme Gradient Boosting.

Validation performance of the final LightGBM model

The detailed performance of the final LightGBM model in the two validation cohorts is presented in Table 3. In the internal validation cohort, the model achieved an AUC of 0.775 (95% CI: 0.743–0.804), with a Brier score of 0.193. Using a probability threshold of 0.5, the model yielded an accuracy of 0.704, a sensitivity of 0.767, a specificity of 0.637, a PPV of 0.692, an NPV of 0.720, and an F1 score of 0.728.

Table 3. Detailed performance of the final LightGBM model.

Validation cohort	AUC (95% CI)	Brier score	Accuracy	Sensitivity	Specificity	PPV	NPV	F1 score
Internal validation	0.775 (0.743–0.804)	0.193	0.704	0.767	0.637	0.692	0.720	0.728
Temporal validation	0.802 (0.774–0.829)	0.181	0.740	0.750	0.729	0.761	0.716	0.755

The classification metrics were calculated using a prespecified probability threshold of 0.5. AUC, area under the receiver operating characteristic curve; CI, confidence interval; LightGBM, light gradient boosting machine; NPV, negative predictive value; PPV, positive predictive value.

In the temporal validation cohort, the LightGBM model achieved an AUC of 0.802 (95% CI: 0.774–0.829), with a Brier score of 0.181. At the same probability threshold, the model showed an accuracy of 0.740, sensitivity of 0.750, specificity of 0.729, PPV of 0.761, NPV of 0.716, and F1 score of 0.755. Overall, the final model showed no evidence of performance degradation in the temporal validation cohort, suggesting relatively stable discrimination and calibration among patients diagnosed in a more recent period. In the sensitivity analysis excluding regional lymph nodes examined (Table S12), the 8-predictor LightGBM model showed only minor changes in performance compared with the 9-predictor primary model. In the internal validation cohort, the 8-predictor model achieved an AUC of 0.769 (95% CI: 0.736–0.799) and a Brier score of 0.195, compared with 0.775 (95% CI: 0.743–0.804) and 0.193, respectively, for the primary model. In the temporal validation cohort, the corresponding AUC and Brier score were 0.801 (95% CI: 0.771–0.829) and 0.182, compared with 0.802 (95% CI: 0.774–0.829) and 0.181 for the primary model. These findings suggest that the overall predictive performance of the primary model was not highly dependent on regional lymph nodes examined.

Model interpretability

SHAP analysis showed that tumor laterality, age at diagnosis, histologic subtype, tumor size, and the number of examined regional lymph nodes were among the features with the greatest contributions to the output of the LightGBM model (Figure 4). Because categorical variables were encoded before modeling, some SHAP entries represent category-specific encoded features rather than entire raw variables. Overall, larger tumor size and older age at diagnosis tended to shift model predictions toward a higher probability of DM. The contributions of categorical variables, such as tumor laterality and histologic subtype, reflected differences in model output across categories relative to their respective reference groups. These SHAP findings should be interpreted as feature contributions at the model level rather than causal effects of individual variables on DM.

Figure 4.

SHAP summary plot of the final LightGBM model, showing the top 10 encoded features and their contribution to model output. NOS, not otherwise specified; SHAP, Shapley additive explanations.

Secondary exploratory prognostic analysis among children with DM

As a secondary exploratory analysis, we further evaluated clinical factors associated with 36-month mortality among children with DM. In the full metastatic cohort, 1,704 patients with determinable 36-month mortality status were included, of whom 491 died within 36 months after diagnosis. In the complete treatment-extended multivariable model, age at diagnosis, race/ethnicity, year of diagnosis, tumor laterality, tumor size, and receipt of primary-site surgery were associated with the odds of 36-month mortality. Figure 5A displays selected estimates from this model, and the complete results are provided in Table S10. Older age at diagnosis and larger tumor size were associated with higher odds of 36-month mortality, whereas more recent year of diagnosis was associated with lower odds. Patients who underwent primary-site surgery had lower odds of 36-month mortality; however, this finding should be interpreted as an observational association between treatment receipt and outcome, rather than as evidence of a causal protective effect of surgery.

Figure 5.

Multivariable associations with 36-month mortality among metastatic patients. Selected treatment-extended model estimates (A) and site-specific metastases (B) are shown. Treatment-related results are observational, not causal. CI, confidence interval.

In the exploratory analysis of metastatic sites in the modern cohort, the complete-case analysis included 361 children with DM, of whom 80 died within 36 months after diagnosis. After adjustment for age at diagnosis, sex, race/ethnicity, and histologic subtype, lung metastasis was associated with higher odds of 36-month mortality [adjusted odds ratio (OR) =3.09, 95% CI: 1.36–7.04; P=0.007]. No statistically significant associations with 36-month mortality were observed for bone metastasis, brain metastasis, liver metastasis, distant lymph node metastasis, or metastasis to other sites (Figure 5B; Table S11). Given the limited sample size and exploratory nature of this analysis, findings related to specific metastatic sites should be interpreted cautiously.

Prototype web-based calculator

Based on the final LightGBM model, we further developed a prototype web-based calculator for DM (Figure 6). The prototype uses the nine predictors retained in the final model and applies the complete fitted LightGBM pipeline to generate individualized predicted probabilities, thereby providing a research-oriented interface for visualization of model predictions and model interpretation. The calculator currently runs only in a local research environment and has not been deployed to a persistent public server; accordingly, we do not present it as a publicly accessible clinical prediction tool.

Figure 6.

Prototype web-based calculator for displaying the predicted probability of distant metastasis using the final LightGBM model. The calculator is currently maintained as a local research prototype, and the displayed patient profile is illustrative only and is not intended as an individual clinical recommendation. AUC, area under the receiver operating characteristic curve; LightGBM, light gradient boosting machine; NOS, not otherwise specified.

Discussion

In this population-based pediatric tumor registry study, we developed and temporally validated an interpretable ML model to estimate the risk of DM at diagnosis in children with NB. The model was restricted to demographic and tumor-related variables available at diagnosis or during baseline diagnostic/pathological assessment and excluded post-diagnostic treatment information such as surgery, radiotherapy, and chemotherapy. Among the seven algorithms evaluated, LightGBM was selected as the final primary model, achieving AUCs of 0.775 and 0.802 in the internal and temporal validation cohorts, respectively, without evidence of reduced discrimination or calibration during temporal validation. DCA suggested net benefit across part of the evaluated threshold range, SHAP analysis provided model-level interpretation of feature contributions, and the local web prototype provided a visual interface for model predictions and research-oriented interpretation. Overall, the findings suggest that routinely available information from diagnosis and baseline assessment can support a relatively stable and interpretable estimate of DM risk at diagnosis in children with NB.

The presence or absence of DM at diagnosis is a pivotal determinant of risk stratification, treatment intensity, and long-term prognostic assessment in children with NB. It also represents a clinically important factor that can be assessed during the early diagnostic period. For this clinical question, a risk estimation tool based on routinely available variables obtained at diagnosis or during baseline diagnostic/pathological assessment may provide a useful reference for risk communication, research stratification, and standardized characterization of study cohorts (28-32). It should be emphasized that the present model is not intended to replace systematic staging evaluation, imaging assessment, bone marrow examination, pathological confirmation, molecular testing, or multidisciplinary decision-making. Instead, its value lies in providing a risk estimation framework that requires fewer data inputs, is anchored to a clearly defined baseline assessment stage, and is easier to reproduce across routine data settings. The use of routine variables available at diagnosis or during baseline diagnostic/pathological assessment was a central design principle of this study. Accordingly, the intended application point of the primary model is after routine diagnostic and pathological baseline information has become available, rather than at the initial presentation before pathological assessment. Many previously developed cancer prediction models have relied on complex laboratory, molecular, or imaging-derived features. Although such features may improve apparent predictive performance, they may also limit model accessibility, reproducibility, and transportability in routine research settings and across heterogeneous data environments (33-36). Bera et al. reviewed the applications of radiomics and AI in predicting cancer prognosis, treatment response, and molecular phenotypes, noting that imaging-derived features may enhance outcome prediction but that clinical translation remains constrained by challenges related to standardization, interpretability, validation, and generalizability (37). In the present study, candidate predictors were restricted to variables available at diagnosis or during baseline diagnostic/pathological assessment, and treatment-related variables were excluded. This design helps reduce the risk of information leakage that may arise when post-diagnostic information is used to predict metastatic status at diagnosis.

Regarding model selection, although XGBoost achieved the numerically highest AUC in the temporal validation cohort, the temporal validation AUC of LightGBM was very similar, with only a small absolute difference between the two models. In addition, LightGBM showed superior performance in the internal validation cohort. Considering its internal validation performance, temporal stability, and consistency with the subsequent interpretability analysis and prototype web-based calculator development, LightGBM was selected as the final primary model. The model did not show reduced performance among patients diagnosed between 2018 and 2023, suggesting a degree of stability in a more recent diagnostic period. However, this finding should be interpreted as validation along the temporal dimension rather than independent external validation. Because the temporal validation cohort and development cohort were derived from the same registry system, these results should not be regarded as direct evidence of external generalizability across independent institutions, geographic regions, or healthcare systems.

The model interpretation results indicated that tumor laterality, age at diagnosis, histologic subtype, tumor size, and the number of examined regional lymph nodes were among the features with greater contributions to the output of the LightGBM model. Larger tumor size and older age at diagnosis tended to shift predictions toward a higher probability of DM, which is broadly consistent with current understanding of the clinical heterogeneity and aggressive disease behavior of NB. Although the number of examined regional lymph nodes did not show a statistically significant association with metastatic status in univariable analysis, it still contributed to the multivariable ML model, possibly because it provides complementary information when considered jointly with other predictors or partly reflects the intensity of diagnostic and baseline assessment. These interpretations should remain at the model level: SHAP values quantify contributions to model output and should not be interpreted as evidence of causality or modifiable intervention targets. Importantly, the sensitivity analysis excluding regional lymph nodes examined showed only small changes in AUC and Brier score in both validation cohorts, suggesting that the overall predictive performance of the primary model was not highly dependent on this variable.

As a supplementary exploratory analysis, we further evaluated clinical factors associated with 36-month mortality among children with DM. In the full metastatic cohort, age at diagnosis, race/ethnicity, year of diagnosis, tumor laterality, tumor size, and receipt of primary-site surgery were associated with the odds of 36-month mortality. Older age at diagnosis and larger tumor size were associated with higher odds of 36-month mortality, whereas more recent year of diagnosis was associated with lower odds. Patients who underwent primary-site surgery showed lower odds of 36-month mortality; however, this finding should be interpreted only as an observational association between treatment receipt and outcome, and not as evidence of a causal protective effect of surgery. Treatment selection, disease burden, overall clinical condition, and clinical decision-making pathways may all influence this association, and indication bias is difficult to fully address using retrospective registry data (38-42). In the modern metastatic-site subcohort, lung metastasis was associated with higher odds of 36-month mortality, whereas no significant associations were observed for other metastatic sites. Given the limited sample size of this analysis and the time-period-related structural missingness of metastatic-site information, these findings should be regarded as exploratory and require further validation in larger cohorts with more complete clinical data.

This study has several limitations. First, although internal validation and temporal validation were performed, temporal validation is not equivalent to independent external validation, and generalizability across different institutions, geographic regions, and healthcare systems requires confirmation in external cohorts and prospective studies. Second, population-based registry data lack several biological, molecular, and laboratory variables that are important for NB risk stratification, including MYCN amplification, ALK alterations, tumor ploidy, International Neuroblastoma Risk Group (INRG) stage, lactate dehydrogenase, and ferritin. Third, as a retrospective study, the analysis remains susceptible to selection bias, missing information, and residual confounding. Fourth, missing values were imputed using the training-cohort median or mode; although this strategy helps prevent leakage from the validation cohorts, it may not fully capture complex missing-data mechanisms. Fifth, information on specific metastatic sites showed substantial structural missingness in earlier diagnostic years, so these analyses were exploratory and restricted to a modern subcohort. Sixth, the prototype web-based calculator is currently maintained only as a local research interface for visualization and model interpretation and should not replace standard staging evaluation, clinical diagnosis, or treatment decision-making.

In summary, this study developed and temporally validated an interpretable ML model based on routinely available information from diagnosis and baseline diagnostic/pathological assessment to estimate the risk of DM at diagnosis in children with NB. The model showed relatively stable performance in both internal and temporal validation, while SHAP analysis, DCA, and the local web prototype improved the interpretability and visual presentation of model results. Further independent external validation and prospective studies are warranted to evaluate its applicability across diverse clinical data environments.

Conclusions

Using population-based pediatric tumor data, this study developed and temporally validated an interpretable ML model based solely on routinely available variables at diagnosis or during baseline diagnostic/pathological assessment to predict the risk of DM at diagnosis in children with NB. The final primary model, LightGBM, showed stable performance in both internal and temporal validation, with acceptable calibration and evidence of net benefit across part of the evaluated threshold range in DCA. SHAP analysis and the local web prototype further supported interpretation and visualization of model outputs. The model may provide a reference for early risk stratification and standardized research reporting in pediatric NB, but it should not replace systematic staging evaluation, imaging assessment, pathological confirmation, or multidisciplinary decision-making. Independent external validation and prospective studies are required to confirm its generalizability and clinical applicability.

Supplementary

The article’s supplementary files as

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. The study was conducted in accordance with the Declaration of Helsinki and its subsequent amendments. Because the data are publicly available and de-identified, this study was exempt from Institutional Review Board approval, and the requirement for informed consent was waived.