Toward the best generalizable performance of machine learning in modeling omic and clinical data

Abstract

There are often performance differences between intra-dataset and cross-dataset tests in machine learning (ML) modeling. However, reducing these differences may reduce ML performances. It is thus a challenging dilemma for developing models that excel in intra-dataset testing and are generalizable to cross-dataset testing. Therefore, we aimed to understand and improve performance and generalizability of ML in intra-dataset and cross-dataset testing. We evaluated 4,200 ML models of classifying lung adenocarcinoma (LUAD) deaths using the The Cancer Genome Atlas (TCGA, n=286) and Oncogenomic-Singapore (OncoSG, n=167) datasets, and 1,680 models of classifying glioblastoma deaths using TCGA (n=151) and Clinical Proteomic Tumor Analysis Consortium (CPTAC, n=97) datasets. After examining performance distributions of these ML models, we applied a dual analytical framework, including statistical analyses and SHapley Additive exPlanations-based meta-analysis, to quantify factors’ importance and trace model success back to design principles. We also developed a framework to identify the best generalizable model. Strikingly, Jarque-Bera test revealed significant deviations of model performances from normality in both cancer types and testing contexts. Simple linear models with sparse feature sets consistently dominated in LUAD experiments, whereas non-linear models dominated in glioblastoma ones, suggesting that the best modeling strategy appears cancer-type/disease dependent. Importantly, both robust Analysis of Variance (ANOVA) and Kruskal–Wallis tests consistently identified differentially expressed genes as one of the most influential factors in both cancer types. The proposed multi-criteria framework successfully identified the model that achieved both the best cross-dataset performance and similar intra-dataset performance. In summary, ML performance distributions significantly deviated from normality, which motivates using both robust parametric and non-parametric statistical tests. We quantified and provided possible exploitability on the factors associated with cross-dataset performances and generalizability of ML models in two cancer types. A multi-criteria framework was developed and validated to identify the models that are accurate and consistently robust cross datasets.

Introduction

The advent of high-throughput sequencing technologies has revolutionized biomedical research.^{1, 2} It provides unprecedented access to transcriptomic data that holds immense potential for advancing precision medicine.^{3, 4} Across oncology, transcriptomic datasets offer a powerful resource for developing prognostic models that can predict patient survival outcomes and guide clinical decision-making.^{5, 6} Machine learning (ML) has emerged as an indispensable tool for deciphering these complex, high-dimensional datasets, leading to the development of numerous predictive models.^7–9

Despite this promise, the lack of broad generalizability remains a critical and persistent challenge that hinders the translation of these models into clinical practice.¹⁰ ML models’ superior performances often cannot be transferred to external, independent datasets.^{11, 12} This performance gap is largely attributable to technical variations, platform-specific biases, and batch effects, which are inherent to real-world, heterogeneous data.¹³ Current efforts to mitigate this issue typically fall into two parallel streams: “data-centric” approaches, which focus on data harmonization and batch effect correction (e.g., ComBat-Seq),^{14, 15} and “model-centric” approaches, which emphasize the selection of powerful or robust algorithms (e.g., extreme gradient boosting (XGB), domain adaptation).^{16, 17} However, the fundamental limitation lies in the conspicuous absence of a methodological framework to systematically evaluate their joint impact. This leaves a crucial knowledge gap regarding the complex, interactive effects among various modeling choices, such as feature selection, data normalization, and algorithm selection.¹⁸ However, this knowledge gap is rarely investigated in a systematic, integrated manner.

Therefore, we systematically investigated whether and how the interplay of the fundamental modeling factors of “data-centric” and “model-centric” paradigms^19–21 impacts the performance and cross-dataset robustness of binary classification models, using lung adenocarcinoma (LUAD) and glioblastoma (GBM) datasets as examples. The LUAD dataset consists of The Cancer Genome Atlas (TCGA)²² and Oncogenomic-Singapore (OncoSG)²³ cohorts, while the GBM dataset includes the TCGA and the Clinical Proteomic Tumor Analysis Consortium (CPTAC)²⁴ cohorts. To classify survival outcomes in LUAD and GBM, respectively, we constructed and evaluated a total of 4,200 ML models for LUAD and 1,680 for GBM by combining various feature selection methods, data normalization strategies, and ML algorithms. These models were rigorously examined in both intra-dataset and cross-dataset settings. A novel framework was also developed to identify the best performing yet generalizable ML models.

Methods

Cohorts, Data, and Preprocessing

Figure 1 provides a schematic overview of the general data preprocessing and model development pipeline that forms the basic meta-data (See baseline characteristics of the cases in the 4 datasets in Supplementary Table 1). Since the sample sizes of TCGA were larger than the independent datasets, TCGA datasets were used for model training and intra-dataset testing for both cancer types (n=286 for LUAD and n=151 for GBM). Generalizability of the selected models was evaluated using the OncoSG cohort (n=167) for LUAD cases and the CPTAC cohort (n=97) for GBM cases in cross-dataset testing, respectively. For each modeling run, the TCGA cohort was randomly split into a training set and an intra-dataset test set (80%:20% ratio). The cross-testing performance was averaged over three runs, with each on a randomly selected 90% subset of the independent cross-testing dataset.

Figure 1:

Schematic Overview of the Data Preprocessing and Model Development Pipeline. DEGs, Differentially Expressed Genes; NDEGs, Non-Differentially Expressed Genes.

We analyzed the gene features (RNA-seq, Fragments per kilobase of transcript per million mapped reads) and clinical features that were shared across both intra-dataset and cross-dataset settings. The raw transcriptomic data was initially Z-score normalized. Any remaining missing values were imputed using the median for the respective feature.

To select gene features for our classification models, we performed differential expression analysis on the training set using an Analysis of Variance (ANOVA) F-test to identify genes associated with survival status. Genes with low p-values were designated as Differentially Expressed Genes (DEGs), while those with high p-values, indicating minimal variation between survival groups, were designated as Non-Differentially Expressed Genes (NDEGs) and were utilized as stable internal controls for one of our normalization strategies.²⁵

The Modeling Factor Space

To systematically investigate the drivers of model performance, we defined a comprehensive space of modeling factors, that spans the entire modeling workflow, encompassing choices from both data-centric and model-centric paradigms. The key factors and their respective choices are:

1. Data-Centric Factors (choices related to data preparation):

   • Data Normalization (norm_method): six distinct normalization strategies were applied to the initial Z-scored data: Non-Parametric Normalization (NPN), Quantile Normalization (QN), QN followed by a Z-score transformation (QNZ), Normalization using Internal Control Genes (NICG),^26–30 binarization, and direct use of the Z-scored data (raw_data).
   • Feature Selection (DEG & NDEG): A wide range of feature subsets were created by selecting varying P threshold of DEGs (e.g., 0.1%, 1%, 10%) or NDEGs (e.g., 99%, 95%), allowing us to test the impact of feature space composition and dimensionality.
2. Model-Centric Factors (choices related to the learning algorithm):

   • Algorithm (model): Five classification algorithms were employed: Support Vector Machine (SVM),³¹ Logistic Regression (LR),³² XGB,^{33, 34} Least absolute shrinkage and selection operator (LASSO),^{35, 36} and a Multilayer Perceptron (MLP).^{37, 38}
   • Configuration (weights): For each algorithm, we also tested the effect of applying class weights (weights=yes/no) to address the inherent class imbalance in the survival data.
   • Optimization Target (target): During hyperparameter tuning, models were optimized for either standard Accuracy (ACC) or Balanced Accuracy (BACC).

The full factorial combination of all levels across all these factors resulted in a set of 4,200 unique tuned ML models for LUAD (Supplementary Table 1). Since the cohorts within GBM were relatively balanced in sample distribution, the ‘weights’ modeling factor was not considered, resulting in a set of 1,680 tuned models.

Each of these tuned models was then systematically executed according to the predefined workflow (Figure 1) and randomly repeated five times. The execution involved training each unique model configuration on the training set and subsequently evaluating its classification performance on both the intra-dataset test set and cross-dataset test set.

Meta-Dataset Construction and Primary Evaluation Metric

The performance results from all pipeline executions were then compiled and aggregated into our final meta-dataset. This compilation was performed separately for the intra-dataset and cross-dataset results, creating two parallel meta-datasets.

While a broad suite of performance metrics was collected (including accuracy, BACC, F1-score, kappa, recall and area under the curve (AUC)). BACC was selected as the primary performance metric for this study. Given the notable class imbalance in some cohorts, the (standard) accuracy seems to be an unreliable metric. Although other robust metrics like the Matthews Correlation Coefficient (MCC) exist,^{39, 40} BACC offers a key advantage for cross-dataset evaluation: it is prevalence-independent, highly interpretable as the average of sensitivity and specificity, and consistently scaled between 0 and 1.⁴⁰ This makes it a more robust and appropriate choice for comparing model generalization performance across heterogeneous datasets with potentially different class distributions.⁴⁰

Statistical Evaluation of Modeling Factors

We first characterized the distribution of ML performances across all tuned models and then selected the appropriate statistics to analyze them. Accordingly, we performed a distributional diagnosis of BACC using the Jarque-Bera test^{41, 42} for formal inference and Quantile-Quantile (Q-Q) plots⁴³ for visual inspection. This diagnosis shows that the ML performances’ distribution significantly deviates from normality.

These results suggest that conventional parametric tests relying on strict normality assumptions (such as classical ANOVA) may yield unreliable inferences. To address this issue, and to ensure robust statistical testing, we employed two complementary approaches: (1) the non-parametric Kruskal-Wallis H-test,⁴⁴ which makes no distributional assumptions and assesses differences in median performance, and (2) a robust ANOVA with heteroscedasticity consistent (HC3) corrections, which remains parametric but adjusts for heteroscedasticity and deviations from normality while evaluating contributions to performance variance. The H-statistics and F-statistics from these two robust methods were then used to rank the global influence of each factor in both the intra-dataset and cross-dataset settings.

Meta-Analysis for Deconstructing Interactions

We applied a SHAP-based meta-analytical framework to the meta-dataset for more granular, context-dependent explanation of performance.^45–47 The XGB model was trained as a ‘meta-model’ to predict a tuned model’s ‘BA_value’ from its constituent modeling factors. XGB was chosen due to its superior ability to capture complex non-linearities and high-order interactions, robust handling of mixed data types, and compatibility with the SHAP framework.^{48, 49}

The SHAP ‘TreeExplainer’ framework was then applied to this meta-model.⁵⁰ This approach provides instance-level attributions (SHAP values) for each factor choice in every tuned model, visualizes the entire distribution of these interaction effects via faceted plots, and can help understand the complete decision logic of modeling strategies with a previously unattainable level of granularity.

A Multi-Criteria Framework for Evaluating Model Trade-offs

Based on the distribution and SHAP analyses, we propose a novel framework that combines both key performance metric (BACC chosen here) and statistical significance in the differences between intra-dataset and cross-dataset testing. It includes three steps, namely baseline performance, generalization ability, and cross-dataset robustness.

1. Step 1: Identify the models with high baseline-performance.

   For each modeling strategy, we conducted multiple independent experiments on both intra- and cross-dataset test sets to obtain a series of BACC values. Five random repeats were conducted for intra-dataset testing of each tuned model and 15 (5 × 3) random repeats for cross-dataset testing of each tuned model in this study. Based on the means of these BACC values, strategies were ranked to identify the high-performing models (e.g., within the top 10 percentile of the ranking) in both intra- and cross-dataset testing. These models were included in the candidate strategy pool and defined as candidate models.

2. 2. Step 2: Assess generalizability of the candidate models and select the most robust strategies.

   The Shapiro-Wilk test^{51, 52} was then employed to assess whether the BACC values of a candidate model obtained from its repeats in the intra- and cross-dataset testing, respectively, followed a normal distribution. If >66% of the candidate models’ repeats fit a normal distribution, an independent Student’s t-test would be used to calculate the p values between the intra- and cross-dataset results for candidate strategies; otherwise, the non-parametric Mann-Whitney U test was employed.^52–54 Only the strategies exhibiting no statistically significant difference (p > 0.05) between the two settings are retained for the final ranking in Step 3.

3. 3. Step 3: Rank selected candidate models by cross-dataset testing performance.

   Final candidates are ranked by their mean BACC on the cross-dataset evaluation. The top-ranked strategy that passes all three stages will be designated as the champion model for its robustness and generalization ability.

Results

Overall Model Performance and Distributional Properties

For LUAD, the 4,200 tuned models yielded a wide spectrum of performance, with BACC fluctuating across a broad range (Supplementary Figure 1). The Jarque-Bera test confirmed significant deviations from normality for both intra-dataset (JB-statistic = 1160.62, p = 9.416E-253) and cross-dataset (JB-statistic = 10838.24, p < 1E-308) results. This was visually supported by Q-Q plots, which, while showing an approximate adherence to normality in the central region, revealed significant deviations in the tails (Figure 2). These findings underscored the necessity of employing robust statistical methods for all subsequent analyses.

Figure 2:

Q-Q plots show the residuals from model performances of intra-dataset and cross-dataset testing in classifying deaths of lung adenocarcinoma and glioblastoma patients, respectively. BA, balanced accuracy; GBM, glioblastoma; LUAD, lung adenocarcinoma.

Similarly, for GBM, the Jarque–Bera test indicated non-normality for both intra-dataset (JB = 22.52, p = 1.288 E-5) and cross-dataset (JB = 217.13, p = 7.088 E-48) results, consistent with the Q–Q patterns in Figure 2.

Statistical Analysis Reveals Different Factor Importance

Robust ANOVA and the non-parametric Kruskal–Wallis test offered complementary perspectives on modeling factor importance insights (Table 1 and Supplementary Figure 2).

Table 1.

Comparison of Factor Importance Rankings from Robust ANOVA and Kruskal-Wallis Test

Evaluation Setting		Modeling Factor	Robust ANOVA		Kruskal-Wallis Test
			F-statistic	p-value	H-statistic	p-value
LUAD	Intra-dataset	NDEG	433.789	<1E-308	1453.109	<1E-308
		DEG	240.334	3.34E-298	1299.939	1.12E-277
		model	346.706	1.60E-289	1025.459	1.08E-220
		target	764.628	2.43E-165	674.217	1.21E-148
		norm_method	121.709	2.10E-127	462.665	9.10E-98
		weights	159.002	2.53E-36	113.601	1.59E-26
	Cross-dataset	model	2142.627	<1E-308	6281.526	<1E-308
		weights	505.046	2.09E-111	216.439	5.41E-49
		DEG	742.865	<1E-308	4279.915	<1E-308
		target	686.834	1.41E-150	83.833	5.38E-20
		norm_method	156.706	4.65E-166	609.466	1.82E-129
		NDEG	57.968	6.43E-49	404.379	3.15E-86
GBM	Intra-dataset	DEG	321.682	<1E-308	1511.533	<1E-308
		NDEG	157.820	1.63E-99	481.439	5.02E-104
		model	57.295	8.98E-48	191.451	2.59E-40
		norm_method	56.312	8.83E-58	240.297	6.63E-50
		target	16.762	4.28E-05	16.851	4.04E-05
	Cross-dataset	DEG	681.202	<1E-308	3647.641	<1E-308
		norm_method	464.307	<1E-308	1963.245	<1E-308
		NDEG	203.583	1.78E-130	653.085	3.12E-141
		model	83.385	1.85E-70	278.130	5.64E-59
		target	71.573	2.81E-17	89.518	3.04E-21

DEG: Differentially Expressed Genes; NDEG: Non-Differentially Expressed Genes; norm_method: normalization method; ANOVA: Applied Analysis of Variance; GBM, glioblastoma; LUAD, lung adenocarcinoma.

The bolded are the top 2 ranked factors.

For LUAD, the results revealed two key insights. First, there were significant differences in modeling factor importance between cross- and intra-dataset testing. In the intra-dataset testing, factors related to model configuration and data processing (namely, `target` or ǸDEG`-based feature selection) were the most influential. In the cross-dataset setting, however, `model` was the overwhelmingly dominant factor. Concurrently, the importance of `target` and ǸDEG` decreased.

Second, the two robust statistical methods yielded slightly different top-ranked factors in the intra-dataset testing (Table 1). Robust ANOVA, which assesses contribution to performance variance,⁵⁵ identified `target` as most influential (F = 764.628), while the Kruskal-Wallis test, which is more sensitive to shifts in the rank distribution,⁴⁴ ranked ǸDEG` as dominant (H = 1453.109). This highlights the limitations of relying solely on high-level importance rankings and motivates our subsequent, deeper investigation using SHAP meta-analysis to deconstruct the underlying mechanisms. Interestingly, the two agreed on the top 2 influencing factors in the cross-dataset testing, including model and DEG. But the 3^rd and 4^th ranked factors differed by the statistical method.

For GBM, by contrast, the two statistical methods showed strong concordance (Table 1). In the cross-dataset setting, both statistical methods highlighted DEG and norm_method as the most influential factors. Their intra-dataset rankings also largely overlapped, underscoring the robustness of these findings. Although there were minor differences in the ranking of factor importance between intra- and cross-dataset settings, DEG consistently ranked first by both statistical methods and testing setings.

Meta-Analysis: Deconstructing the Drivers of Generalizability

To further investigate the differences in factor importance rankings derived from the Robust ANOVA and Kruskal–Wallis tests, we analyzed the SHAP values and their first-order interactions for the top 30 most impactful factors, using LUAD as an example (Figure 3). In both intra- and cross-dataset testing, factor values related to algorithm choice—particularly model_LR and several LASSO-based configurations such as model_LASSO_BACC_W—consistently ranked high. This indicated that, at least in LUAD, the choice of a simpler linear algorithm was a primary determinant of generalization success. Further analysis of the interaction between algorithm choice (model) and the predictive feature space (DEG selection) revealed a robust and stable path to cross-dataset performance: LR with highly sparse DEG sets (e.g., p ≤ 0.01) consistently outperformed other strategies, with LASSO providing comparable stability. These findings reinforce the principle of parsimony in LUAD, where simpler models combined with carefully pruned feature sets reduce overfitting and capture generalizable biological signals.

Figure 3:

Violin plots of top 30 SHAP-ranked factor values across intra-dataset (TCGA) and cross-dataset (OncoSG) testing results in classifying deaths of lung adenocarcinoma patients. SHAP, SHapley Additive exPlanations; TCGA, The Cancer Genome Atlas; OncoSG, the Oncogenomic-Singapore; LUAD, lung adenocarcinoma.

Supplementary Figures 3–8 further illustrate how data processing and model configuration influence model generalizability in LUAD. Unlike the relatively clear advantage of linear models, no single normalization method consistently outperformed the others. As for model configuration, applying class weights to address data imbalance provided minimal benefit, whereas selecting an appropriate optimization target (e.g., ‘BACC’) contributed meaningfully to improving cross-dataset robustness.

Strikingly, a different pattern emerged in GBM experiments. The SHAP analysis fully aligned with both Robust ANOVA and Kruskal–Wallis rankings, further confirming that DEG selection was the most important modeling factor across both intra- and cross-dataset testing. Importantly, the SHAP plots (Supplementary Figures 9) revealed that DEG thresholds at 4% and 10% provided consistent, positive contributions to model performance, serving as two stable critical thresholds for feature-space definition. At the algorithmic level, non-linear models such as MLP and SVM ranked top by performance, surpassing linear models in stability and robustness. This suggests that, unlike LUAD, GBM may require more expressive architectures to capture the complex biological and clinical feature interactions underlying its predictive space.

Taken together, these results demonstrate that, while LUAD benefits most from parsimonious linear models paired with sparse DEG sets, GBM highlights the central role of DEG-based feature-space construction and the need for more flexible, non-linear modeling strategies.

Identifying Elite Strategies with the Multi-Criteria Framework

Finally, we implemented a multi-criteria framework to identify the best-performance yet generalizable strategies for both LUAD and GBM in three steps.

For LUAD, we first applied BA_value-based filtering criteria to 4,200 tuned models. In line with common practices for identifying top contenders in high-throughput exploratory analyses⁵⁶, we created a candidate pool of 99 models that simultaneously ranked within the top 10% by BA_value across intra-dataset and cross-dataset testing. Second, because only 28.5% of the models’ BA_value repeats showed normal distribution (Shapiro–Wilk test, p>0.05) across intra- and cross-dataset settings, we applied the non-parametric Mann-Whitney U test to identify a list of 13 elite models (Table 2), that had no significant performance degradation/differences between intra- and cross-dataset testing (p>0.05). Finally, the list was ranked by the cross-dataset performance.

Table 2.

13 elite strategies selected by the Multi-Criteria Framework on Mann-Whitney U test (LUAD)

Target	model	P threshold for DEG	P threshold for NDEG	Norm-method	weights	Intra-dataset		Cross-dataset		p_value (intra vs cross)
						mean	Std.	mean	Std.
BACC	LR	0.8%	99%	QN	no	0.732	0.119	0.635	0.036	0.116
BACC	LASSO	1%	99%	NPN	yes	0.735	0.153	0.617	0.049	0.190
BACC	LASSO	0. 8%	99%	NPN	yes	0.722	0.158	0.616	0.039	0.662
BACC	LR	0. 1%	98%	binary	yes	0.710	0.091	0.612	0.025	0.055
BACC	LR	0.8%	99.5%	NICG	yes	0.729	0.137	0.611	0.031	0.256
BACC	LASSO	0.7%	98%	binary	yes	0.741	0.102	0.609	0.023	0.055
BACC	LR	0.4%	99%	binary	yes	0.721	0.132	0.605	0.027	0.055
BACC	LR	0.1%	92%	NICG	yes	0.721	0.138	0.603	0.022	0.055
BACC	LR	1%	99%	NPN	yes	0.711	0.125	0.596	0.060	0.055
BACC	LR	0.4%	98%	QN	no	0.717	0.178	0.589	0.063	0.119
BACC	LASSO	0.1%	99%	NPN	yes	0.761	0.162	0.578	0.013	0.055
BACC	LASSO	1%	95%	raw_data	yes	0.717	0.132	0.575	0.020	0.055
BACC	LR	0.4%	98%	QN	yes	0.715	0.105	0.623	0.042	0.097

P threshold for DEG : select the DEGs by P<threshold; P threshold for NDEG: select the NDEGs by P>threshold; Norm-method: Normalization method; Std.: Standard Deviation; Intra-dataset: intra-dataset testing results; Cross-dataset: cross-dataset testing results; p_value (intra vs cross) : p-value from an Mann-Whitney U test on these selected models’ performances across intra-dataset and cross-dataset testing results; BACC: Balanced Accuracy; LR: Logistic Regression; LASSO: Least absolute shrinkage and selection operator; NPN: Non-Parametric Normalization; QN: Quantile Normalization; NICG: Normalization using Internal Control Genes; LUAD: lung adenocarcinoma.

The best/preferred model is highlighted in bold.

We then sought to validate whether these performance-driven selections were aligned with our explainability-based insights. Interestingly, the vast majority of the 13 preferred models shared the core design principles that SHAP had independently identified as drivers of robust generalizability: they predominantly employed LR or LASSO, universally used BACC as the optimization target, and consistently favored highly sparse DEG feature sets. This remarkable alignment provides a powerful cross-validation between our performance-based filtering and our interpretability analysis, confirming the internal consistency and effectiveness of our entire evaluation framework.

For GBM, we evaluated 1,680 tuned models. Based on BACC filtering, we identified a candidate pool of 71 models. Since 81.9 % of the models’ BA_value repeats did not reject the null hypothesis of normality (Shapiro–Wilk test, p > 0.05), we employed an independent-samples Student’s t-test to compare intra- and cross-dataset performance, and finally selected models with no significant cross-dataset performance-degradation (p > 0.05). The 17 elite models were ultimately selected (Table 3). The dominant roles of DEG and NDEG exhibited strong consistency with both the statistical analyses and the SHAP results (Supplementary Figure 9).

Table 3.

17 elite strategies selected by the Multi-Criteria Framework on Independent Student’s t-test (GBM)

Target	model	P threshold for DEG	P threshold for NDEG	Norm-method	Intra-dataset		Cross-dataset		p_value (intra vs cross)
					mean	Std.	mean	Std.
ACC	MLP	4%	99.5%	NICG	0.690	0.111	0.648	0.027	0.446
BACC	MLP	4%	99.5%	NICG	0.710	0.098	0.632	0.029	0.151
BACC	SVM	4%	99%	NPN	0.714	0.103	0.612	0.037	0.092
BACC	SVM	0.8%	99.5%	NICG	0.698	0.109	0.611	0.031	0.148
BACC	LR	4%	99.5%	raw_data	0.684	0.100	0.610	0.039	0.177
ACC	LR	4%	99.5%	NICG	0.705	0.102	0.610	0.031	0.103
ACC	MLP	4%	95%	NPN	0.707	0.104	0.606	0.034	0.096
BACC	LR	4%	98%	raw_data	0.690	0.134	0.599	0.033	0.204
BACC	MLP	4%	95%	NPN	0.691	0.109	0.598	0.034	0.131
BACC	LR	4%	99.5%	NICG	0.695	0.108	0.598	0.030	0.115
ACC	SVM	4%	99%	NPN	0.696	0.089	0.597	0.034	0.067
ACC	SVM	4%	98%	NPN	0.692	0.089	0.596	0.046	0.073
ACC	LR	4%	98%	raw_data	0.685	0.130	0.596	0.025	0.199
BACC	SVM	10%	95%	NICG	0.688	0.089	0.594	0.029	0.077
BACC	MLP	10%	99.5%	NPN	0.705	0.116	0.594	0.022	0.099
ACC	LR	4%	99%	raw_data	0.705	0.118	0.592	0.019	0.099
BACC	SVM	4%	99.5%	NICG	0.710	0.150	0.592	0.037	0.155

P threshold for DEG : select the DEGs by P<threshold; P threshold for NDEG: select the NDEGs by P>threshold; Norm-method: Normalization method; Std.: Standard Deviation; Intra-dataset: intra-dataset testing results; Cross-dataset: cross-dataset testing results; p_value (intra vs cross) : p-value from an independent-sample Student’s t-test on these selected models’ performances across intra-dataset and cross-dataset testing results; BACC: Balanced Accuracy; ACC: standard Accuracy; LR: Logistic Regression; MLP: Multilayer Perceptron; SVM: Support Vector Machine; NPN: Non-Parametric Normalization; NICG: Normalization using Internal Control Genes; GBM: glioblastoma.

The best/preferred model is highlighted in bold.

Discussion

By systematically evaluating two independent examples (4,200 LUAD and 1,680 GBM tuned models) and applying a SHAP-based meta-analytical framework, we demonstrate that the “important” modeling factors shift dramatically when moving from intra-dataset to cross-dataset testing. This analysis shows that a model’s cross-dataset performance is not always determined by fixed ranking of modeling choices, but by the interplay among factors under varying data or testing contexts. This insight challenges conventional model development and evaluation practices and motivates us to establish a new, integrated framework for building the best generalizable cross-dataset ML models from real-world, heterogeneous data.

This study seems to have several scientific contributions. First, we challenge the conventional reliance on intra-dataset testing with robust empirical evidence for the “under-specification” phenomenon in bioinformatics, demonstrating that high in-distribution performance is a poor proxy for out-of-distribution (e.g., cross-dataset) generalizability.⁵⁷

Second, we are the first, to our knowledge, to systematically examine the ML performance distributions arising from different modeling factors in a meta-analytical context. After confirming their non-normal nature, we employed a robust toolkit of parametric and non-parametric tests,⁵⁸ to credibly establish that the relative importance of different modeling factors shifts dramatically depending on the evaluation context (i.e., intra-dataset vs. cross-dataset).

Third, inspired by recent works on explaining hyperparameter optimization,^{45–47, 59} we pioneer a novel meta-analytical application paradigm using SHAP.⁴⁹ While traditional tests ranked factor importance, they failed to explain why a model succeeded or failed under specific conditions.⁶⁰ To address this failure, our approach can attribute a model’s cross-dataset generalization performance to the full spectrum of upstream modeling decisions, including algorithm choice, feature selection, and data normalization.⁶¹ It also appears to move significantly beyond prior analyses that mostly focused on single-model hyperparameters.⁵⁹

Fourth, the convergence of statistical and SHAP analyses shows that the optimal modeling strategy appears cancer-type dependent. In LUAD, the choice of algorithm was decisive. Simple linear models (e.g., LR and LASSO) combined with sparse predictive feature sets consistently outperformed complex non-linear models in cross-dataset generalization in LUAD experiments. By contrast, feature-space definition emerged as dominant in GBM experiments. The use of DEG-based feature sets (particularly at 4% and 10% thresholds) showed the strongest contributions to performance. Interestingly, this differs from our prior breast cancer study using paired RNA-seq and microarray data, in which the normalization method itself played a substantial role.²⁵ Together, while ML model (e.g., linear versus non-linear), feature selection, and normalization are all critical, the key factor(s) associated with better generalizability may differ by disease context and/or dataset characteristics.

Finally, based on these insights, we propose a novel, multi-criteria framework for cross-dataset model selection that achieves an optimal balance between performance, cross-dataset generalization, and decision logic stability. In this framework, selection of the appropriate statistical test for evaluating model generalizability should be based on the model-performance distribution (normal vs not). Our empirical comparison (Tables 2 versus Supplementary Table 2, and Tables 3 versus Supplementary Table 3) revealed greater consistency in the results of the two tests for GBM. This aligns with statistical literature indicating that parametric and non-parametric tests yield similar outcomes when data approximate normality.^{24, 52–54} Although established statistical principles indicate that parametric tests (e.g., Student’s t-test) are more powerful under normality than non-normality, non-parametric tests (e.g., Mann-Whitney U) offer robustness when normality assumptions are violated.^{52–54, 62, 63} We found that the best models selected by both parametric and non-parametric tests were identical in both LUAD and GBM experiments. This consistency further validates the robustness of our adaptive testing strategy. This framework thus appears to offer a practical pipeline for developing the best generalizable ML models in cross-dataset setting.

While this study provides a generalizable methodology and actionable insights in cross-dataset ML performance, it has several limitations. First, our analysis was conducted on LUAD and GBM datasets, and extending this work to additional cancer/disease types and across more diverse populations will be critical to assess its generalizability. Second, while our modeling space was broad, our works were limited to commonly used ML models. Therefore, future work should incorporate more theoretically robust algorithms such as invariant risk minimization⁶⁴ and Group Distributionally Robust Optimization⁶⁵, as well as modern approaches including self-supervised models (e.g., contrastive learning⁶⁶) and foundation models (e.g., large-scale pre-trained omics models⁶⁷). Third, because DEGs were selected based on ANOVA’s p-value thresholds and the training sets varied in each experiment, the resulting gene lists varied to some extent. Thus, future works are warranted to investigate the impact of a fixed set of genes versus dynamically selected genes on model performance, and their clinical implications. Finally, we focused on BACC as the performance metric. Incorporating other complementary metrics such as accuracy could provide a more comprehensive evaluation,⁶⁸ though additional performance metric(s) may also complicate or skew the evaluation/ranking. Despite these limitations, the framework we propose is readily transferable to other omics domains and disease types, providing a simple yet robust path toward the best generalizable ML models.

Through systematic, multi-level analyses of ML performances in classifying survivals of two cancer/disease types, we provide new empirical evidence for understanding and optimizing the cross-dataset generalization of ML models on classifying cancer survival using transcriptomic and clinical data. We also developed and validated a multi-criteria framework for model selection that offers a transparent, data-driven approach to identify models that are not just accurate, but consistently robust cross datasets. Our data suggest that researchers should move beyond the sole pursuit of high performance within a single, intra-dataset testing context, and also focus on generalizability, explainability, and cross-dataset performance of ML modeling.

Data Availability Statement

The data sets used and/or analyzed of this study are available on the cBioPortal website (https://www.cbioportal.org/). The program coding is available from the corresponding authors on reasonable request.