Benchmarking computational methods for multi-omics biomarker discovery in cancer

Abstract

Multi-omics profiling characterizes cancer biology and supports biomarker discovery for prognosis and therapy selection. Although numerous computational multi-omics biomarker identification methods have been proposed, their ability to identify clinically relevant biomarkers has not been systematically evaluated, leaving it unclear whether the resulting biomarker nominations are reliable for downstream validation. Here, we systematically benchmark 20 representative statistical, machine learning and deep learning methods using curated gold-standard prognostic and therapeutic biomarkers across five real-world datasets. We evaluate performance in terms of both biomarker identification accuracy and stability. Overall, DeePathNet and DeepKEGG achieve the best performance. Across methods, effective biomarker recovery is associated with the integration of biological knowledge, global feature interactions, multivariate feature attribution, and effective regularization. Analysis of omics type contributions reveals method- and modality-specific biases, highlighting the importance of broader omics integration. We further evaluate methods on simulated datasets to probe sensitivity with controlled signal and noise. By aggregating results from top-performing methods, we construct consensus biomarker panels that nominate candidates for potential investigations. Finally, we provide user-friendly interfaces to allow researchers to benchmark new methods against the 20 baselines or apply selected methods for biomarker identification on custom multi-omics datasets. Our benchmark is publicly available at https://github.com/athanzli/CancerMOBI-Bench.

Introduction

High-throughput technologies have enabled large-scale profiling of human cancers across genomic, transcriptomic, epigenomic, proteomic, and metabolomic layers [1, 2]. Collectively, these assays constitute a comprehensive and complementary view of tumor biology, substantially expanding the landscape for discovering biomarkers that can inform prognosis and therapy selection from a vast space of molecular candidates [3]. Nevertheless, high dimensionality, tumor heterogeneity, and complex cross-omics dependencies make robust and clinically meaningful biomarker discovery inherently challenging [4].

Over the past several years, a diverse array of computational methods has been developed to address this need (Supplementary Table 1). Statistical and machine learning (ML) methods extend classical frameworks such as matrix factorization, canonical correlation analysis (CCA), or uniform manifold approximation and projection (UMAP) to extract shared patterns across omics and derive interpretable latent representations [5–7]. More recently, deep learning (DL) architectures, including feedforward neural networks (FNN), autoencoders, graph neural networks, and transformers, have been applied for multi-omics fusion by learning the nonlinear and hierarchical relationships across molecular layers [8–15]. These methods have achieved strong performance on a broad spectrum of cancer-related tasks, including prediction of cancer subtypes, metastatic status, pathological stages, survival outcomes, and therapeutic responses [8–10, 14–16]. Beyond predictions, many incorporate interpretability modules to identify biologically meaningful features, such as disease-associated subnetworks [17], dysregulated pathways [15], and most commonly, molecular biomarkers [7, 9–16, 18–21].

Despite the methodological progress, systematic evaluations of biomarker identification performance remain limited. Within individual method studies, evaluations of discovered biomarkers are typically proxy-based or qualitative, relying on functional enrichment analysis or literature support [7, 9–16, 18–21]. Such assessments are inferential and prone to selection bias, providing limited quantitative validation. Also, existing benchmark studies often rely on predictive performance as a proxy for biomarker identification performance [22–32]. However, high predictive accuracy does not necessarily imply reliable biomarker identification, as models can achieve strong prediction using features that are statistically discriminative but not biologically meaningful, particularly in high-dimensional, correlated multi-omics data [33]. Although some utilize synthetic datasets with predefined discriminative features for direct and quantitative assessment, these datasets are often simplistic and fail to match the complexity and heterogeneity of real-world cancer data [6, 26, 28, 34, 35]. Moreover, many recent multi-omics and DL methods remain absent from prior benchmarks [22, 24, 25, 29–32, 34, 36, 37].

Translational implications can arise from these gaps. As biomarker development proceeds from early discovery toward tests intended for specific clinical applications, successful translation requires analytical validity, clinical validity, and clinical utility [38, 39]. Because <1% of published cancer biomarkers enter clinical practice, more rigorous evaluation of discovery-stage methods is needed to better prioritize approaches with downstream translational potential [40].

To address these limitations, we provide a systematic benchmark of computational methods for multi-omics biomarker identification. A major obstacle has been the lack of task-specific biomarker sets for ground-truth evaluation. To address this, we collected Tier I biomarkers as defined by AMP/ASCO/CAP guidelines [41] from Clinical Interpretation of Variants in Cancer (CIViC) [42], MSK’s Precision Oncology Knowledge Base (OncoKB) [43], and Cancer Genome Interpreter (CGI) [44]. Using these gold-standard biomarkers, our benchmarking enables direct, quantitative, and clinically meaningful assessment of biomarker identification performance. Instead of relying on indirect evaluation through predictive performance, our approach directly quantifies each method’s ability to recover curated, clinically relevant biomarkers via rank-based metrics. We constructed five real-world task datasets from the cancer genome atlas (TCGA) [45] through systematic filtering to achieve both computational feasibility and alignment between biomarkers and tasks.

In total, we benchmark eight statistical and ML methods and twelve DL methods, covering a wide range of representative modeling and feature-identification approaches. Performance was evaluated across diverse experimental settings and metrics. Several methods exhibit strong ability to recover clinically validated biomarkers, frequently surfacing gold-standard biomarkers among the top ranks across experiments. Analysis of omics type contributions further characterizes biomarker- and method-specific biases. Transformer-based methods integrating biological pathways with advanced post hoc feature-attribution methods achieve the highest accuracy and strong stability, whereas univariate feature-ablation methods underperform on both dimensions. Simulation experiments show that statistical and ML methods are more sensitive to shifts in the discriminative signals of ground-truth features, but discrepancies with real-data results underscore the indispensability of real-world cancer cohorts. Additionally, we derive consensus panels of multi-omics cancer biomarkers by aggregating the results of top-performing methods, providing candidates for potential downstream investigations. Our benchmark is publicly available with user-friendly evaluation pipelines that enable researchers to benchmark new methods against the 20 baselines on the curated task datasets, or to apply selected methods to other multi-omics data for biomarker identification.

Results

Benchmarking design and evaluation framework

We conducted a systematic benchmark of 20 computational methods for multi-omics biomarker identification, including 8 statistical and ML methods and 12 DL methods selected from literature (Supplementary Table 1) that encompass diverse methodological designs (Fig. 1c). Their model families, multi-omics fusion strategies, use of prior biological knowledge, and biomarker identification approaches are summarized in Table 1, with detailed descriptions provided in Supplementary Methods.

Figure 1.

Overview of benchmarking workflow. (a) Biomarkers were collected from three oncology knowledge bases, including CIViC [42], OncoKB [43], and CGI [44], and those with Tier I evidence determined by AMP/ASCO/CAP guidelines [41] were retained. (b) Real-world data included 30 multi-omics data subsets encompassing five tasks and six omics combinations. Tumor types and drugs for prognostic and therapeutic tasks were matched to available biomarkers, and determined through a series of filtering rules (Supplementary Methods). Real-world data were used as statistical reference for simulated multi-omics data generation with InterSIM [49]. (c) Twelve DL methods were benchmarked with different architectural backbones, including FNN, autoencoder, graph, and transformers. Eight statistical and ML methods were also benchmarked. (d) Rank-based metrics for performance quantification. Six accuracy metrics and three stability metrics were used (Methods).

Table 1.

Overview of the computational multi-omics biomarker identification methods benchmarked in this study

Name	Model	Multi-omics fusion	Prior knowledge	Supervision	Biomarker identification	Ante hoc/ Post hoc	Year
DL
P-Net [16]	Sparse neural network	Early	Biological pathways	Supervised	DeepLIFT	Post hoc	2021
GENIUS [20]	CNN	Early	Physical genomic positioning	Supervised	IGs	Post hoc	2023
TMO-Net [8]	Variational autoencoder	Intermediate	None	Supervised	IGs	Post hoc	2024
CustOmics [9]	Variational autoencoder	Mixed	None	Supervised	SHAP	Post hoc	2023
MOGONET [10]	GCN	Late	None	Supervised	Feature ablation	Post hoc	2021
MoAGL-SA [19]	GCN and self-attention	Intermediate	None	Supervised	Feature ablation	Post hoc	2024
MORE [18]	Hypergraph and self-attention	Intermediate	None	Supervised	Feature ablation	Post hoc	2024
MOGLAM [12]	GCN and self-attention	Intermediate	None	Supervised	Feature weights	Ante hoc	2023
GNN-SubNet [11]	Graph isomorphism network	Early	PPI network	Supervised	GNNExplainer	Post hoc	2022
Pathformer [15]	Transformer encoder	Early	Biological pathways	Supervised	SHAP	Post hoc	2024
DeePathNet [14]	Transformer encoder	Early	Biological pathways	Supervised	SHAP	Post hoc	2024
DeepKEGG [13]	Transformer encoder	Intermediate	Biological pathways	Supervised	DeepLIFT	Post hoc	2024
Statistical and ML
MCIA [46]	Matrix decomposition	Intermediate	None	Unsupervised	Latent component weights	Ante hoc	2014
MOFA [5]	Matrix decomposition	Intermediate	None	Unsupervised	Latent component weights	Ante hoc	2018
GAUDI [7]	UMAP transformations and XGBoost/RF	Intermediate	None	Unsupervised	SHAP	Post hoc	2025
DIABLO [6]	Sparse generalized CCA	Intermediate	None	Supervised	Latent component weights	Ante hoc	2019
asmbPLS-DA [47]	Sparse PLS-DA	Intermediate	None	Supervised	Latent component weights	Ante hoc	2023
Stabl [35]	Sparse regularized model	Intermediate	None	Supervised	Max selection frequency	Ante hoc	2024
GDF [17]	Greedy decision forest	Early	PPI network	Supervised	Gini gain	Ante hoc	2022
DPM [48]	Directional -value merging	Intermediate	Inter-omics regulatory directions	Supervised	Merged -values	Ante hoc	2024

PPI, protein–protein interaction; PLS-DA, partial least-squares discriminant analysis; RF, random forest; CNN, convolutional neural network; GCN, graph convolutional network; IG, integrated gradient.

To anchor evaluation in clinical relevance, we curated Tier I cancer biomarkers, as defined by AMP/ASCO/CAP guidelines [41]. These biomarkers were harmonized across CIViC [42], OncoKB [43], and CGI [44] (Fig. 1a, Supplementary Tables 2 and 3). After filtering, the curated biomarkers were mapped to five benchmark task datasets constructed from TCGA, including survival risk classification for Breast invasive carcinoma (BRCA), Lung adenocarcinoma (LUAD), and Colon and Rectum adenocarcinoma (COADREAD), and drug response prediction for Cisplatin with bladder urothelial carcinoma (BLCA) and Temozolomide with brain lower grade glioma (LGG) (Fig. 1b, Supplementary Methods, Table 2). The curated biomarker set spans multiple clinically established alteration classes, including sequence variants, copy-number alterations, expression-based markers, DNA methylation markers, gene fusions, and limited protein-level markers (Supplementary Table 3), capturing major mechanisms with established clinical significance such as oncogenic driver activation, tumor suppressor inactivation, DNA-repair deficiency, epigenetic silencing, and immune-related expression states [41–44].

Table 2.

Summary statistics of the real datasets used in this study.

Task	Dataset	Sample size				No. features		No. biomarkers
		Low risk	High risk	CNV	DNAm	mRNA	miRNA	SNV
Survival	BRCA	324	323	18 645	46 859	19 504	1581	15 118	14
	LUAD	212	211	18 645	46 859	19 497	1581	17 103	14
	COADREAD	187	185	18 645	46 859	19 471	1555	18 394	12
		Response	Non-response
Drug response	Cisplatin (BLCA)	40	20	18 645	46 859	19 288	1404	10 502	1
	Temozolomide (LGG)	20	103	18 645	46 859	19 411	1392	3656	2

Since the benchmarked methods employed different omics combinations in their original case studies (Supplementary Table 4), we defined a unified set of omics combinations to achieve comparability across methods. For each task, we combined messenger RNA (mRNA) expression with two additional omics types from microRNA (miRNA), DNA methylation (DNAm), copy number variation (CNV), and single-nucleotide variation (SNV). This produced six distinct omics combinations per task, leading to 30 multi-omics subdatasets that were each evaluated with five-fold cross-validation (Fig. 1b). To complement real-world data, we also generated simulated multi-omics data using InterSIM [49]. These simulations contain predefined discriminative features serving as ground truth, enabling controlled assessment of method sensitivity across a range of signal levels (Fig. 1b, Supplementary Table 5, Supplementary Methods).

Biomarker identification performance was evaluated using nine complementary metrics capturing different aspects of accuracy and stability (Fig. 1d, Methods). For real-world cohorts where ground truth biomarkers are partially known, rank-based metrics were applied to reward the prioritization of validated biomarkers without penalizing unvalidated candidates. For simulated datasets with complete ground truth, evaluation adopted metrics such as AUROC and accuracy. Stability was quantified as the similarity of ranked gene lists across cross-validation, using metrics that reflect top-rank consistency, global ranking agreement, and biomarker ranking variability (Methods). For comparability, all method output was converted to gene-level ranking lists prior to metric calculation (Methods).

Accuracy on real-world data

Accurate identification of clinically relevant candidates from high-dimensional molecular space represents a hallmark of an effective biomarker discovery method. Therefore, we first examined the performance of each method for the recovery of clinically validated biomarkers in real-world cancer cohorts. Across five TCGA-derived tasks and six omics combinations, we compared methods using rank-based metrics that quantify different aspects of biomarker prioritization: average recall (AR) for overall prioritization, reciprocal rank (RR) for early retrieval of a single biomarker, and NDCG for early recovery of multiple biomarkers. These were complemented by a Mann–Whitney U test on global rankings and by success counts within the top 10, 50, and 100 ranks (Fig. 2a and b, Table 3, Methods). Detailed results of each omics combination are reported in Supplementary Figs 1–5 and 11–15.

Figure 2.

Overall performance on real-world data. (a) Accuracy for all tasks and metrics. For a particular method and task, the height of the bar represents the averaged metric value of the cross-validation mean across all six omics combinations. (b) Mann–Whitney U test P-values (negative -transformed). Each data point represents a single experiment corresponding to a specific omics combination and cross-validation fold. The vertical dashed line represents a -value of . (c) Stability for all tasks and metrics. For a particular method and task, the height of the bar represents the averaged metric value across all six omics combinations. NDCG: normalized discounted cumulative gain. “-” indicates unavailability due to lack of biomarkers in the method’s prior knowledge gene set.

Table 3.

The number of times a method ranks at least one gold-standard biomarker within the top 10, 50, or 100 ranks out of all the 30 experiments (including six omics combinations with five-fold cross-validation).

Method	Survival BRCA			Survival LUAD			Survival COADREAD			Drug response Cisplatin (BLCA)			Drug response Temozolomide (LGG)			Total
	Top 10	50	100	Top 10	50	100	Top 10	50	100	Top 10	50	100	Top 10	50	100	Top 10	50	100
DL
DeePathNet	21	29	29	6	23	28	17	22	26	–	–	–	–	–	–	44	74	83
DeepKEGG	10	20	22	15	22	23	17	26	26	0	0	2	4	7	9	46	75	82
MOGLAM	10	10	12	10	10	10	10	10	10	0	0	0	10	10	12	40	40	44
TMO-Net	15	15	15	6	10	15	7	11	12	0	0	1	11	12	13	39	48	56
CustOmics	7	13	15	0	1	3	0	1	3	0	0	0	9	13	13	16	28	34
GENIUS	7	11	11	3	6	8	6	11	14	0	0	0	8	8	8	24	36	41
Pathformer	6	10	10	7	12	14	8	11	11	1	2	3	0	2	2	22	37	40
GNN-SubNet	3	3	4	1	2	5	1	4	6	0	0	0	1	3	4	6	12	19
P-Net	0	1	1	0	0	0	0	0	2	0	0	0	0	0	0	0	1	3
MOGONET	7	12	15	0	2	3	1	4	4	0	0	0	2	3	3	10	21	25
MORE	0	2	2	0	1	1	1	2	4	0	2	2	0	0	0	1	7	9
MoAGL-SA	1	1	2	1	1	2	0	0	0	0	0	0	0	0	0	2	2	4
Statistical and ML
DIABLO	15	15	15	8	9	9	7	8	10	0	0	0	15	15	15	45	47	49
GAUDI	15	15	15	7	11	12	1	5	10	0	0	0	13	15	15	36	46	52
GDF	1	19	23	1	11	19	0	7	14	0	1	1	0	0	0	2	38	57
Stabl	18	23	24	0	6	12	0	11	18	0	0	0	1	3	7	19	43	61
asmbPLS-DA	6	12	16	0	0	1	2	11	16	0	0	1	0	1	1	8	24	35
MOFA	0	0	0	0	5	9	0	0	0	0	0	0	0	0	0	0	5	9
DPM	0	2	3	0	1	3	0	0	0	0	1	2	0	0	0	0	4	8
MCIA	0	0	0	0	0	1	0	0	0	0	0	0	0	1	1	0	1	2

“–” indicates unavailability due to lack of biomarkers in the method’s prior knowledge gene set.

We identified a group of relatively accurate methods, including DeePathNet, DeepKEGG, MOGLAM, TMO-Net, DIABLO, and GAUDI, which frequently recovered validated biomarkers among the top ranks, with a top-10 success rate of around 26.7% (40 out of 150 experiments; Table 3) as well as comparable overall accuracy (Fig. 2a and b). Within this group, the transformer-based, pathway-informed methods DeePathNet and DeepKEGG were top performers. In survival prediction tasks, DeePathNet and DeepKEGG recovered a validated biomarker within the top 10 in around half of all experiments (44/90 and 42/90, respectively) and achieved the highest NDCG scores, indicating strong ability to surface multiple biomarkers simultaneously. Expanding the ranking threshold from top-10 to top-100 further accentuates this advantage, increasing the success rate of DeePathNet and DeepKEGG to 50% across all tasks and 90% in survival tasks, whereas MOGLAM, TMO-Net, DIABLO, and GAUDI showed minimal improvement (Table 3). This divergence aligns with the omics contribution analysis (Fig. 3), suggesting that DeePathNet and DeepKEGG exploit a broader spectrum of omics combinations to retrieve relevant signals.

Figure 3.

Omics type contributions in biomarker identification. (a) Pie charts showing the omics types driving the identification of a biomarker. Each pie has 30 portions, with each representing a single experiment with a specific cross-validation fold (five in total) and omics combination (six in total). A portion is colored by the dominant omics type when the biomarker is ranked in the top 1% (Methods), and left gray otherwise. (b) Omics type proportions (y-axis) from high (left) to low (right) ranks (x-axis). Each task contains six columns corresponding to the six omics combinations. Proportions were calculated at each of the 1000 rank percentile cutoffs. The ranks were based on the raw feature scores before gene-level conversion (Methods), which were concatenated from the five score lists output by cross-validation.

At the lower end of performance, methods relying on feature ablation (MOGONET, MoAGL-SA, and MORE) rarely recovered biomarkers across both success counts and accuracy metrics (Table 3, Fig. 2a and b). Their underperformance likely reflects the complementarity among correlated multi-omics features, where ablation of a single feature is compensated by others [33]. Among statistical and ML methods, MOFA, DPM, and MCIA consistently ranked lowest across tasks and metrics. DPM is intrinsically univariate, while MOFA and MCIA rank features via unsupervised matrix decomposition that may not align with clinical endpoints (Supplementary Methods). In contrast, GAUDI, despite also being unsupervised, achieved substantially better performance by combining tree-based models with SHAP-based attribution, highlighting the benefit of multivariate attribution even in unsupervised settings (Fig. 2a and b).

Deep neural networks are not intrinsically superior to conventional statistical and ML models. Although DL methods exhibit a higher performance ceiling, with DeePathNet and DeepKEGG achieving the highest accuracy and maintaining it across multiple cancers and omics combinations (Supplementary Figs 1–5), well-designed statistical methods such as DIABLO and GAUDI achieved competitive accuracy. Conversely, Pathformer achieved only moderate accuracy despite using a transformer backbone, and MOGLAM was the strongest graph-based method despite most graph methods ranking low. These contrasts indicate that specific methodological designs, rather than model family, primarily determine success. The factors underlying these differences are systematically analyzed in the following sections.

Task complexity also affects performance. Almost every method underperformed on the BLCA cisplatin drug–response prediction task, particularly on top-rank-biased metrics such as NDCG and RR, indicating the increased difficulty of surfacing ERCC2 relative to other targets (Fig. 2a, Supplementary Figs 4 and 14). This likely stems from two factors. Clinically, ERCC2’s predictive signal is specific to cisplatin-based neoadjuvant chemotherapy with pathologic downstaging endpoints, hence mixing metastatic settings, radiographic endpoints, or carboplatin treatments introduces label noise [50]. Biologically, only helicase-domain loss-of-function ERCC2 variants truly abrogate nucleotide-excision repair and confer cisplatin sensitivity, thus collapsing all ERCC2 mutations into a single feature further obscures the signal [51]. Nevertheless, Pathformer appeared as a notable exception, as it was the only method consistently identifying ERCC2 within the top ranks when using the mRNA+CNV+SNV omics combination (Supplementary Figs 4 and 14). Relative to other transformer-based methods, Pathformer employs a more expressive criss-cross attention mechanism and architectural design (Supplementary Methods). This suggests that more sophisticated architectures can extract weak signals from noisy data under certain omics combinations, though such advantages may trade off against performance on less difficult tasks.

Stability on real-world data

The reproducibility of biomarkers is essential for clinical translation, and high stability is a prerequisite for reproducible biomarker discovery across diverse cohorts. We quantified stability using the five cross-validation folds per omics combination per task, averaging the pairwise similarities between gene ranking lists (Methods). Rank-biased overlap (RBO) assesses top-rank consistency, rank percentile standard deviation (RPSD) measures the dispersion of biomarker rankings across splits, and Kendall’s quantifies the overall consistency across full rankings (Fig. 2c, Methods). Stability results of each omics combination are reported in Supplementary Figs 6–10.

In general, we found that statistical and ML methods were markedly more stable than DL methods, whereas the latter exhibited greater variability, encompassing both the most and least stable methods (Fig. 2c). A subset of DL methods, including MOGLAM, Pathformer, and DeePathNet, surpassed all statistical and ML methods, achieving both high top-rank stability (RBO) and strong global stability (low RPSD and high Kendall’s ). By contrast, DIABLO, which is the most stable statistical and ML method, reached the best global stability but failed to attain comparable RBO, suggesting that although its rankings are internally coherent, they are less consistent at top than those produced by the most stable DL methods. At the opposite extreme, several DL methods, including P-Net, GENIUS, GNN-SubNet, MOGONET, MoAGL-SA, and MORE, were less stable than the least stable statistical and ML methods. Their RBO scores fall below that of asmbPLS-DA, the least stable statistical and ML baseline at the top ranks, and their global stability was lower than that of GDF, which had the highest RPSD and lowest Kendall’s among statistical and ML methods (Fig. 2c). These results reflect the sensitivity of deep neural network training, as small changes in the input samples can drive stochastic gradient-based optimization into distinct solution basins [52], producing substantially different top-ranked features identified by post hoc attribution methods.

Among the most stable DL methods, incorporating biological pathway structures (Pathformer, DeePathNet, and DeepKEGG) and employing explicit regularization (MOGLAM) appeared to improve stability, while DL methods lacking such design choices tended to be among the least stable (Fig. 2c).

Jointly considering all methods, those that fail to capture global feature dependencies (GNN-SubNet, P-Net, and GENIUS) were generally less stable and also underperformed in accuracy, suggesting that insufficient modeling of long-range molecular interactions diminishes the stabilizing contribution of global context [53] (Fig. 2c).

Determinants of biomarker identification performance

Joint analysis of accuracy and stability revealed substantial variation across the benchmarked methods. Autoencoder-based methods such as TMO-Net and CustOmics achieved strong accuracy but were unstable, Pathformer exhibited high stability but only moderate accuracy, and conventional statistical methods including MCIA and MOFA were inaccurate but highly stable. Nevertheless, high accuracy and stability are achievable simultaneously: MOGLAM, DeePathNet, DeepKEGG, and DIABLO were both accurate and stable, whereas P-Net, GNN-SubNet, MOGONET, MORE, MoAGL-SA, and DPM underperformed on both dimensions (Fig. 2). These contrasts indicate that the decisive factor is specific methodological design rather than model family. Across the diverse tasks and omics combinations evaluated, four methodological factors consistently distinguished high- from low-performing methods.

Feature attribution

The approach used to infer feature importance emerged as a primary performance determinant. Methods employing multivariate attribution, including SHAP (DeePathNet, Pathformer, CustOmics, and GAUDI), DeepLIFT (DeepKEGG and P-Net), IGs (TMO-Net and GENIUS), GNNExplainer (GNN-SubNet), and ante-hoc feature weighting (MOGLAM, DIABLO, asmbPLS-DA, and Stabl), consistently outperformed those relying on univariate feature ablation (MOGONET, MoAGL-SA, and MORE) or per-feature statistical tests (DPM) (Fig. 2a and b, Table 3). In high-dimensional multi-omics data, features are correlated both within and across molecular layers. Univariate approaches underestimate the importance of features whose effects are distributed across correlated molecular partners, because removing or testing a single feature can be compensated by its correlated counterparts [33]. Multivariate methods evaluate each feature’s contribution in the context of others, capturing the joint dependency structure essential for identifying clinically relevant biomarkers. The importance of this factor is also illustrated by GAUDI, which demonstrates that combining tree-based models with SHAP can compensate for the absence of supervised labels, outperforming several supervised DL methods that rely on weaker attribution approaches (Fig. 2a and b).

Integration of biological knowledge

Encoding biological pathway structures into model architectures was associated with high accuracy and stability. The two top-performing methods, DeePathNet and DeepKEGG, encode pathway knowledge into transformer backbones (Table 1, Supplementary Methods). Pathway-informed architectures constrain the hypothesis space to biologically plausible feature groups, reducing overfitting, and directing the model toward functionally coherent molecular associations. However, knowledge encoding alone is insufficient. P-Net also incorporates pathways but underperforms due to sparse connectivity that suppresses inter-pathway communication, and GNN-SubNet uses PPI networks but is limited by local message-passing (Supplementary Methods). These contrasts indicate that the effectiveness of prior knowledge depends on the architecture’s capacity to propagate learned representations across the encoded biological structures.

Scope of feature interactions

Methods modeling long-range dependencies across the feature space exhibited markedly higher accuracy and stability than those restricted to local interactions. Transformer self-attention enables pathways to attend to one another, capturing cross-pathway dependencies that reflect the interconnected nature of biological processes. In contrast, CNNs (GENIUS) are constrained by local receptive fields, sparse networks (P-Net) suppress inter-node communication, and graph message-passing (GNN-SubNet) attenuates with distance [53]. Among statistical methods, DIABLO’s generalized CCA models global cross-omics covariance, contributing to its competitive performance (Supplementary Methods). These observations indicate that when biomarker-associated features span multiple interacting molecular layers, architectures restricted to local interactions cannot adequately capture the cross-layer dependencies required for accurate and stable biomarker identification (Fig. 2).

Regularization

Regularization improved stability without compromising accuracy. MOGLAM learns graphs adaptively from the data and applies regularization on its feature-indicator matrix, yielding the highest stability among methods (Fig. 2c, Supplementary Figs 6–10). DL methods without explicit regularization (P-Net, GENIUS, GNN-SubNet, MOGONET, MoAGL-SA, and MORE) exhibited low stability, reflecting the sensitivity of stochastic optimization to training set perturbations [52]. The co-occurrence of high accuracy and stability in methods such as MOGLAM, DIABLO, and Stabl demonstrates that these objectives are simultaneously achievable through appropriate regularization (Fig. 2).

These four factors interact synergistically. DeePathNet and DeepKEGG combine all four (pathway-informed transformer architectures with global self-attention, multivariate attribution, and implicit regularization via structured pathway encodings), achieving the highest overall performance (Fig. 2, Table 3). Methods lacking multiple factors (e.g. MOGONET, MoAGL-SA, and MORE with local graph interactions, univariate ablation, and no prior knowledge) consistently show weak performance. This indicates that strong biomarker identification is not driven by a single design choice, but instead emerges from the combination of multiple favorable design principles.

Contribution of omics types to biomarker identification

Different molecular layers characterize complementary aspects of tumor biology, but their relative contributions to biomarker discovery remain unclear. Although most methods evaluated one or two omics combinations in their original case studies (Supplementary Table 4), a systematic comparison across a broader set of omics combinations is essential for uncovering the underlying biological mechanisms and informing both method development and experimental design. Consistent with this need, we observed substantial differences in performance across omics combinations (Supplementary Figs 1–15). In many cases, the omics combinations chosen in the original studies did not yield superior performance compared with untested alternatives, supporting our inclusion of an extended set of omics combinations. We also found that for almost all methods, high accuracy was confined to a specific omics combination, with DeePathNet and DeepKEGG being notable exceptions. In survival tasks, DeePathNet and DeepKEGG identified biomarkers within the top 10 ranks across nearly all omics combinations (RR ; Supplementary Figs 1c, 2c, and 3c). To further dissect these results, we analyzed omics type contributions at both the biomarker and method level.

To pinpoint which omics types drive biomarker discovery, we examined the contributions of all five omics types for each identified biomarker (Fig. 3a). Here we define a biomarker as “identified” when it appeared within the top 1% of the gene-level ranking list (Methods). For each such case, we recorded the omics type with the highest feature score (Methods) as dominant. Figure 3a provides a comprehensive summary for all experiments. Detailed results of each omics combination are reported in Supplementary Figs 16–21.

We found that most identified biomarkers were driven by one predominant omics layer. For instance, for the temozolomide drug response prediction task within LGG, the top-performing experiments consistently required SNV data. IDH1 was typically ranked at the top owing to SNV (Fig. 3a, Supplementary Figs 17, 19, and 21), consistent with the causal IDH1 R132 mutation and its co-occurrence with MGMT promoter methylation, the canonical predictor of temozolomide benefit [54]. In breast cancer (BRCA) survival, TP53 was almost always recovered through SNV (Fig. 3a, Supplementary Figs 17, 19, and 21), reflecting the prevalence of missense mutations in the DNA-binding domain that stratify poor outcomes, which is especially common in aggressive basal-like/TNBC [55, 56]. By contrast, FCGR2B was primarily identified through mRNA expression, particularly by Stabl and asmbPLS-DA (Fig. 3a, Supplementary Figs 16–21), where transcriptomic signatures reflect immune activation rather than direct tumor genetics in breast cancer prognosis [57]. These examples illustrate how the dominant layer responsible for identification often reflects underlying tumor biology.

Nevertheless, several biomarkers were occasionally identified through non-canonical omics layers. For example, IDH1 was identified by DeepKEGG through CNV (Supplementary Fig. 16), TP53 by DeePathNet and DeepKEGG through DNAm (Supplementary Figs 16 and 19), and FCGR2B by DeepKEGG, MOGLAM, and TMO-Net through CNV or miRNA (Supplementary Figs 16, 18, and 21). These anomalous identifications underscore the complex interplay across omics modalities and the potential for cross-omics compensation.

Since methods can identify biomarkers through omics layers that do not commonly drive the underlying biology, we next interrogated whether methods have intrinsic preferences for specific omics types. We summarized the proportions of each omics type at different rank percentile cutoffs using each method’s raw feature scores or rankings, per omics combination and task (Fig. 3b). Some methods demonstrated clear biases. As an example, GDF relied almost exclusively on mRNA, DNAm, and miRNA, whereas CNV and SNV rarely appeared as dominant (Fig. 3). Consequently, whereas most methods identified TP53 primarily via SNV, GDF identified TP53 through mRNA, DNAm, and miRNA (Fig. 3a, Supplementary Figs 16–21). Other methods exhibited more diverse patterns. Notably, DeePathNet, DeepKEGG, P-Net, and GAUDI were the only four methods for which all five omics types appeared as dominant contributors across biomarkers, and three of these (DeePathNet, DeepKEGG, and GAUDI) were among the top performers. DeePathNet and DeepKEGG, in particular, displayed especially diverse contributions. For a single biomarker, as many as four different omics types could be dominant (e.g. PIK3CA with DeePathNet; NRAS, KRAS, and GNAS with DeepKEGG in the COADREAD survival task). By contrast, most other methods typically have a single dominant omics type per biomarker (Fig. 3a).

As it may be challenging to know a priori which omics combination will perform best, these findings indicate that including a broader set of omics types is beneficial for biomarker discovery from multi-omics data. Broad inclusion increases the likelihood of incorporating the mechanistic information relevant to specific biomarkers and allows methods to perform optimally, whereas restricting experiments to a single omics combination may overlook biomarkers whose key information resides in an untested molecular layer.

Performance on simulated data

A major challenge in benchmarking biomarker identification methods is the absence of complete ground truths. Our use of curated cancer biomarker knowledge bases, task matching, and rank-based metrics offers a practical solution to this problem for real-world data. In parallel, as a complement to real-world evaluation, we generated simulated multi-omics data with predefined discriminative features, enabling evaluation with metrics such as AUROC and accuracy. We varied the mean-shift of ground-truth features to modulate their discriminative strength (Supplementary Methods). It was observed that simulations reproduced some trends from real-data results. Feature-ablation DL methods (MOGONET, MoAGL-SA, and MORE), which were weak on real cohorts, also underperformed in simulations. DIABLO remained among the top performers (Fig. 4), consistent with its capacity to model coherent multivariate structure. These consistencies increase confidence in methods whose performance appears robust to changes in data generation complexity.

Figure 4.

Performance on simulated data. Cleveland dot plots showing biomarker identification accuracy (a) (bars represent standard deviation across five-fold cross-validation) and stability (b). Colors represent a particular biomarker signal strength (Supplementary Methods).

However, discrepancies were more common. DeePathNet, DeepKEGG, MOGLAM, and TMO-Net became less competitive, and many statistical and ML methods (e.g. MCIA, MOFA, and DPM) improved substantially. Within either the DL or the statistical and ML group, accuracy rankings differed from those within real cohorts (Fig. 4a). These shifts indicate that simplified generative mechanisms introduce a bias toward methods optimized for less noisy discriminative effects, while attenuating the relative advantages of methods intended to recover more complex relationships.

Across discriminative strengths, statistical and ML methods exhibited greater variability than DL methods (Fig. 4), indicating higher sensitivity to the magnitude of predefined effects. This is consistent with the fact that simulated datasets lack the heterogeneity and cross-omics dependencies observed in real tumors, thus simulations tend to favor conventional statistical and ML methods that assume simpler and less noisy underlying structure. In contrast, the capacity of DL models to utilize subtle or composite multivariate patterns becomes evident primarily in real-world datasets characterized by substantial heterogeneity and nonlinear cross-omics relationships. These discrepancies underscore the limitations of relying on synthetic discriminative features as benchmarking ground truth. Although widely used in prior work [6, 26, 28, 34, 35], simulation-based evaluations did not align with assessments based on real-world cohorts and clinically established biomarkers.

Stability patterns on simulated data were more consistent with those from real cohorts than accuracy (Figs 4b and 2c), suggesting that stability is driven more by methodological design than by the numerical properties of data. Nonetheless, statistical and ML methods showed larger stability shifts as discriminative strength varied, indicating higher sensitivity to the structures within simulated data.

Consensus multi-omics biomarker panels

Our unified benchmarking framework, which evaluates multiple methods across shared tasks and omics combinations, provides a basis for deriving consensus biomarker panels. Therefore, we prioritized candidate biomarkers within each omics type by aggregating results from multiple top-performing methods. Since design choices and omics-specific dependencies vary widely across methods, the resulting rankings show only modest concordance among related approaches and weak concordance across distinct methodological families (Supplementary Fig. 22). Integrating these rankings can therefore uncover robust biomarker candidates that are less susceptible to method-specific biases and more representative of integrative molecular signatures, producing an accurate and reproducible consensus.

For each benchmarked task, we selected top-performing method-omics combination pairs, and derived consensus rankings via robust rank aggregation (RRA) [58] (Methods). RRA converts cross-method agreement into per-feature -values, accommodates partial rankings, and alleviates outliers and method-specific biases. The consensus provides statistically robust significance levels with controlled false discovery rates (FDRs), enabling distillation of sparse, accurate, and reproducible biomarker candidate panels (Table 4). As an example, in BRCA survival, the consensus identified 10 mRNA biomarkers (FDR ), 10 miRNA biomarkers (FDR ), 14 DNA methylation (DNAm) biomarkers (FDR ), 14 CNV biomarkers (FDR ), and 8 SNV biomarkers (FDR ) (Table 4). Derived from diverse methods and omics combinations that accurately and consistently recover clinically validated biomarkers, these panels provide high-confidence candidates for biological and clinical investigation. Full consensus rankings for all tasks and multi-omics features are available in Supplementary Data 1.

Table 4.

Consensus multi-omics biomarker panels by task and omics type, omitting gold-standard biomarkers.

Task	Omics	-value	Consensus panel
Survival BRCA	mRNA		GNG5, MYC, TPGS1, NTRK3, GATA3, MAP2K6, MLPH, MTOR, PRM1, NPHS2
	miRNA		hsa-let-7d, hsa-mir-16-1, hsa-mir-326, hsa-mir-502, hsa-mir-301b, hsa-mir-18a, hsa-mir-130b, hsa-mir-30e, hsa-let-7g, hsa-mir-331
	DNAm		SEC22B, IL11RA, AL121900.1, NCAPG, DZIP3, PRUNE1, WDR12, BMPR2, LDHA, SMIM26, AL162231.3, TRMT11, ZNF641, ZNF331
	SNV		CDH1, GATA3, TTN, SLITRK3, RYR2, HMCN1, RUNX1, KMT2C
	CNV		SLC25A32, MYC, PIP4P2, SLAMF7, MTSS1, KLHL38, PLEKHO1, TRHR, RIPK2, C1orf54, TG, PABPC1, ARNT, DSCC1
Survival LUAD	mRNA		SNRNP70, RHOT2, PIDD1, ZNF692, TEPSIN
	miRNA		hsa-mir-143, hsa-mir-191, hsa-mir-210, hsa-mir-590, hsa-mir-93, hsa-mir-328, hsa-mir-1307
	DNAm		RPL14, UCHL3, RPS8, AL356512.1, LIN7C, C6orf62, TRAF6, CEP350, BMT2, AL353708.1, RPP40, AP2S1, AC007036.5, IL27RA, ITGB3BP, MFNG, TTC8, BBS12, AC022098.2, FBXW11, AC051619.7, PLEKHF2, PLA2G12B
	SNV		TLR4, TP53, APOB, TNR, RELN, COL11A1, TNN, LAMA2, RYR3, KEAP1
	CNV		CREB3L4, ATP1B1, PKLR, CACNA1S, EFNA4, FCGR1A, SEC61G, CTNNA2, CRTC2, ASAP1, PHKG1, WDPCP, H4C15, PSPH, ATF2, CTSK, RGS2, RAB13, B3GNT2, MAPK13, THBS3
Survival COADREAD	mRNA		MAPK1, GRB2, JAK1, ITGB1, NCOA3, MAPK8, GSK3A, MAP3K1, IKBKB
	miRNA		hsa-mir-191, hsa-mir-17, hsa-mir-454, hsa-mir-93, hsa-mir-186, hsa-let-7d, hsa-mir-130b, hsa-mir-942, hsa-mir-33a
	DNAm		WNT1, SDHC, AL359504.1, PRKACB, MIR618, TRAF6, U47924.1, MIR141, MIR200CHG, CALML3-AS1, CALML3, EGFEM1P, MAPK13, AC018521.2, CRLF3, RNF182, MIR200C, EGOT, AP000911.1, FGF5, CFAP299
	SNV		APC, TP53, GRIK2, DAAM2, PCLO, KMT2C, TTN, TNRC6B, FLG, ARID1A, VCAN, OBSCN, ADGRV1, RYR1, KCNQ2, MUC16, NEB
	CNV		PCK1, PEDS1, NCOA3, NFATC2, BCL2L1, MYLK2, COX4I2, PLCG1, GDF5, ACSS2, ITCH, PABPC1L, STK4, YWHAB, SDC4, RBPJL, ADA, SGK2
Drug response Cisplatin (BLCA)	mRNA		PRF1, PRICKLE1, TAF11L11
	miRNA		hsa-mir-133a-1, hsa-mir-133b, hsa-mir-1-2, hsa-mir-590, hsa-mir-16-2, hsa-mir-186, hsa-mir-1-1, hsa-mir-671, hsa-mir-133a-2, hsa-mir-92a-2, hsa-mir-454, hsa-mir-197, hsa-mir-103a-1, hsa-mir-19a, hsa-mir-423, hsa-mir-99a, hsa-let-7d
	DNAm		UBXN7, AC104581.2, ESRRA, EXOSC5, BCKDHA, AC078916.1, ZNF195
	SNV		SLC12A5, KDM6A, AKAP9, TTN, RYR3, TP53, ARID1A, TNFAIP3, KMT2C, TECTA, ERBB2, OBSCN, PTEN, DNAH3
	CNV		CDV3, CLDN16, CLDN11, ANAPC13
Drug response Temozolomide (LGG)	mRNA		GPR143, PGAP1, PRICKLE3, RASAL1, FBXO17, PCDH15, PDPN, FKBP9
	miRNA		hsa-mir-324, hsa-mir-331, hsa-mir-107, hsa-mir-628, hsa-mir-185, hsa-mir-22, hsa-mir-27b, hsa-mir-23b, hsa-mir-29b-1, hsa-mir-874, hsa-mir-24-2, hsa-mir-24-1, hsa-mir-769
	DNAm		EEF1B2P4, VCL, PTGER1, AC084026.2, CKMT2, PDE2A-AS2, MT1M, EPOR, BISPR, NEAT1, ARAP3, B3GNT5
	SNV		ATRX, TP53, CIC, TTN, NOTCH1, FAT2, FLG, EGFR, OR51F2, ITPR3
	CNV		BAMBI, ARMC4, DNAJC1, OTUD1, CCNY, ARHGAP21

Many consensus panel candidates are established cancer genes, including CDH1 and GATA3 in the BRCA survival SNV panel [55], APC in the COADREAD panel [59], KDM6A in the cisplatin (BLCA) panel [60], and ATRX in the temozolomide (LGG) panel [61]. The recovery of such well-characterized genes provides evidence that the top-performing methods and RRA aggregation identify biologically meaningful candidates. All candidates are defined on routinely profiled molecular readouts and can be tested in independent cohorts using standard assays. However, they represent discovery-stage, computationally prioritized candidates that would require independent analytical and clinical validation before translational use [38].

Practical guidance for method selection

To facilitate practical use of the benchmark, we provide a method selection guide organized around three considerations: whether sample-level labels are available, whether GPU resources are accessible, and whether preservation of the original feature space is desired (Fig. 5). DeePathNet and DeepKEGG are the strongest general-purpose choices, achieving the highest accuracy with strong stability via pathway-informed transformer architectures with advanced attribution methods. When labels are unavailable, GAUDI is preferred; when GPU resources are unavailable, DIABLO offers strong performance with lightweight computation; when preservation of the original feature space is desired, MOGLAM is recommended. Methodological designs that are consistently unsuitable, including univariate feature ablation, univariate statistical scoring, and unsupervised matrix decomposition without multivariate attribution, are not recommended.

Figure 5.

Practical guidance for method selection. Decision tree for selecting multi-omics biomarker identification methods based on sample-label availability, GPU accessibility, and whether preservation of the original feature space is desired. Methods are grouped into recommended, acceptable, and not recommended tiers according to overall benchmark performance. Representative runtime and peak memory across three data scales are provided for practical reference. GPU peak memory is provided for DL methods, and CPU peak memory is provided for non-DL methods.

Method selection does not depend on the specific omics types available, as all benchmarked methods support arbitrary omics combinations in our evaluation pipeline. We recommend including mRNA with two additional omics types to conform with our benchmark settings. For more robust biomarker prioritization, aggregating rankings from selected methods via RRA [58] can reduce method-specific biases.

Discussion

In this study, we presented a systematic benchmark for computational biomarker identification from multi-omics data, evaluated against clinically validated reference biomarkers. The results uncovered certain methodological designs that generally exhibited strong or weak accuracy and stability in biomarker identification, providing guiding principles for future method development and practical recommendations for method selection across different research scenarios. Analysis of omics type contributions indicates that biomarker identification is strongly influenced by the molecular modalities included and by model-specific biases, underscoring the importance of utilizing diverse omics layers for biomarker discovery. Results on simulated data showed that simplified generative assumptions can produce baseline results differing substantially from those observed in real-world cohorts, emphasizing the indispensability of real-world evaluations when assessing translational applicability. The consensus biomarker panels derived from the most accurate and stable methods and omics combinations provide computationally reproducible candidates that may inform potential biological and clinical investigations. Finally, the released evaluation framework enables researchers to benchmark new methods against the 20 baselines, or to apply selected methods for biomarker identification on other multi-omics data.

Challenges are identified that warrant future work. First, the incompleteness of available ground truth remains a fundamental limitation for real-world benchmarking. Although we compiled biomarkers from multiple curated cancer knowledge bases [42–44] and used rank-based metrics to mitigate the effects of missing annotations, biomarkers that have yet to be discovered or clinically characterized are naturally absent from evaluation. This is particularly relevant for therapeutic tasks, where the number of clinically validated biomarkers is smaller, which may lead to underestimation when models identify biologically relevant but undocumented candidates. As the coverage of cancer biomarker resources expands, future benchmarks will be able to incorporate broader reference sets. In addition, while our study focused on Tier I biomarkers [41] to maintain translational relevance, the inclusion of lower tiers in future work may help evaluate each method’s capacity to detect plausible but currently unvalidated candidates.

Another limitation is that all benchmark tasks were derived from TCGA and therefore do not constitute external validation on independent non-TCGA cohorts. Such datasets remain scarce when requiring matched mRNA, miRNA, DNA methylation, CNV, and SNV profiles together with biomarker-aligned clinical annotations and sufficient sample size. Nevertheless, TCGA includes substantial cross-center heterogeneity, as samples within each task originate from multiple tissue source sites, and their compositions vary across both tasks and cross-validation folds (Supplementary Fig. 24). This provides a partial internal test of robustness, but dedicated benchmarking on more independent cohorts will be a valuable direction for future work.

Our conclusions should also be interpreted within the scope of gene-centered multi-omics biomarkers. The curated biomarker set spans multiple clinically established alteration classes, covering major mechanisms with established clinical significance (Supplementary Table 3). However, many therapeutic interventions and drug responses are mediated through protein abundance, post-translational regulation, and signal transduction pathways, and the associated biomarkers remain underrepresented in both current oncology knowledge bases and cancer cohorts. Incorporating such modalities with matched clinical reference standards will be a potential direction for extensions.

Methods

Overview of benchmarked methods

Here, we give an overview of the benchmarked methods. A detailed description of each is further provided in Supplementary Methods.

Deep learning methods

Based on architectural backbones, DL methods can be broadly grouped into four categories: FNN-based (GENIUS [20] and P-Net [16]), autoencoder-based (TMO-Net [8] and CustOmics [9]), graph-based (MOGLAM [12], GNN-SubNet [11], MOGONET [10], MORE [18], and MoAGL-SA [19]), and transformer-based (DeePathNet [14], DeepKEGG [13], and Pathformer [15]) (Fig. 1c).

For FNN-based methods, P-Net constructs a hierarchical sparse neural network based on gene-pathway and pathway-biological process connections [16]. GENIUS employs CNNs and integrates prior biological knowledge by converting multi-omics data into “gene images,” using genomic coordinates to define spatial arrangements across omics layers as image channels [20].

Autoencoder-based methods learn low-dimensional representations across omics. Standard dense neural networks are typically employed as backbones. TMO-Net applies multiple variational autoencoders for intermediate fusion, capturing both self- and cross-modal associations [8]. CustOmics uses a hierarchical mixed-integration strategy, learning omics-specific sub-representations via separate autoencoders, which are then combined through a central variational autoencoder [9].

Graph-based methods can be generally divided into two types. Models using samples as graph nodes typically build a sample-level similarity graph for each omics type and perform fusion afterwards. MOGONET fuses the predictive output of each omic-specific GCN in the label space, while MoAGL-SA, MORE, and MOGLAM use GCN or hypergraphs with self-attention mechanism to learn a joint graph representation of multi-omics features [10, 12, 18, 19]. The method using genes as graph nodes (GNN-SubNet) constructs the graph using PPI network as prior knowledge, learning graph feature representations followed by global-pooling for sample-level predictions [11].

Transformer-based methods integrate pathway knowledge into the model architecture. DeePathNet groups features into pathways and applies self-attention over pathway-level embeddings. Pathformer transforms gene embeddings constructed from per-omic gene-level statistics into pathway embeddings, and uses criss-cross attention for pathway crosstalk modeling. Both DeePathNet and Pathformer use early fusion by integrating multi-omics features before model inputs, while DeepKEGG applies intermediate fusion via omics-specific encoders [13–15].

Most DL methods use post hoc feature attribution methods for biomarker identification. These model-agnostic algorithms are applied after training, computing feature scores using gradient-based methods represented by DeepLIFT [62] and IGs [63], as well as other types of methods such as SHAP [64] and GNNExplainer [65]. Some methods use performance drop after zeroing out features (feature ablation) to attribute feature importance. Notably, MOGLAM is an ante hoc method, learning a sparsity-regularized feature-indicator matrix during training, which directly encodes feature importance.

Statistical and machine learning methods

In addition to DL methods, multi-omics biomarker identification methods based on more conventional statistical or ML algorithms can be traced earlier in the research line. MCIA [46] was one of the earliest models for multi-omics integration and biomarker identification, performing joint projections of omics into low-dimensional spaces by maximizing their shared covariance structure. MOFA [5] introduced a statistical framework using matrix factorization for factor analysis. More recently, GAUDI [7] applied UMAP-based transformations followed by XGBoost or random forest with SHAP [64] for biomarker identification. These methods are unsupervised without relying on sample-level labels.

In contrast, DIABLO [6] and asmbPLS-DA [47] are supervised learning methods based on sparse generalized canonical correlation analysis (sGCCA) and sparse partial least-square discriminant analysis (sPLS-DA), respectively. Both impose sparsity constraints to identify biomarkers through non-zero component loadings. Stabl [35] wraps a sparse base learner (e.g. logistic regression with regularization) in bootstrap subsampling to compute per-feature selection frequencies, and identify those above a reliability threshold as biomarkers. GDF [17] applies a greedy decision forest over a PPI network using Gini importance for feature scoring. DPM [48] aims at pathway-level modeling but identifies gene-level biomarkers by integrating regulatory directionality (positive/negative associations) across omics with a P-value merging algorithm. Unlike DL methods, most statistical and ML methods achieve ante hoc biomarker identification via feature weights updated during model optimization.

Collection of gold-standard biomarkers

A critical challenge precluding quantitative evaluation of biomarker identification performance lies in the lack of an off-the-shelf biomarker set, i.e. both clinically validated and aligned with the task of interest. To address this, we collected biomarkers with strong clinical evidence from established oncology biomarker knowledge bases, including CIViC [42], OncoKB [43], and CGI [44]. To remove confounding effects introduced by biomarkers with weak evidence, such as those identified in silico by computational models without clinical validation, we harmonized the evidence levels of the three knowledge bases into a unified evidence system (Level A, B, C, and D) as defined by the AMP/ASCO/CAP guidelines [41] (see Supplementary Table 2 for details) and retained only those with Tier I evidence. These biomarkers are supported by professional guidelines, FDA-approved therapies, or well-powered clinical trials [41]. The curated dataset was stratified into prognostic markers (derived from OncoKB and CIViC) and predictive markers (aggregated from all three sources). To ensure consistency with TCGA data, cancer type names in each source were mapped to TCGA project identifiers. Molecular profiles in CIViC were mapped to HGNC gene symbols (Data availability), as were the gene names from the other two sources, to resolve aliases. Therapeutic agent names were harmonized using the standardized terms collected by Ding et al. [66], supplemented by a manually defined mapping to resolve investigational codes, salt forms, spelling errors, and formatting inconsistencies. This curation enables a direct and robust assessment of a method’s ability to identify biomarkers that have a high likelihood of succeeding clinical trials and achieving widespread application.

Derivation of gene-level rankings

Since the collected biomarkers are at the gene level, we adopted a gene-centric approach to enable metric calculations. Specifically, for DNA methylation and miRNA, the average scores of the CpG sites or miRNA molecules regulating the same gene were used as the DNAm- or miRNA-type score for that gene, according to regulatory annotations provided by TCGA and miRTarBase (Data availability). For mRNA, CNV, and SNV, and methods with a gene-centric design, including GENIUS, DeePathNet, DeepKEGG, Pathformer, P-Net, GNN-SubNet, GDF, and DPM (Supplementary Methods), this step was omitted since the output scores are already at gene-level. Following this, gene scores from different omics types were aggregated through max-pooling, resulting in a single score for each gene. For methods that output sample-specific feature scores, the gene scores were further averaged across samples within the same class, and then converted to absolute values and averaged across classes. Lastly, genes with zero scores were randomly permuted, and a final gene-level ranking was formed.

Derivation of consensus multi-omics biomarker panels

Performance varied substantially across different omics combinations (Supplementary Figs 1–15). For this reason, we treated each five-fold experiment separately, where a five-fold experiment refers to a five-fold cross-validation performed within a specific method and omics combination. We first computed an overall performance score for each experiment,

where each metric is averaged across the five-folds. All experiments were then ranked in descending order according to this score.

Next, based on the dominant omics type within the top 1% rankings, for each of the five omics types (mRNA, miRNA, CNV, SNV, and DNAm), we selected the top three experiments that included that omics type. For each selected experiment, we applied the RRA algorithm [58] using the RobustRankAggreg R package (v1.2.1) to integrate the rankings from the five-folds. RRA produces an aggregated ranking with Benjamini–Hochberg corrected P-values that indicate consensus significance. After obtaining three aggregated rankings for a given omics type, we applied RRA again to integrate these three lists and derive a final consensus ranking for that omics type. During this process, for methods that do not require gene-level inputs (i.e. those except Pathformer, DeePathNet, DPM, GNN-SubNet, GENIUS, GDF, and P-Net; Supplementary Methods), we converted CpG scores to gene level using mean aggregation for DNAm. For gene-centric methods, we converted miRNA scores to gene level. Full results are provided in Supplementary Data 1.

Evaluation metrics

Accuracy metrics

We adopt rank-based metrics for evaluating biomarker identification accuracy. Let denotes the gene set in a gene-level ranking list, and let be the subset of genes in that are reference biomarkers. Let be the rank of biomarker ().

Average recall.

, higher means more accurate.

Normalized discounted cumulative gain. NDCG has been widely used in information retrieval [67, 68] studies and can measure the prioritization of relevant items in top rankings. The discounted cumulative gain (DCG) is defined as

The ideal DCG (IDCG) is obtained by placing all biomarkers at the top of the list:

The NDCG is then

with indicating a perfect ranking in which every biomarker precedes all non-biomarkers.

Reciprocal rank. RR is a metric commonly used in information retrieval and question answering to measure how highly the first relevant item is ranked [69]. Let denotes the rank of the highest-ranked biomarker, RR is defined as

assigning a score of when a biomarker occupies the highest rank and decaying hyperbolically as the first biomarker appears deeper in the ranking list.

Mann–Whitney U test p-value. We compute a one-sided Mann–Whitney test comparing to using scipy.stats.mannwhitneyu(alternative=‘less’, method=‘exact’) (SciPy v1.14.1). The reported quantity is the P-value , where smaller indicates stronger evidence that biomarkers rank higher than non-biomarkers.

Area under the ROC curve. For each , let be its gene-level score (larger = more biomarker-like) and . The AUROC is the area under the ROC curve obtained by thresholding . Equivalently,

We compute AUROC with sklearn.metrics.roc_auc_score (scikit-learn v1.7.2).

Accuracy@K. Let be the set of the top- genes. Then

the fraction of the top- predictions that are known biomarkers.

Stability metrics

For stability, we calculated the average similarities between each pair of the five-fold ranking lists.

Average Kendall’s . Kendall’s [70] is widely used for measuring the overall similarity between two rankings. Given rankings and , it is defined as

where is the number of concordant pairs (same relative order in both rankings) and the number of discordant pairs. A higher value indicates higher similarity, with denoting perfectly concordant and denoting the least concordant (). We average over all pairs from the five-fold rankings as the final metric:

Rank-biased overlap. Although Kendall’s offers a global perspective on the overall ranking, including both top-ranked candidates and lower-ranked genes, it may be significantly affected by the instability at lower ranks. We therefore further adopted RBO [71], a common metric to emphasize top-ranking similarity:

where denotes the top- genes in ranking , and is the decay rate. Throughout our experiments, was set to to effectively account for the top 50 ranks, as evaluated and reported by Webber et al. [71]. RBO, with higher values indicating higher concordance. Similar to Kendall’s , we average RBO over all pairs from the five-fold rankings as the final metric.

Rank percentile standard deviation. We define RPSD to measure the variability of known biomarkers’ ranks across cross-validation folds. For each biomarker and fold , let

be its rank percentile, and let

be the mean rank percentile for across folds. We first compute the per-biomarker standard deviation

and then average over all biomarkers

Lower RPSD indicates more consistent ranking of biomarkers across folds, with 0 denoting perfect stability.

Key Points.

• This study systematically benchmarks 20 statistical, machine learning, and deep learning methods for multi-omics biomarker identification using clinically validated Tier I biomarkers across five real-world cancer datasets.

• Methodological analysis reveals that effective biomarker recovery is driven by the integration of biological knowledge, global feature interactions, multivariate feature attribution, and regularization.

• Discrepancies observed between simulated and real-world results underscore the limitations of simulated data and the necessity of benchmarking on real-world cancer cohorts to ensure translational applicability.

• Analysis of omics type contributions reveals method- and modality-specific biases, demonstrating that biomarker identification is strongly influenced by the molecular layers included and that broader omics integration increases the likelihood of capturing relevant biological signals.

• Consensus biomarker panels derived by aggregating rankings from top-performing methods via RRA provide high-confidence candidates that mitigate method-specific biases and may inform future biological and clinical investigations.

• A user-friendly evaluation pipeline enables researchers to benchmark new methods against the 20 baselines on the curated task datasets, or to apply selected methods to their own multi-omics data for biomarker identification.

Author contributions

R.L. conceived and supervised the study. A.Z.L. conducted the literature review, designed the methodology, developed the software, generated the visualizations. A.Z.L., Y.D., Y.L., L.C., and R.L. analyzed the results. A.Z.L. and R.L. drafted the manuscript.

Conflicts of interest

None declared.

Funding

None declared.

Data availability

Our benchmark is publicly available at https://github.com/athanzli/CancerMOBI-Bench. Our benchmarking datasets and results are available at 10.5281/zenodo.17860662 [72]. TCGA omics and clinical data were downloaded from the GDC data portal at https://portal.gdc.cancer.gov (accessed 6 November 2024). Reference biomarkers were retrieved from https://www.oncokb.org/actionable-genes for OncoKB (accessed 15 June 2025), https://civicdb.org/releases/main for CIViC (1 January 2025 release), and https://www.cancergenomeinterpreter.org/data/biomarkers for CGI (latest version, accessed 15 June 2025). The HGNC complete gene set information was downloaded from https://storage.googleapis.com/public-download-files/hgnc/tsv/tsv/hgnc_complete_set.txt (accessed 11 February 2025). miRNA target gene information was retrieved from miRTarBase at https://mirtarbase.cuhk.edu.cn/∼miRTarBase/ (accessed 21 April 2025). Methylation array manifest file (HM450.hg38.manifest.gencode.v36.tsv.gz) and TCGA antibodies descriptions file (TCGA_antibodies_descriptions.gencode.v36.tsv) were downloaded from https://gdc.cancer.gov/about-data/gdc-data-processing/gdc-reference-files. All biological pathway data files were downloaded from the corresponding methods’ data repositories. PPI network topology file was downloaded from the STRING [73] database at https://string-db.org/ (accessed 28 October 2024).

Computational resources

Experiments were primarily conducted on a Linux Ubuntu 22.04 system equipped with 8 NVIDIA RTX A6000 GPUs (48 GB each), a dual-socket AMD EPYC 7763 processor (264 cores, 2 threads per core), and 1008 GB system memory. Experiments requiring larger GPU memory were run on a Linux Ubuntu 24.04 system equipped with one NVIDIA H200 GPU (SXM, 141 GB), a dual-socket Intel Xeon Platinum 8570 processor (256 cores, 2 threads per core), and 2 TB system memory.