Unified heterogeneity-aware benchmark of drug synergy prediction: a cross-study analysis of traditional machine learning and graph deep learning models

Abstract

Drug synergy prediction holds great promise in accelerating combination therapy development and improving treatment efficacy in cancer and other complex diseases. However, progress in this area is hindered by considerable heterogeneity across experimental datasets, including variability in the number of drug combinations, inconsistencies in synergy scoring methodologies, and differences in data quality. Here, we present the first comprehensive benchmarking framework specifically designed to accommodate inter-dataset heterogeneity. This framework integrates 13 independent datasets encompassing 454,794 retained drug combination–cell line entries, 4247 drugs, and 187 cell lines, all of which are cancer cell lines. Our comparative evaluation of seven computational models for drug synergy prediction reveals that model performance strongly depends on both the scale and quality of the datasets. The graph model JointSyn performed favorably on datasets with larger numbers (> 10,000) of retained drug combinations, while the traditional random forest model performed competitively on smaller-scale datasets. We also observe that the ZIP scoring metric yields the highest accuracy in large-scale data, whereas HSA is more effective in sparse-data scenarios. However, different synergy metrics show significant variability in performance across datasets, suggesting that different synergy metrics capture distinct aspects of drug interactions, and the choice of metric can substantially affect model evaluation and cross‑dataset consistency. Furthermore, we find that well-designed small datasets can match or even surpass the performance of larger benchmarks, suggesting that different metrics are applicable to different datasets/testing scenarios. Our benchmark provides a robust foundation for fair model evaluation and paves the way for the development of more generalizable and preclinically relevant drug synergy prediction methods.

Graphical abstract

Supplementary Information

The online version contains supplementary material available at https://doi.org/10.1186/s13321-026-01242-1.

Scientific contribution

We establish the first heterogeneity-aware benchmark for drug synergy prediction by integrating 13 datasets to quantify critical heterogeneity drivers. Through a comparative evaluation of 7 representative models, spanning traditional machine learning and state-of-the-art (SOTA) deep learning architectures, across 4 synergy scores, we compare the performance of graph-based and non-graph-based approaches under different benchmark settings and provide a comparative reference for metric behavior across datasets, collectively providing benchmarking standards to support future preclinical evaluation of synergy prediction models.

Supplementary Information

The online version contains supplementary material available at https://doi.org/10.1186/s13321-026-01242-1.

Introduction

Due to the biological complexity of cancer, the development of such diseases often involves multiple independent gene mutations [1]. However, traditional monotherapies, which typically target a single protein or pathway, often exhibit limited therapeutic effect and are prone to drug resistance [2]. Moreover, these monotherapies may induce excessive activation or inhibition of drug targets, triggering compensatory or protective mechanisms that compromise treatment effectiveness [3]. Due to the inhibitory ability on multiple cancer-related pathways or targets, drug combination might achieve superior therapeutic efficacy, better drug resistant capability and reduced toxicity compared to monotherapies [4]. Over 350 drug combinations have already been approved for clinic cancer treatment [5]. However, relying solely on clinical experience and in vitro/in vivo experiments for drug combination discovery is quite costly and time-consuming for exploring the vast combinatorial space. The scale of this challenge is exemplified by the DrugComb database containing 8397 drugs that theoretically allow over 35 million pairwise combinations, far exceeding the capacity of experimental testing [6]. The complexity of combination testing grows exponentially when considering critical experimental variables: dosage optimization across concentration gradients, and biological variability among 2320 cancer cell lines derived from 33 human tissues. While DrugComb documents 739,964 experimentally tested combinations, this represents less than 0.5% of all theoretically possible configurations [6]. This substantial gap between combinatorial possibility and experimental coverage highlights the urgent need for computational approaches to systematically prioritize viable combinations for targeted clinical validation.

The concurrent evolution of high-throughput technology and deep learning (DL) has reached a pivotal inflection point [7], enabling accelerated acquisition of drug synergy datasets [8] while catalyzing the development of large-scale databases such as DrugComb [9], DrugCombDB [10], and SYNERGxDB [11]. Drug combination synergy prediction methods can be broadly categorized into four major approaches: (1) neural networks, such as DeepSynergy [12] and AuDNNSynergy [13] that utilize multiple omics data such as gene expression data, gene mutation data and copy number variation data to achieve excellent performance on large datasets but are limited by computational complexity scaling and reduced interpretability when constrained by dataset size; (2) ensemble tree-based models, including random forests and XGBoost [14, 15] which are widely used for their robustness to overfitting, computational efficiency, and ability to identify predictive biomarkers; (3) network-based approaches that provide mechanistic insights through biological pathways but depend heavily on annotated network completeness [16]; (4) factorization methods that are powerful in predicting missing values in incomplete data tensors (e.g., combination response or synergy data) by learning from the co-occurrences of subsets of variables [17]. Despite the proliferation of diverse computational approaches, their real-world applicability remains obscured by critical benchmark heterogeneities: dataset fragmentation, inconsistency in synergy metrics (Loewe vs. Bliss), and evaluation protocol discrepancies (regression/classification tasks compounded by unreported hyper-parameter tuning in 60% of published studies [17]). This lack of standardized assessment protocols in practical application scenarios impedes reliable performance assessment, thus motivating our work to establish a more unified evaluation framework for benchmarking model performance across heterogeneous datasets.

The recent growth of relevant data and advancements in graph neural networks (GNNs) have catalyzed transformative drug synergy prediction. Operating on graph-structured data, GNNs demonstrate exceptional performance in modeling complex biological relationships while maintaining interpretability through their inherent graph-based reasoning frameworks [18]. In terms of utilizing data, most of these models focus on several directions, one for modeling drug molecules as graphs [19–22], another for modeling the relationships between drug molecules and other biological entities as graphs [16, 23–27], or combining the two [28, 29]. What’s more, computational methods used in drug combination prediction could enable the new use of old medicines, thereby increasing the indications for medication, facilitating research on mechanisms, and expanding the space for drug search [30, 31].

However, prevailing evaluation frameworks for drug synergy prediction suffer from critical limitations. Most models are validated either on isolated datasets (predominantly single sources like DrugComb’s ONEIL or ALMANAC) or on indiscriminately pooled databases (full DrugComb collections) [19–29]. This bifurcation represents two problematic extremes: the former fails to assess cross-dataset generalizability and robustness by ignoring inherent heterogeneity, while the latter obscures dataset-specific biases through aggregation. Large-scale experimental efforts, such as that by Jaaks et al. [32], have generated valuable synergy profiles across many cell lines; systematic reviews, including Kong et al., have catalogued existing computational methods [33]; and Vlot et al. have highlighted disagreements among synergy metrics and potential artefacts in their assumptions [34]. Nevertheless, these studies do not offer a unified, empirical benchmark that systematically compares diverse models under controlled dataset conditions while accounting for metric variability. Compounding these issues, most studies rely on single synergy metrics as the prediction target despite ongoing controversy about optimal scoring methodologies. Consequently, existing evaluations neither reflect biological variability across independent studies nor account for computational robustness to scoring disagreement that may be a key determinant of preclinical applicability. While nascent work examines heterogeneity impacts [35, 36], such efforts remain constrained by simplistic models or narrow dataset scopes. We therefore posit that rigorously benchmarking heterogeneity effects across diverse experimental contexts and multiple synergy metrics is indispensable for developing preclinically translatable combination therapies.

In this study, we conduct a systematic benchmarking analysis of dataset heterogeneity in drug synergy prediction. Specifically, we utilize 13 datasets from independent studies and provide a multifaceted comparison of 4 mainstream methodologies for calculating synergy metrics across 7 prediction approaches. Our empirical analysis reveals their distinct predictive capabilities. Model efficacy exhibits scale dependency, with some graph-based approaches performing favorably in larger benchmark settings, while traditional ML algorithms such as RF and XGBoost remain competitive in smaller cohorts. Finally, we quantitatively explore the impact of dataset heterogeneity on model prediction capabilities when applied to data derived from diverse research sources. We find that methodological variations in experimental protocols, biological context differences in cellular/molecular/tissue composition, and scale disparities across datasets may all contribute to performance differences. Notably, graph DL approaches may not consistently deliver superior predictive accuracy compared to traditional methods, although they may still offer interpretability advantages. Crucially, our analysis reveals two systemic challenges: (1) synergy sparsity with pronounced zero-synergy-centered distributions, and (2) metric discordance across established evaluation criteria (HSA [37]/Loewe [38]/Bliss [39]/ZIP [40]). All benchmark datasets analyzed in this study were curated from the DrugComb portal, and the four synergy metrics (Bliss, HSA, Loewe, and ZIP) were obtained from the DrugComb harmonization pipeline.

Results and discussion

Detection of the heterogeneity of drug synergy predictions from three dimensions

We aim to detect the heterogeneity of drug synergy predictions from different dimensions. To achieve this goal, we conducted this experimental comparison based on the DrugComb database from three dimensions: dataset, model, and drug synergy metric type (Fig. 1a). The prediction target in this benchmark is the continuous synergy score associated with each observed drug combination rather than a binary synergistic/non-synergistic label; therefore, regression-based metrics were used throughout. We construct a unified workflow to develop several models to predict drug synergy scores (Fig. 1b). The full methodological details for the comparative analyses can be found in the Materials and Methods section.

Fig. 1.

Overview of the framework on detecting the heterogeneity between drug combination datasets. a The strategy of detecting heterogeneity across datasets; b The workflow of drug synergy scores prediction

Characterization of benchmark datasets

In this study, we chose the latest version of the drug combination database from the DrugComb portal, which is the most commonly used database in this field [6]. To mitigate systemic inconsistencies in feature representations across various studies, particularly concerning potential discrepancies in the mapping of molecular descriptors and gene expression to biological entities, we implemented a comprehensive data recalibration framework. This involved systematically reconstructing all DrugComb drug and cell line representations directly from primary data sources. This recalibration was motivated by observations of potential inaccuracy propagation in existing resources, including misattribution of gene expression profiles between cell lines with missing data, utilization of contaminated cell lines later deemed unreliable, and discordance between SMILES string annotations and actual chemical entities. Detailed procedures for filtering drugs, drug combinations, and cell lines (e.g., removal of technical replicates, handling of missing values, standardization of drug names) are provided in the Materials and Methods section. Our final benchmark contains 13 datasets with 454,794 retained drug combination–cell line entries, 4247 drugs, 187 cell lines (Supplementary Table 1). These datasets exhibit significant heterogeneity across multiple dimensions including data volume, drug/cell line composition, tissue origins, and experimental conditions. As the largest repository of drug combination screenings in our collection, ALMANAC aggregates experimental data from multiple laboratories using heterogeneous protocols and this methodological heterogeneity potentially introduces confounding variables and batch effects requiring explicit statistical adjustment (Table 1) [41]. In contrast, ONEIL employs a rigorously standardized experimental protocol exclusively utilizing technical replicates to generate highly reproducible, quality-assured data [42]. What’s more, NCATS_ES (FAKI/AURKI) contains 1870 drugs, the largest number of drugs screened (Table 1). ASTRAZENECA features the largest cellular diversity with 96 cell lines screened (Table 1) [43]. Unless otherwise specified, “data count” in this study refers to retained drug combination–cell line entries after preprocessing rather than the number of dose-matrix elements. A value of zero after preprocessing indicates that no entries from that dataset meet the final inclusion criteria for downstream benchmarking.

Table 1.

The features of the benchmark datasets (before processing vs. after processing)

Dataset	Data count		Number of drugs		Number of cell lines		Number of tissues
	Before	After	Before	After	Before	After	Before	After
ALMANAC	311604	288893	103	103	60	56	9	9
FRIEDMAN	198722	122738	108	105	36	24	1	1
ONEIL	92208	19822	38	38	39	34	6	6
CLOUD	40160	0	283	0	1	0	1	0
FRIEDMAN2	28122	0	76	0	10	0	1	0
ASTRAZENECA	19475	12550	116	101	146	96	10	8
MOTT	16445	0	223	0	3	0	1	0
FLOBAK	9984	1392	19	19	8	8	7	7
NCATS_DIPG	8778	0	2446	0	2	0	2	0
NCATS_ES(NAMPT + PARP)	4625	2981	94	93	4	4	3	3
NCATS_HL	2602	2400	1867	1864	4	3	2	1
NCATS_ES(FAKI/AURKI)	1875	1869	1873	1870	1	1	1	1
FORCINA	1818	1817	1818	1817	1	1	1	1
MATHEWS	1119	0	477	0	1	0	1	0
WILSON	745	0	31	0	2	0	1	0
DYALL	428	104	17	17	2	1	2	1
YOHE	270	180	25	25	3	2	2	1
BOBROWSKI	262	0	34	0	1	0	1	0
NCATS_SARS-COV-2DPI	205	0	56	0	1	0	1	0
FALLAHI-SICHANI	111	28	10	10	5	4	1	1
SCHMIDT	95	0	4	0	5	0	1	0
MILLER	82	0	13	0	1	0	1	0
NCATS_2D_3D	69	20	5	5	2	2	2	2
NCATS_MDR_CS	68	0	17	0	2	0	1	0
PHELAN	62	0	16	0	1	0	1	0
NCATS_ATL	30	0	22	0	1	0	1	0
Total unique across datasets	739,964	454,794	4268	4247	288	187	17	15

^*Data count: number of raw entries before filtering (including technical replicates)

^†Processed: number of unique drug-cell line pairs after removing replicates and missing values

^‡A value of zero after processing indicates that all raw entries for that dataset were removed, e.g., due to missing synergy scores or incomplete annotations

To evaluate the cross-datasets consistency of model-predicted synergistic scores, we systematically analyzed intersecting compounds, cell lines, and drug combinations (Fig. 2). Our assessment focused on overlapping therapeutic pairs, revealing that ALMANAC and ONEIL share the largest validated overlap of 353 drug combinations of which 350 are exclusive to this pair, while ALMANAC and FRIEDMAN jointly contain 223 overlapping pairs of which 221 are exclusive to this pair, as shown in Fig. 2a [44]. Besides, FRIEDMAN and ONEIL share 178 drug combinations of which 176 are exclusive to this pair, as shown in Fig. 2a. For the UpSet plot (Fig. 2a) and correlation analysis (Fig. 2c), only dataset pairs with ≥ 2 overlapping combinations were considered. Figure 2b shows all pairs without this threshold. In summary, these overlapping drug combination data permit rigorous assessment of datasets heterogeneity effects on predictive performance through controlled benchmarking of identical drug combinations across multiple experimental environments.

Fig. 2.

The overlap of the benchmark datasets (only datasets with > 2000 drug combinations are shown). a The overlap information of retained drug combinations data between partial datasets used in this study. Overlapping combinations are defined as identical drug-cell line triples (drug A, drug B, cell line) present in both datasets. The column on the left side of the figure shows the total data volume of each dataset. b The overlap information of unique cell line data between partial datasets used in this study. c The overlap information of unique drug data (displaying only intersections with cardinality > 1) between partial datasets used in this study

To further quantify cross-dataset heterogeneity, we calculated the pairwise correlations across four synergy scoring metrics, including Highest Single Agent (HSA) [37], Loewe additivity [38], Bliss independence [39], and Zero Interaction Potency (ZIP) [40], using the overlapping drug combination data in each dataset. HSA defines synergy as $E_{\mathit{AB}}>\text{max}(E_A,E_B)$, Loewe additivity is based on the isobole equation $\frac{a}{A}+\frac{b}{B}=1$, Bliss independence assumes independent drug effects with expected effect $E_{\mathit{AB}}=E_A+E_B-E_A{E}_B$, and ZIP compares dose–response curves to a null model of no interaction. When evaluating the overlapping drug-drug-cell line combinations across different datasets, we found that their synergy scores exhibited unexpected correlations that were often close to zero, though theoretically expected to be consistent for identical biological entities. For example, for Loewe scores between FRIEDMAN and ONEIL, Pearson’s r = 0.334 (p < 0.001). The majority of cross-dataset comparisons yielded absolute Pearson correlations below 0.300 (see Supplementary Table 2 for all values and p‑values). This substantial discordance is largely attributable to differences in experimental conditions across studies, including variations in dose concentration ranges and grid resolutions, cell culture protocols (e.g., serum concentration, incubation time), assay platforms (e.g., viability vs. confluence measurements), and batch effects in compound handling. These factors collectively demonstrate significant heterogeneity between datasets, even for identical drug‑drug‑cell line combinations.

Evaluation of model robustness across varying datasets

Following systematic benchmark curation, we conducted a comparative evaluation of seven computational models, including three traditional ML methods (XGBoost, Random Forest [RF], and Multilayer Perceptron [MLP]), three DL methods based on GNNs (JointSyn [29], CGMS [16], DeepDDS [22]), and one DL method not utilizing GNNs (MTLSynergy [45]). This comprehensive assessment utilized 13 drug combination datasets based on four synergy metrics, including HSA, Loewe additivity, Bliss independence and ZIP. HSA defines synergy as combination efficacy exceeding maximal monotherapy effect, providing intuitive clinical interpretability (e.g., FDA approval criteria) but risking false positives by ignoring dose-additive effects [37]. Loewe additivity detects mechanistic synergy through deviations from dose–response curves, particularly suited for drugs sharing targets/pathways but insensitive to independent-acting compounds [38]. Bliss independence identifies synergy via probabilistic effect multiplication, optimized for drugs targeting divergent pathways but frequently conflicting with Loewe predictions [39]. ZIP integrates Loewe's curve-fitting and Bliss' probabilistic approaches into a unified interaction landscape model, resolving inconsistencies while capturing complex dose-matrix patterns [40]. These metrics are not equally prioritized: HSA supports clinical validation, Loewe/Bliss illuminate mechanistic contexts, and ZIP offers high-resolution screening, reflecting an unresolved standardization debate in synergy quantification [46]. As depicted in Fig. 3, which is organized by model to emphasize cross‑dataset robustness, the Pearson correlation coefficient (PCC) performance demonstrates significant variation across datasets. Comprehensive error metrics (MSE, RMSE, MAE, R²) are further visualized in Supplementary Figs. 1–2, confirming differential model behaviors across evaluation frameworks.

Fig. 3.

The PCC performance across datasets. a Heatmap of PCC for each model on each dataset. Each model was evaluated using within‑dataset fivefold cross‑validation (random split of drug‑cell line pairs; no pair leakage). Hyperparameters were tuned on ONEIL using the Loewe synergy metric and fixed for all other datasets and metrics. Reported PCC values are fivefold averages. b Number of datasets (out of 4 large‑scale datasets: ONEIL, ALMANAC, FRIEDMAN, ASTRAZENECA; and 9 small‑scale datasets) where each model achieves PCC ≥ 0.8, shown separately for each synergy metric (Bliss, HSA, Loewe, ZIP). This panel quantifies model robustness across dataset sizes. See Supplementary Table 6 for details

In this study, we define dataset scale based on the number of unique drug combinations: “large‑scale” datasets contain more than 10,000 pairs, and “small‑scale” datasets contain fewer than 3000 pairs. Among our 13 datasets, the large‑scale datasets are ONEIL, ALMANAC, FRIEDMAN, and ASTRAZENECA; the other datasets are the small‑scale datasets.

Analysis of large-scale drug combination datasets (> 10,000 combinations) revealed distinct performance patterns among computational models across different synergy metrics, which can be interpreted as indicators of dataset difficulty. FRIEDMAN appears easier to model in three out of four metrics (excluding Loewe) when evaluated against the XGBoost, RF, MLP, JointSyn, CGMS and MTLSynergy models, though not against DeepDDS under the current benchmark setting (Fig. 3). ONEIL and ALMANAC were relatively easier to model, with ONEIL achieving particularly strong performance in Loewe additivity prediction that surpassed FRIEDMAN's benchmarks. Notably, while ASTRAZENECA contained a comparable number of drug combinations to ONEIL, it consistently posed greater challenges for all models. One possible explanation for this observation is the difference in cell line diversity (96 cell lines in ASTRAZENECA versus 34 in ONEIL), which may increase data heterogeneity and pose greater challenges for predictive models. Further analysis of dataset characteristics clarifies these performance patterns. The data density per cell line in ALMANAC is substantially lower than in FRIEDMAN, with ALMANAC containing approximately 60 cell lines compared to FRIEDMAN's 20, yet offering only twice the drug combinations (288,893 versus 122,738). This imbalance yields significantly fewer training samples per cell line in ALMANAC. Experimental consistency further differentiates these datasets: both ONEIL and FRIEDMAN originate from single studies employing standardized protocols [42, 44], while ALMANAC aggregates data from heterogeneous sources like the NCI-60 project [41]. These observations collectively demonstrate that optimal model performance depends not merely on dataset scale but fundamentally requires rigorous experimental controls to ensure reproducibility, sufficient combination-cell line observations, and minimized batch effect variations.

When evaluating small-scale drug combination datasets (< 3000 combinations), most exhibited poor performance across models and synergy metrics, reflecting the inherent difficulty of prediction with limited data (Fig. 3). However, FLOBAK emerged as an exceptional case and demonstrated robust predictive capability across multiple evaluation frameworks, occasionally surpassing the performance of larger benchmark datasets. This comparative advantage likely stems from its rigorous experimental design that incorporates triplicate biological replicates [47]. These observations suggest that dataset quality, particularly methodological rigor in experimental execution, outweighs raw data volume as a determinant of model efficacy.

It is worth noting, however, that high performance on a given dataset does not guarantee generalizability to other experimental settings; datasets with smaller scale or higher homogeneity may yield optimistic estimates that do not translate to more diverse data.

When evaluating model performance across different datasets, we observed significant scale-dependent behavior, as shown in Fig. 4, which is organized by models to highlight performance patterns across models under the current unified evaluation setting. A comprehensive analysis of error metrics, including MSE, RMSE, MAE, CI (Concordance Index) and R², further revealed differential responsiveness among models, as detailed in supplementary analyses (Supplementary Figs. 3–7). The GNN JointSyn demonstrated strong performance in large-scale datasets, achieving the highest overall Bliss synergy score on the FRIEDMAN dataset with a PCC of 0.97. In contrast, the fingerprint‑based RF performed well in most small-scale environments, maintaining a consistently leading PCC across four different synergy scores in small datasets. Notably, DeepDDS faced greater challenges compared to other graph models in complex datasets (Fig. 4a), with its performance falling below traditional ML models across nearly all datasets and synergy metrics. These results suggest that graph-based representations may be advantageous in certain large-scale settings, as done in JointSyn and CGMS, especially when drug entities are embedded within biologically informed relational contexts [16, 29], whereas models that rely on fixed molecular fingerprints (RF, XGBoost) perform better on small, homogeneous datasets where such descriptors are sufficient. Furthermore, JointSyn demonstrated empirically greater stability than CGMS across various datasets (Fig. 4b), indicating that while biological context is crucial, incorporating molecular graphs provides additional predictive power and enhances model interpretability through explicit visualization of pharmacophores [29]. While input data richness is crucial, the choice of representation learning architecture also fundamentally affects predictive performance. Crucially, although the MTLSynergy model does not employ GNNs, it still achieved performance comparable to JointSyn and almost surpasses both CGMS and DeepDDS, which do utilize GNNs. This suggests that the use of graph structures alone does not universally guarantee superior predictive performance across heterogeneous datasets, while still offering good interpretability. Besides, the consistent strong performance of traditional ML models in benchmark rankings highlights their effectiveness for small-scale datasets (Fig. 4c). Notably, RF demonstrated marginally greater stability than XGBoost under most experimental conditions, suggesting that increased model complexity does not inherently improve robustness. This observation challenges prevailing trends in drug combination prediction, where XGBoost is frequently prioritized, as RF achieved comparable accuracy with superior reliability (Fig. 4b) [48]. Thus, for conventional modeling scenarios, fingerprint‑based models like RF may offer optimal balance between performance and computational efficiency.

Fig. 4.

The PCC performance of models organized by models. Each model was evaluated using within‑dataset fivefold cross‑validation (random split of drug‑cell line pairs; no pair leakage). Hyperparameters tuned on ONEIL (Loewe) and fixed elsewhere. Reported PCC are fivefold averages. See Methods for details. a Heatmap shows the PCC performance of models organized by dataset. b Violin plot shows PCC ranking performance of models organized by dataset. Pairwise comparisons between models on the same dataset were performed using the Mann–Whitney U test (two-sided). Asterisks indicate statistically significant differences: * $p<0.05$, ** $p<0.01$, *** $p<0.001$. c The stacked bar chart displays the performance of different models in the top 3 PCC rankings on various datasets

A thorough analysis of the distributions of synergy scores across various drug combination datasets revealed two fundamental characteristics (Fig. 5a). While the four synergy metrics provided by DrugComb (HSA, Loewe, Bliss, ZIP) are widely used in benchmarking studies, recent literature has introduced more nuanced frameworks, such as the multi‑dimensional synergy of combinations (MuSyC) approach, which aims to account for dose‑response relationships and pathway crosstalk more comprehensively. These developments underscore that no single metric is universally optimal; rather, the choice of synergy metric should be tailored to the experimental context and research question. Firstly, all metrics displayed a marked tendency to concentrate near zero, indicating that most drug‑cell line pairs show additive or only weak synergistic effects, which is consistent with previous large‑scale combination screens [32, 33]. Strongly synergistic drug combinations constitute only a minor subset within the overall dataset. Notably, the selective nature of these strong synergies (active only in specific cell lines) may actually enhance their clinical translatability, as broad‑spectrum synergy could increase the risk of adverse effects. Secondly, significant disparities were observed in the distributional patterns among different metrics. Specifically, the ZIP and Bliss metrics exhibited a Pearson correlation of 0.98, while other pairwise combinations yielded correlations ranging from 0.13 to 0.31 (see Supplementary Table 3 for exact values), reflecting divergent pharmacological foundations [46].

Fig. 5.

The PCC performance across synergy metrics. a Overall distribution of four types of drug combination synergy metrics. The x‑axis is the synergy score value; the y‑axis is the frequency. The plot is based on all 454,794 drug-cell line pairs across 187 cancer cell lines. b Heatmap shows the average model performance of models on specific dataset across synergy metrics. Each cell shows the mean performance of all seven models evaluated on that dataset‑metric combination. Each model was evaluated using within‑dataset fivefold cross-validation (random split of drug-cell line pairs; no pair leakage). Hyperparameters tuned on ONEIL (Loewe) and fixed elsewhere. Reported PCC are fivefold averages. See Methods for details

The performance assessment further revealed substantial metric-dependent variability across datasets (Fig. 5b). This inconsistency, particularly evident in correlation-based assessments (PCC/R²), could be attributed to experimental heterogeneity and the fundamental incompatibility between linear correlation assumptions and the complex nature of pharmacological interactions. Crucially, scale-dependent patterns in performance became evident. ZIP demonstrated superior accuracy, as evidenced by lower RMSE and MAE values, in large-scale datasets with > 10,000 combinations. The advantage stems from its dose–response surface modeling approach, which integrates the strengths of both the Bliss and Loewe models [40], whereas HSA exhibited better performance in smaller datasets with < 3000 combinations, owing to its parametric robustness against sparse data noise [37].

Despite these observed differences in performance across metrics and dataset scales, current methodologies predominantly rely on Loewe scores as the primary training targets, which may introduce systematic bias. To mitigate this limitation, we conducted parallel evaluations using all four scoring paradigms. Our findings indicate that no single metric can comprehensively capture the full spectrum of drug interaction mechanisms, highlighting the need for a more nuanced and multi-metric approach to evaluating drug combination effects.

Conclusion

In this study, we conducted a comprehensive and systematic benchmark analysis to evaluate how dataset heterogeneity affects drug synergy predictions. Our findings highlight several critical insights that are highly relevant to the field of drug combination prediction. Firstly, we observed a considerable degree of heterogeneity among different drug combination datasets, which was clearly evidenced by the marked disparities in data volume, the range of drugs involved, the diversity of cell lines and tissues, as well as the variations in experimental conditions. Such heterogeneity poses a significant challenge to model performance and generalizability [35]. Our analysis revealed that datasets generated through rigorous experimental protocols and standardized conditions generally led to superior model performance, highlighting the importance of high-quality data in drug combination prediction. Secondly, our comparative evaluation of seven computational models across 13 datasets revealed a strong dependence of model performance on both the scale and quality of the datasets. GNNs, particularly the JointSyn model that integrates an entity relationship graph and a drug molecular graph, excelled on large-scale datasets, while traditional ML models such as RF performed better in small-scale datasets [29]. This suggests that the choice of an appropriate model should be guided by the characteristics of the datasets. More complex models are likely to offer advantages when dealing with larger datasets, while simpler models tend to be more effective for smaller ones. Moreover, when comparing graph models with DL methods that do not incorporate graph structures, we found that GNNs do not always guarantee better prediction accuracy, even though they may enhance interpretability. Thirdly, the distribution of drug combination synergy scores shows a marked concentration near zero, with only a small subset of combinations showing strong synergy. This finding suggests limited clinical translation potential for most pairs. Furthermore, different synergy score metrics showed significant variability in their performance across various datasets, suggesting that no single metric can universally capture the mechanisms of drug interactions. This highlights the need for a more comprehensive evaluation framework that considers multiple synergy metrics. Tools such as SynergyFinder v3.0 have begun to address this need by providing consensus synergy scores that integrate multiple metrics, offering a promising direction for future benchmarking [49]. We acknowledge several limitations: hyperparameters were tuned only on ONEIL using Loewe; several relevant models (e.g., DeepSynergy) were excluded due to compatibility; we did not evaluate cross‑dataset generalization, split by drug/cell line, or account for dose‑range differences. Isolating the effect of individual dataset characteristics would require controlled experiments beyond our scope. Despite these, our study emphasizes the critical role of datasets quality and heterogeneity in drug synergy prediction. Our findings provide valuable guidance for researchers in selecting appropriate models and datasets for their studies, ultimately contributing to more accurate and reliable predictions of drug synergies.

Methods and materials

Dataset collection and preprocessing

In this study, the term “drug combination” refers to a unique triple of (drug A, drug B, cell line), with synergy scores aggregated across all tested dose‑response matrix points (as provided by DrugComb). All counts of drug combinations represent such unique triples, not individual dose‑response data points. We collected drug combination data from DrugComb (version 1.5) which has been the largest public database for in vitro high-throughput combination treatment screening studies [6]. All synergy scores are also sourced from DrugComb, and models are trained only on drug‑cell line pairs with available scores. For drug A, drug B, and cell line triplet combinations, we excluded data points where any element (drug A, drug B, or cell line) was missing. Additionally, within individual studies, we averaged synergy scores across replicate drug combination data to eliminate technical replicates from the datasets.

During our investigation, we identified instances of misuse and misapplication of preprocessed features in existing public datasets, prompting us to reconstruct the feature representations for all DrugComb data entities directly from primary sources to facilitate rigorous datasets benchmark. We obtained the Simplified Molecular Input Line Entry System (SMILES) [50] of drug molecules from Pubchem [51]. Specifically, drug molecules within drug combinations were converted into Morgan fingerprints using RDKit [52]. And cell line features were derived from transcript per million (TPM) expression data obtained from the Cancer Cell Line Encyclopedia (CCLE) [53]. Notably, for cell lines with missing gene expression data in CCLE, we supplemented TPM values from Cell Model Passports (CMP) [54]; expression values across these sources were harmonized by imputing missing entries using the mean available values per gene. All TPM expression data were converted into the $\text{log}(TPM+1)$ form. Then expression values were filtered using the landmark gene set which contains about 1000 carefully chosen genes to prioritize cancer-relevant genes which were selected to capture a broad range of cellular pathways and have been shown to be representative of the transcriptome across many cancer types, including those present in our benchmark. Drug combinations still lacking essential gene expression data and drug SMILES data after this imputation process were systematically excluded from the final datasets. Subsequently, the datasets were randomly split into 5 distinct subsets (folds) at the level of drug combinations. For each fold, subset-specific normalization was applied: the mean and standard deviation of each feature were calculated exclusively from the training set and then used to normalize both the training and the test sets. This procedure rigorously prevents data leakage, as no test-set information is used in the normalization.

Building upon this foundational data processing pipeline, we implemented specific refinements aligned with the original methodological frameworks of state-of-the-art (SOTA) models to ensure consistency across comparative benchmarks. This ensures a fair comparison by respecting each model’s original design choices.

For the CGMS, in addition to Morgan fingerprints, we also calculated the non-zero variance RDKit molecular descriptors of drugs [16]. As CGMS also uses single drug sensitivity data as training data, we also collected single drug sensitivity data from DrugComb and cleaned it as a supplement [16]. And we filtered gene expression data with top 5001 highest-variance genes instead of landmark gene set. Similarly, MTLSynergy also used single drug sensitivity data, so we used almost the same processing method, except that SMILES and drug molecule descriptors were used for drug molecule characterization without modeling into a molecular graph [45]. As for the JointSyn, we used the PaccMann cancer-relevant gene [55] which reports 2128 drug sensitivity-related genes to filter the gene expression data and convert SMILES into molecular graph representation by RDKit. For the DeepDDS, the only difference from the basic processing method is the use of RDKit to convert SMILES into molecular graph representation [22]. In order to maintain the consistency of the drug combination datasets used for model training and testing, it should be pointed out that we supplemented all models with cell line gene expression data from CMP, and only took the mean of repeated combinations in the data, which may differ from the processing method in the original study. We have provided detailed information on data processing in Supplementary Table 4.

Synergy score calculation method

Generally speaking, the task of predicting drug combinations is divided into classification tasks and regression tasks. However, most computational models that currently treat prediction tasks as classification tasks are essentially derived by artificially dividing quantitative collaborative effects based on regression tasks and this boundary line is often unverified and inconsistent [19, 21–25]. So, we chose to treat the prediction task only as a regression task to ensure rigor. In regression tasks, there are four commonly used models for quantifying the synergistic effects of drug combinations: HSA model, Loewe model, Bliss model, and ZIP model. DrugComb provides the four types of synergy scores and we conducted parallel experiments on all models on these four types of synergy scores [6].

Models for predicting drug combinations

In this study, we selected three traditional models (MLP, RF, XGBoost), three SOTA graph deep learning models (DeepDDS, CGMS, JointSyn) and one deep learning model MTLSynergy, which does not use graph neural networks. MLP was built by pytorch (version 1.13.1), RF was built by scikit-learn (version 1.3.0) and XGBoost was built by xgboost (version 2.0.3). And we used Optuna (version 3.5.0) to search hyper-parameters of traditional models [56].

To construct a representative benchmark covering both traditional machine learning and graph‑based deep learning approaches, we selected models according to a structured rationale that considers both methodological paradigms and practical constraints.

Within graph‑based drug synergy prediction, two dominant paradigms exist. The first encodes drug molecules as molecular graphs and learns representations via graph neural networks, as exemplified by JointSyn, DeepDDS. The second constructs heterogeneous graphs that link drugs, cell lines, and other biological entities to enable relational reasoning; CGMS exemplifies this paradigm as well, by incorporating drug sensitivity data as auxiliary nodes. To avoid confounding factors introduced by additional data modalities, we prioritized models that rely primarily on drug chemical structures and cell line gene expression, without incorporating external knowledge graphs, protein–protein interaction networks, or multi‑omics data, unless such data are uniformly available across all datasets. This constraint ensures that performance differences are attributable to architectural choices rather than data advantages.

Within this scope, we applied additional practical criteria. We required that models have publicly accessible implementations that could be integrated into our unified preprocessing pipeline and that they accept standardized input features such as Morgan fingerprints for drugs and preprocessed gene expression for cell lines. We also aimed for coverage of both traditional machine learning approaches, including Random Forest, XGBoost, and MLP, alongside graph‑based methods to enable comparison across complexity levels.

Following these criteria, we included seven representative models: RF, XGBoost, MLP, JointSyn, DeepDDS, CGMS, and MTLSynergy. Several other relevant models, including AuDNNSynergy, SDCNet, AttenSyn, KGANSynergy, and Hypergraphsynergy, were not included. These models typically rely on multi‑omics inputs, external knowledge graphs, or custom data representations that are not uniformly available across our 13 datasets, or their code was not readily reproducible in our environment.

MTLSynergy. MTLSynergy is a multi-task deep learning framework that jointly predicts anticancer drug combination synergy and monotherapy sensitivity within a unified model, integrating both regression and classification objectives. It employs pre-trained autoencoders to compress high-dimensional drug and cell-line features, enabling efficient shared representation learning.

DeepDDS. DeepDDS is an end-to-end deep learning framework that integrates graph attention networks (GAT) or graph convolutional networks (GCN) to encode drug molecular structures with a multilayer perceptron for cell-line gene expression. Here, we only used the GAT that was claimed to perform the best in the study, and the other settings were consistent with the original study.

CGMS. CGMS models each drug combination–cell-line triplet as a heterogeneous complete graph and derives an order-independent whole-graph embedding via a heterogeneous graph attention network, ensuring stable synergy predictions regardless of drug input order. And it trains simultaneously on synergy and monotherapy sensitivity tasks with multi-task learning.

JointSyn. JointSyn introduces a dual-view deep-learning framework that jointly encodes drug combinations and cancer cell lines via a graph-attention joint graph and Morgan-fingerprint–based MLPs, yielding complementary embeddings for robust synergy prediction.

For the SOTA models, we used hyper-parameters consistent with the original study, except in cases where no hyper-parameters are provided in the model, we used grid search to tune within the range given in the original study. All parameter tuning work was conducted on the ONEIL dataset and Loewe synergy scores, as they are the most commonly used datasets and synergy score types in drug synergy prediction [57]. We acknowledge that tuning hyperparameters on ONEIL/Loewe may favor specific model–metric settings and thus does not constitute a fully nested optimization strategy for every dataset and scoring system. We provided the hyper-parameters of models in Supplementary Table 5.

Following an initial hyper-parameter optimization conducted exclusively on the ONEIL dataset using the Loewe synergy score as the prediction target, we applied the resultant fixed hyperparameter set to partition all datasets into five folds for cross-validation. For cross-validation, we randomly partitioned the set of unique drug‑cell line pairs into 5 folds of approximately equal size. Each fold served as the test set once, with the remaining 4 folds used for training. This procedure ensures that no identical drug‑cell line pair appears in both training and test sets, but the same drug or cell line may appear across folds. Finally, the reported performance of the model was the mean value of the evaluation metric across the 5 folds, combined with multiple indicators such as PCC, R², MSE, RMSE, and MAE.

Author contributions

Y.C. collected data, developed the method, analyzed the data, and wrote the manuscript; Q.Y. and L.J. helped to refine the research through constructive discussion and revised the manuscript; Y.K., C.H. and T.H. conceived and supervised the project, interpreted the results, and wrote revisions to the manuscript. All authors have read and agreed to the published version of the manuscript.

Funding

This work was supported by grants from the “Pioneer” and “Leading Goose” R&D Program of Zhejiang (2025C01117), the National Key R&D Program of China (2024YFA1307501), and Central Guidance for Local Science and Technology Development Funds Project (2025ZY01022).

Data availability

The datasets and source code used in this article is available at [https://github.com/cheng1213319/datasets\_across] (https:/github.com/cheng1213319/datasets_across).

Declarations

Competing interests

The authors declare no competing interests.