A guide to transcriptomic deconvolution in cancer

Abstract

Cancer tissues are heterogeneous mixtures of tumor, stromal and immune cells, where each component comprises multiple distinct cell types and/or states. Mapping this heterogeneity and understanding the unique contributions of each cell type to the tumor transcriptome is crucial for advancing cancer biology, yet high-throughput expression profiles from tumor tissues only represent combined signals from all cellular sources. Computational deconvolution of these mixed signals has emerged as a powerful approach to dissect both cellular composition and cell-type-specific expression patterns. Here, we provide a comprehensive guide to transcriptome deconvolution, specifically tailored for cancer researchers, presenting a systematic framework for selecting and applying deconvolution methods, considering the unique complexities of tumor tissues, data availability, and method assumptions. We detail 43 deconvolution methods and outline how different approaches serve distinctive applications in cancer research: from understanding tumor-immune surveillance to identifying cancer subtypes, discovering prognostic biomarkers, and characterizing spatial tumor architecture. By examining the capabilities and limitations of these methods, we highlight emerging trends and future directions, particularly in addressing tumor cell plasticity and dynamic cell states.

Introduction

The central dogma of molecular biology describes the unidirectional flow of genetic information from DNA to RNA to protein, emphasizing the critical role of RNA as a messenger. Unlike DNA, RNA expression levels vary substantially across genes, cells, and biological conditions, providing crucial instructions for protein synthesis. The composition of RNA molecules varies greatly across cell types and states, making gene expression patterns invaluable for understanding cellular functions within tissues¹.

Quantifying RNA molecules across cells and tissues is essential for elucidating the molecular mechanism underlying human diseases. Initial efforts to quantify RNA content at the cellular level began with techniques like Northern blotting² and quantitative PCR (qPCR)³. Additional methods like RNA fluorescent in situ hybridization (FISH)^4,5 and specialized flow cytometry⁶ techniques were later developed to detect RNA alongside protein markers⁷. While pioneering, these techniques were limited to analyzing only a few genes or cells simultaneously. Technological advances have since transformed the field, beginning with microarrays in 1995 ref^8–10 and advancing to short-read sequencing in the 2000s ref^11–14. These approaches typically involve RNA extracted from tissue samples containing thousands to millions of heterogeneous cells. Further advances have been driven by the development of single-cell RNA sequencing (scRNA-seq), which has become a powerful tool for dissecting the complex cellular landscape of cancer^15–18, given the heterogeneous nature of most tumor tissues. By 2015, scRNA-seq became mainstream in biological research^19,20, with continuous further refinements over the following decade.

Cancer is characterized by profound changes in gene expression and pathway activity, which drive its hallmark features²¹. Solid tumors consist of at least three major groups of cells: tumor cells, immune cells, and non-immune stromal cells, including cancer-associated fibroblasts, endothelial cells, and other cells (Box 1). Unlike bulk RNA-seq, which produces a composite signal from these diverse populations, scRNA-seq enables the analysis of individual cell types, providing critical insights into cancer biology. This approach has revealed substantial variability in tumor cell states^22–24, ranging from fully differentiated to undifferentiated, as well as the diverse functional states of fibroblasts²⁵ and immune cells^22,26, highlighting the dynamic nature of the tumor microenvironment (TME). Despite its strengths, scRNA-seq faces several technical and analytical limitations (Fig. 1), which are exacerbated by the complexity of tumor tissues (Box 1). Bulk RNA-seq, while less detailed, is more resource-efficient and more logistically suitable for large cohort studies (Fig. 1), with extensive data available for thousands of patients across various cancers. Therefore, while scRNA-seq excels in resolving cellular heterogeneity, bulk RNA-seq remains indispensable for analyzing large patient cohorts with long-term clinical outcome data, which is essential for establishing prognostic biomarkers.

Box 1.

Unique challenges of transcriptomic deconvolution in cancer

Key features that create significant barriers to accurate transcriptomic deconvolution include:

Tumor heterogeneity and plasticity.

Cancer tissues exhibit remarkable variability in cellular composition at multiple levels, and the proportion of malignant cells can range from less than 10% to over 90% between different patients or among tumor regions of the same patient. Further, tumor cells display exceptional transcriptional diversity^15,166,167 and phenotypic plasticity [G], whereby transitioning between cellular states is associated with vastly different transcriptional landscapes^23,30. This plasticity violates a core assumption of most deconvolution approaches: that reference profiles remain stable across samples. Even when single-cell studies identify recurring cancer cell states, their dynamic nature and tendency to exist along a continuum rather than as discrete entities¹⁶⁸ impedes accurate deconvolution.

Tumors also contain a diverse mixture of infiltrating immune cells, stromal cells, and vascular components, all in varying activation states and proportions. These elements create distinct microenvironmental niches. Cancer-associated stromal cells also exhibit a wide range of activation states that differ markedly from or are absent in their counterparts in healthy tissues^169–173. For example, cancer-associated fibroblasts can exist in at least 4-6 distinct functional states not observed in normal tissues^174–177, while tumor-associated macrophages display polarization signatures beyond the classical M1 and M2 dichotomy observed in normal inflammation^178,179.

While heterogeneity exists in non-malignant contexts¹⁸⁰, even in blood samples, the combinatorial complexity arising from genetic instability, clonal evolution, and microenvironmental remodeling creates unique, cancer-specific challenges for deconvolution. Even within the same tumor type, substantial variation in cellular composition can occur between patients and across different stages of disease progression - at a magnitude far surpassing what is typically observed in normal tissues.

[bH1] Limitations of reference data. A fundamental challenge in cancer deconvolution is obtaining comprehensive reference data for all cell types and subtypes present in tumor tissues. Recent benchmarking studies indicate that cell-type deconvolution methods, when applied to tumor samples, can exhibit reduced accuracy compared to non-cancer applications^51,181. Using peripheral blood-derived immune cell signatures for tumor deconvolution can result in lower accuracy (correlation coefficients as low as r=0.04) compared to using tumor-derived references (r>0.8)¹⁸². This gap in performance due to reference data limitations constitutes a major technical challenge in deconvolution and represents a critical distinction between cancer and non-cancer applications.

Technical challenges in reference generation.

Generating reliable scRNA-seq and snRNA-seq data to create a reliable reference for cancer presents substantial technical hurdles^155,156,158. Low capture efficiency [G] and dropout effects [G] result in sparse expression matrices that may miss key marker genes. Cell doublets can create artificial hybrid expression profiles that confound reference signatures¹⁸³. Tissue dissociation can bias against certain cell populations. These factors can further mask rare cell-type-specific markers and immune cell populations (for example, neutrophils¹⁵⁷) and less characterized stem-like epithelial cells¹⁸⁴ are likely underrepresented in single-cell data¹⁸⁵. These limitations can introduce large biases in cell-type specific estimates, when single-cell data are used as a reference.

Multicollinearity [G] in cell types.

Cell types or states often display highly correlated transcriptional programs, creating multicollinearity issues that confound accurate deconvolution^79,186,187. This is particularly evident in related immune cell states within the TME (for example, different T cell subtypes) and in tumor cells with gradient-like state transitions. Standard deconvolution methods show substantial estimation errors when cell types share significant portions of their marker genes - a common challenge in the heterogeneous tumor microenvironment^74,76. Only a few methods adopted steps to explicitly address this challenge. For example, CIBERSORT uses ν-support vector regression (ν-SVR) with LASSO regularization [G] to select discriminative marker genes and mitigate the effects of collinearity between cell types²⁷, and DeMixT introduced a profile likelihood-based score to select genes that more closely follow the model assumption of distinctions in gene expression between mixing components⁶¹. In general, cell subtypes with collinear gene expression patterns exhibit higher error rates or uncertainty than others⁶⁸. Therefore, the benchmarked deconvolution accuracy of a given method is a fluid concept, as it may decrease substantially when the more granular cell subtypes are of primary interest, when high-resolution deconvolution is required. Other methods like MuSiC implement a hierarchical approach to first deconvolve major cell types and then further deconvolve cell subtypes within each major group, thereby reducing the dimensionality of the problem at the initial step^29,44.

Rare cell population detection.

A recognized challenge in single-cell analysis is that biological discoveries are often limited by difficulties in capturing and profiling rare cell populations that may disproportionately influence the clinicopathological outcome of tumors^{160,161,188,189}. In cancer, examples include stem-like tumor cells, cells of minimal residual disease, specific effector or regulatory immune subtypes, or early metastatic precursors^190–193. It is nearly impossible to estimate, through deconvolution, rare cell populations that have yet to be fully characterized. Even for those that are well characterized, the transcriptomic signal from rare cells is easily obscured by dominant signals from more abundant cell types and background noise [G]. This unfavorable signal-to-noise ratio typically becomes problematic when fractions fall below 1-5%¹⁹⁴. Consistent detection and accurate quantification of very rare populations (that are below 1% within tumors) remains one of the biggest challenges in cancer transcriptomic deconvolution. While computational advances continue to emerge in this area, rigorous benchmarking of method sensitivity for rare populations is still needed. Establishing robust lower detection limits for deconvolution methods, potentially drawing insights from ultra-sensitive techniques in liquid biopsy analysis¹⁹⁵, will be crucial for reliable transcriptomic quantification of clinically significant rare cell populations.

Figure 1. Benefits and limitations of bulk and scRNA-seq data generation.

Schematic overview of the bulk and scRNA-seq data generation. scRNA-seq begins with high-quality tissue samples dissociated into single cells or nuclei. These samples undergo rigorous quality control, both experimentally (for example, cell sorting and filtering out low viability cells) and bioinformatically (for example, excluding cells with low unique molecular identifier (UMI) counts, low gene reads, high mitochondrial content, or doublets). The resulting data is a carefully curated single-cell resolution atlas characterized by high costs, low sample sizes, and complex experimental procedures. Widespread use of scRNA-seq is hampered by several technical challenges. The dissociation of fresh tissues varies in efficiency across cancer types¹⁵⁵, often depletes some cell populations^51,156,157, and can induce substantial transcriptional changes¹⁵⁶. Single-nucleus RNA sequencing (snRNA-seq) can address some of these limitations, enabling the profiling of fragile or adherent cell types, particularly from frozen tissues¹⁵⁸. However, it brings other disadvantages, such as lower transcriptome coverage, loss of cytoplasmic RNA information, and an increased risk of ambient RNA contamination¹⁵⁹. Both scRNA-seq and snRNA-seq approaches are further subjected to shallow sequencing depth and dropout effects, resulting in sparse data¹⁶⁰. From the generated raw data, the manual curation and annotation of single cells involves a series of subjective decision-making steps, where technical effects such as doublets or low-quality cells confound the biological variation of cells¹⁶¹. Batch effects from experimental complexity and patient heterogeneity impede cross-study data integration¹⁶², hindering accurate identification of cell subpopulations. These challenges are further compounded by limitations in clinical application, as most of the available single-cell datasets present a small number of patients and lack long-term follow-up data. 10X scRNA-seq databases like 3CA¹⁶³ (965 patients from 76 studies), and TISCH2 ref¹⁶⁴ (969 patients from 139 studies) showed notably smaller cohort sizes, and neither of the databases provide clinical information. Researchers may have to perform extensive literature review to identify scRNA-seq data linked to clinical metadata (see Supplementary Table 1 for details of scales of data availability for four major cancer types). In contrast, bulk RNA-seq can be applied to a variety of sample types, including fresh, fresh-frozen, and FFPE tissues. The process involves tissue lysis followed by sequencing, allowing for application across large cohorts due to its cost-effectiveness, low technical variability, robustness to sample quality, and straightforward experimental protocols. Bulk RNA-seq, while confounded by cellular heterogeneity, benefits from long-term clinical follow-up. As of December 2024, cBioPortal¹⁶⁵ (a database for bulk cohorts) contained data from over 20,000 patients from more than 80 studies spanning various cancer types, with survival information available for 86% of the cases. By integrating both types of sequencing data through deconvolution, researchers can mitigate the limitations of each approach, leveraging the comprehensive resolution of scRNA-seq with the broad applicability of bulk RNA-seq to enhance the analysis of complex tissue samples.

The integration of bulk RNA-seq and scRNA-seq data offers a compelling opportunity to advance cancer research by leveraging the strengths of both methodologies. However, bulk RNA-seq data analysis requires interpreting signals derived from a mixture of cell types, which is an important challenge. Mathematical deconvolution [G] addresses this challenge by separating these complex signals into their constituent cell-type-specific components. Overall, current deconvolution methods enable the estimation of cell-type proportions, with some methods further estimating cell-type specific gene expression profiles, thereby extracting detailed biological insights from bulk datasets.

Many deconvolution methods were initially developed and validated using non-cancer tissue samples^27–29, where cellular composition remains relatively stable and cell types maintain consistent gene expression patterns. However, analyzing cancer tissues adds unique complexities (Box 1), including tumor heterogeneity, cellular plasticity and intercellular interactions^23,24,30,31, which were mapped with the help of spatial sequencing technologies^32–35. These features underscore the need for specialized approaches or careful adaptation of existing deconvolution methods in cancer research.

In this Review, we emphasize the pivotal role of transcriptomic deconvolution in harmonizing bulk and single-cell data from cancer, facilitating the estimation of individual cell type contributions, and mapping these insights to clinical outcomes. For the purpose of this review, we define deconvolution as an in silico procedure that uses bulk RNA-seq data to produce cell-type- or compartment-specific measurements, such as absolute or relative counts of cells, or corresponding gene expression levels. We aim to equip cancer researchers with bioinformatics knowledge necessary to fully utilize bulk RNA-seq data, ultimately enhancing our understanding of tumor biology. We guide researchers to effectively navigate through the process of performing deconvolution, select the optimal method(s) based on their specific research question, and understand future developments in the field. To achieve this, we first provide an overview of the available deconvolution techniques, including reference-based, semi-reference-based, and reference-free approaches, along with insights from benchmarking studies; we then detail applications of transcriptomic deconvolution in cancer research, covering estimation of TME cellular composition, tumor subtype identification, extraction of tumor-specific expression profiles, and prediction of patient outcomes and treatment responses; and finally, discuss future directions in methodology development and clinical translation.

Overview of transcriptomic deconvolution techniques

Transcriptomic deconvolution techniques can be broadly categorized into three approaches: reference-based methods, semi-reference-based methods, and reference-free methods (Fig. 2, Table 1). These methods differ in their reliance on prior knowledge and the type of reference data (Box 2) they require, each offering distinct advantages and disadvantages influenced by the biological context under investigation.

Figure 2. Conceptual overview of bulk RNA-seq deconvolution methods.

High-level classification of deconvolution methods based on application scenarios and computational models. Deconvolution methods are categorized into reference-based, semi-reference-based, and reference-free approaches. The first row illustrates reference-based methods, where individual methods employ one of the following computational frameworks [We have specified how each approach employs the framework]: probabilistic models, Non-negative Matrix Factorization (NMF), linear regression, and deep learning frameworks. The second row depicts semi-reference-based methods, combining probabilistic models and NMF. The third row represents reference-free methods, which either utilize pre-defined cell type signatures, leverage gene signatures or operate without any reference, primarily relying on NMF. The outputs are classified into cell-type proportions (solid box, provided by all methods) and deconvolved cell-type-specific expression profiles (dotted box, provided by a subset of methods).

Table 1. Summary of bulk transcriptomic deconvolution tools classified by their underlying principles.

Method	Reference (in addition to the bulk expression data)			Output				Statistical modeling	Applicability a
	Ref. matrix	Marker gene list	Adjacent normal tissue sample	Cell Fraction	Mean expression	Per-sample expression	Marker gene list		Cancer cell-specific	Immune cells in cancer tissues	Non-cancer specific / General
Bulk-reference based
CIBERSORT27	□ b	–	–	□	–	–	–	Support vector regression (SVR)	–	□	□
DeconRNASeq66	□			□				Non-negative constrained least square regression	–	□	□
EPIC28	□ b			□				Constrained least square regression	□	□	□
FARDEEP136	□			□				Adaptive least-trimmed-squares regression	–	□	□
GEDIT137	□			□				Non-negative linear regression		□	□
MIXTURE101	□			□				v-SVR		□	□
PERT138	□			□				Non-negative maximum likelihood		□	□
quanTIseq40	□ b			□				Constrained least square regression		□	–
TIMER39	□ b			□				Constrained least square regression		□
Single-cell-reference based
AutoGeneS41	□	–	–	□	–	–	–	SVR	–	□	□
BayesPrism46	□			□	□	□		Gibbs sampling	□	□	□
Bisque42	□			□	–	–		Non-negative least squares	–	□	□
								(NNLS)
GCDseqR48	□			□	□			Bayesian hierarchical model	–	□	□
CIBERSORTx55	□			□	□	□	□	SVR	□	□	□
CPM139	□			□	–	–	–	SVR		□	□
deconvSeq140	□			□				Generalized linear model		□	□
DeMixSC53	□ b			□				Weighted NNLS (wNNLS) trained with benchmark data from ovarian cancer	□	□	□
DWLS43	□			□				Weighted non-negative least squares (wNNLS)	□	□	□
GTM-decon50	□			□	□		□	Guided topic model	–	□	□
Kassandra102	□			□	–		–	Supervised machine learning	□	□	□
MuSiC2.044	□			□				wNNLS	–	□	□
MuSiC29	□			□				wNNLS		□	□
ProM141	□			□				wNNLS		□	□
RNA-Sieve142	□			□				Maximum likelihood estimate (MLE)		□	□
Scaden49	□			□				Deep learning trained with pseudo-bulk		□	□
SCDC45	□			□				wNNLS		□	□
SQUID52	□			□				Linear transform with dampened weighted least squares	□	□	□
Semi-reference based
BaylCE47	□	–	–	□	□	–	–	Bayesian hierarchical model	□	□	□
Deblender143	□			□	□		□	Least squares NMF Quadratic programming (QP)	□	□	□
DeCompress64	□			□	□			Compressed sensing	□	□	□
DeMix60	–		□	□	□	□		MLE	□c	–	–
DeMixT61			□	□	□	□		Iterated conditional modes (ICM) for MLE	□c	□	–
ISOpureR62			□	□	□	–		Dirichlet-multinomial mixture model	□c	–	–
NITUMID144	□		–	□	□			Semi-supervised NMF	□	□	□
PREDE63	□			□	□			Iterative constrained QP with NMF	□	□	□
SECRET145	□			□	–			Weighted constrained nonlinear optimization	□	□	□
Semi-CAM146	□	□		□	□			Convex analysis of mixtures, semi-NMF	–	□	□
Reference free
CAM70,147	–	–	–	✓	✓	–	✓	Convex analysis of mixtures (CAM)	–	✓	✓
DeClust71				□	□		□	Iterative optimization minimizing mean squared error (MSE)	□	–	–
deconf65				□	□		–	Least squares non-negative matrix factorization (NMF)	–	□	□
linseed68				□	–		□	Simplex corner detection with fast combinatorial non-negative least squares (NNLS)	□	□	□
MMAD148				□	□		–	MLE	–	□	–
TOAST69				□	–		□	Least squares NMF	□	□	□

Available transcriptional deconvolution methods relevant to cancer are listed, including those published up to December 2024. Additional methods that fall under at least one of the following categories were excluded from the table: (1) focus on specific non-cancer tissue types, limiting applicability to tumor deconvolution; (2) proposed as immediate extension beyond existing approaches; (3) primary application to non-transcriptomic data types (for example, methylation data); and (4) lack of validation in cancer contexts or over established methods. For completeness, those additional computational deconvolution methods that are not included in this table but were included in a recent methods review⁷⁹ are provided in Supplementary Table 2.

^a)Applicability classifications are based on each tool’s intended design purpose and/or demonstrated effectiveness in published studies (original method papers and independent benchmarks^{51,52,58,72,74,76,149–153}). Check marks (✓) indicate either: (1) the method was specifically designed for that application, or (2) the method has been successfully validated for that application in published studies. Absence of check marks may reflect design limitations or lack of validation studies rather than definitive unsuitability.

We discern three categories: tumor-specific deconvolution (estimates cancer cell proportions/expression profiles, and/or identifies cancer cell subtypes), immune cell-type estimation (quantifies immune cell populations in cancer tissues), and non-cancer specific or general cell type deconvolution (applicability to healthy tissues and other biological settings not yet demonstrated in cancer context).[Au: Perhaps the 3 applicability need to be supported by references? Maybe instead of the check mark, a specific reference number can be added? Reply: We attempted to add specific references for each application but found it impractical due to varying levels of validation across methods. Some methods like CIBERSORT have dozens of studies per application category, while others have only been demonstrated in original/benchmark studies. Adding references would clutter the table and compromise its function as a quick comparison tool. We do confirm that all checkmarks are based on at least one research paper. ]

^b)The method has a built-in reference matrix without user input.

^c)The method has been demonstrated to output the cancer cell-specific transcript proportion.

Box 2.

Choice of reference data

We define reference data broadly as input data that are required in addition to the raw mixed expression matrix to perform deconvolution. Most methods, except for reference-free deconvolution, require some reference data (Table 1). These come in varying formats: as a matrix of signature or marker gene expression per cell type; single-cell derived pooled mean expression across genes per cell type; or gene expression matrices for individual cell types (with incomplete cell types for semi-reference-based and complete cell types for reference-based deconvolution). The choice of these input data is critical, and therefore we provide the following considerations.

Marker genes and signature genes.

These two concepts are often used interchangeably to represent stably expressed genes that are specific for a given cell type, hence whose expression levels can be used to form a reference matrix. Historically, the term marker gene was also used to annotate a gene that is exclusively expressed in one cell type, for example, for some immune cell types³⁶; these are ideal markers for regression-based deconvolution methods. However, it is challenging to identify stably expressed genes for tumor cells in individual cancer types. Although several studies developed approaches to find these signature genes^75,196, more investigations are warranted.

Comprehensive cell type and tissue representation.

Effective cancer transcriptome deconvolution requires reference data that captures the full spectrum of relevant cell types within their tissue-specific transcriptional context. For example, immune reference panels such as LM22²⁷, derived from PBMCs, have been widely used to profile immune infiltration across tumors. However, these references lack non-immune components and fail to reflect the tissue-specific transcriptional profiles of immune cells themselves, limiting their accuracy⁷⁶. This limitation becomes more pronounced for non-immune populations such as epithelial cells or cancer-associated fibroblasts, which exhibit broader transcriptional diversity and stronger tissue dependence. Unlike many non-cancer applications where one generic reference may suffice, cancer deconvolution demands tissue- and context-specific references to capture the heterogeneity of both primary and metastatic tumors. Existing references often underrepresent rare, transitional, or site-specific cell states that play critical roles in tumor biology. Expanding reference datasets to cover a wider range of cell types, tissue origins, tumor subtypes, and treatment conditions could enhance deconvolution accuracy, though such efforts must be weighed against the complexity of data collection and integration.

Matching sequencing platforms.

Recent studies demonstrated the importance of training the deconvolution model using the measurable discrepancy between platforms in single-cell versus bulk RNA sequencing from the same tumor samples^51,53. This type of advanced reference data is yet to be generated for most tumor tissues, due to experimental challenges.

Reference data provided by the tool.

From the users’ perspective, not having to develop and provide appropriate reference data is a desirable feature of a deconvolution tool. Although this is not possible for many tools, some do provide such utility (annotated methods in Table 1). This useful feature comes with a tradeoff that the internal reference data can generate biased results when they are applied to bulk data that do not align with the key model assumptions. There are two main types of internal reference data: marker genes and gene expression levels for major immune cell types²⁷, and histology-type specific single-cell and bulk RNA-seq paired benchmark data^51,53.

Reference-based deconvolution methods

Reference-based deconvolution is a widely used strategy for analyzing complex mixtures of cell types in heterogeneous tumor tissue samples. This approach relies on predefined reference data over a complete list of cell types, in the form of a gene-by-cell-type expression matrix [G] that captures the expression patterns of distinct cell types within the bulk tumor samples to be deconvolved (Table 1). Earlier work utilized genes uniquely and abundantly expressed in one cell type at a time³⁶, and later used bulk-derived references created from bulk sequencing of purified or sorted cells^27,37,38. More recent work is focused on using scRNA-seq-derived references, which are created by averaging gene expression across cells of the same cell type.

Many bulk reference-based deconvolution methods use constrained least squares regression (for example, TIMER³⁹, quanTIseq⁴⁰, and EPIC²⁸) or support vector regression (CIBERSORT²⁷). The advent of scRNA-seq enhanced the resolution of reference profiles, allowing the development of more refined deconvolution methods. Most single-cell reference-based methods^29,41–45 employ weighted non-negative least squares [G] (wNNLS) with single-cell-based reference matrices to iteratively estimate cell-type proportions from bulk mixtures, differing in their preprocessing of the reference matrices and the weight functions in the wNNLS framework. Other modeling frameworks include hierarchical Bayesian models [G] (BayesPrism⁴⁶, BayICE⁴⁷, and CDseqR⁴⁸), a deep learning framework trained on large-scale single-cell simulated bulk samples (Scaden⁴⁹), and a guided topic modeling approach [G] (GTM-decon⁵⁰). Guided topic modeling identified topics instead of cell types from scRNA-seq data, hence providing an attractive alternative that does not depend on manual cell-type annotations, albeit at a cost of losing some biological interpretability.

A crucial area for further development is addressing the inherent technical differences between bulk and scRNA-seq platforms. These differences lead to divergent gene expression measurements from the same set of cells, which undermines the accuracy of current single-cell-based deconvolution methods^51–54. Recent efforts have specifically addressed the technical differences between scRNA-seq and bulk RNA-seq platforms. CIBERSORTx⁵⁵ uses a batch correction approach called S-mode that transforms single-cell reference profiles to match the distributional properties of bulk expression data. SQUID⁵² leverages a linear transformation matrix⁴² derived from paired single-cell and bulk samples to enhance cross-platform compatibility. DeMixSC⁵³ incorporates matched benchmark datasets where both single-cell and bulk sequencing are performed on the same tumor samples, enabling direct calibration of the deconvolution model to account for platform-specific biases. These approaches help mitigate the technical artifacts that would otherwise compromise deconvolution accuracy when integrating data across platforms.

While this Review primarily focuses on deconvolution methods that directly estimate cell-type proportions, we briefly note that gene set enrichment analysis (GSEA)-based approaches such as xCell³⁷, ESTIMATE⁵⁶ and MCPcounter⁵⁷ provide a different perspective on cellular composition. Unlike regression-based deconvolution methods that estimate absolute proportions of cell types within each sample, GSEA-based methods measure the relative abundance of cell types across a group of samples. These methods compute enrichment scores using predefined gene sets characteristic of different cell types or tissue compartments. For instance, ESTIMATE⁵⁶ provides stromal and immune scores that indicate the relative presence of these compartments across tumor samples, but cannot determine the absolute fraction of these components within any individual sample. A key drawback of enrichment methods is the lack of specificity: gene sets from one cell type or cell state can easily include genes shared by related cell types/states, potentially leading to ambiguous interpretations^37,58,59. Because the gene sets are predefined, the enriched cell types may not include the nuanced cell subtypes of interest. On the other hand, GSEA-based methods circumvent the need to collect external reference data by directly utilizing pre-defined gene sets as their reference system, hence they are straightforward to run and can be useful in exploratory studies, large-scale cohort analyses, and when reference-quality single-cell data are unavailable for the tissue or cancer type of interest. These approaches are especially valuable for rapid assessment of immune infiltration patterns across heterogeneous datasets or when investigating understudied cancer types with limited characterized cell states. However, GSEA-based approaches may be less appropriate when precise quantitative estimates of cell-type proportions are required, when studying rare or novel cell subpopulations not captured in predefined gene sets, or when investigating tumor-specific cellular states that require tailored reference profiles.

Semi-reference-based deconvolution methods

Semi-reference-based deconvolution has become an invaluable tool in cancer research, addressing the specific challenges posed by tumor cell plasticity and heterogeneous tissue samples. Semi-reference-based methods are most useful when reference expression profiles for the complete list of cell types are unavailable, as they can work with incomplete or partial data. Specifically, ´partial´ in this context refers to gene expression profiles or marker gene lists that represent only a subset of the cell types or genes that would typically be included in a full reference profile. Several approaches have been developed to tackle this complex task: DeMix⁶⁰ used an iterative heuristic search for maximum likelihood estimation. This was later superseded by DeMixT⁶¹ using a more principled Iterated Conditional Modes (ICM) [G] algorithm, iteratively refining estimates of cell proportions and gene expression profiles to maximize the likelihood of the observed data. ISOpureR⁶² applies a Dirichlet-multinomial mixture [G] model to estimate cancer cell proportions, maintaining accuracy even with varying levels of tumor purity. PREDE⁶³ enhances cell-type specificity through a Non-negative Matrix Factorization [G] (NMF) framework, accounting for small variances in gene expression. DeCompress⁶⁴ integrates linear subspace identification [G] with compressed sensing techniques [G] to parse complex cellular signals more effectively. BayICE⁴⁷ further enhances the capabilities of semi-reference-based deconvolution by employing a hierarchical Bayesian model that simultaneously estimates cell-type proportions, expression profiles, and novel gene signatures. This approach addresses issues such as shift-invariance [G] to facilitate unbiased estimates of cellular components.

However, easing the data requirements for detecting novel biology comes with a major trade-off, leading to reduced model identifiability and exacerbating the curse of dimensionality. Both are well-known statistical challenges that destabilize parameter estimation. Therefore, some of these methods^47,63 ignore the expression variation and fix cell-type expression at the mean level for the partial references, sacrificing accuracy for robust estimation. Others^60,61 model both mean and variance of cell-type-specific expression levels and employ additional techniques to tackle these challenges, limiting to two or three components to avoid the parameter space becoming excessively large.

By leveraging bulk RNA-seq data alongside partial reference profiles, these methods may offer significant flexibility and cost-efficiency, allowing researchers to deconvolve both well-defined and uncharacterized cell types, thereby providing critical insights into the complex and dynamic nature of tumors where full reference profiles are often unavailable. The corresponding biological scenarios include investigation of highly variable tumor cells or fibroblasts.

Reference-free deconvolution methods

Reference-free deconvolution methods are desirable as they require no additional reference data for any cell types, hence in principle they are the most generalizable and widely applicable. So far, approaches that fall under this category focused on first identifying marker genes from the bulk expression data to be deconvolved and then using these markers to estimate cell-type-specific proportions and expression. The development of these approaches was pioneered early in the field, driven by the natural mathematical framework provided by NMF and the attractive potential for broad application without reference requirements. However, earlier methods (for example, deconf⁶⁵, DeconRNASeq⁶⁶) were limited to deconvolving one patient sample at a time, thereby lacking capability of borrowing strength across samples for unbiased parameter estimation. Early approaches are also too generalized to take in any prior biological knowledge. These disadvantages may contribute to a diminished interest in further developments while semi-reference and reference-based methods were shown to be effective in utilizing marker gene expression. More recently, however, with the successful adaptation of scRNA-seq studies⁶⁷, NMF has been revisited for reference-free deconvolution. For example, Linseed⁶⁸ leverages the mutual linearity of tissue-specific genes to uncover the topological space necessary for NMF deconvolution and TOAST⁶⁹ builds on deconf⁶⁵ but iteratively identifies cell-type-specific features and performs deconvolution. In addition to NMF, CAM⁷⁰ uses convex analysis [G] to geometrically identify extreme points (vertices) of the data structure, which correspond to subpopulation-specific marker genes. Another example is DeClust⁷¹, which employs prior markers to determine the proportions of immune and stromal cells, before stratifying patients based on these cellular compartments.

While flexible and broadly applicable, reference-free methods, at the current time, often underperform compared to reference-based tools when suitable reference data is available⁷². Nevertheless, these methods remain valuable for exploratory studies and can be particularly useful in cancer types that lack well-characterized reference profiles, where their application may provide initial insights into cellular composition without relying on predefined reference data⁷³.

Lessons from benchmarking studies

The unique challenges of cancer transcriptome deconvolution (Box 1) and correspondingly higher demands on the reference data generation (Box 2) require tissue- and context-specific benchmarking approaches to mimic the complex real-data scenarios as closely as possible. While there have been extensive benchmarking efforts (Fig. 3A) to evaluate the performance of deconvolution methods, only a handful of studies^{40,51,53,74,75} took the unique features of cancer tissues into consideration. Other non-cancer-specific benchmarks, for example, based on peripheral blood mononuclear cells (PBMCs) or cell-line mixtures without cancer cells (Box 1), may inform what the worst performing model types under the specific data scenario are, but limited inference can be made for future applications to cancer studies. Overall, creating a gold-standard dataset for evaluating deconvolution methods in cancer is a major unresolved challenge, requiring a compromise of transferability to real-data scenarios for experimental feasibility (Fig. 3A).

Figure 3. Lessons from benchmarking studies.

(A) Overview of the current benchmarking designs. Current experimental designs to generate benchmark datasets for evaluating deconvolution methods vary largely in terms of the technical difficulty in data generation as well as the transferability of their conclusions to real data scenarios. Simulated mixtures (in silico) such as those generated by computational aggregation of single-cell RNA sequencing (scRNA-seq) samples provide convenient but oversimplified conditions. Cell line mixtures (Experimental approach) with predefined proportions of cultured cell lines or purified primary cells provide controlled yet biologically relevant complexity. Transferability to real data depends on the complexity of the design and what type of cell lines are being mixed. Experimental approaches offer increasing biological realism. Bulk RNA-seq paired with immunohistochemistry (IHC) or flow cytometry (both experimental approaches) offers cell composition inferences from adjacent sections (IHC) or flow cytometry of the bulk sample. Matched bulk and single-cell RNA-seq data derived from one aliquot of the same tissue sample captures the full biological and technical complexity, enabling benchmarking and providing robust assessment. (B) To benchmark semi-reference-based methods, we recommend using cell line mixtures with controlled ratios, enabling precise quantification of both cell counts and RNA content. This dual measurement approach allows evaluation of cell count proportions and cell transcript proportions using the methods of interest. (C) For benchmarking reference-based methods, we recommend generating matched data from tissue samples that are dissociated into single-cell or single-nucleus suspensions and divided into two aliquots. Both undergo identical poly(A) mRNA enrichment protocols. This matched approach minimizes confounding effects from tissue digestion differences and technical biases, enabling simultaneous benchmarking and reference construction. (D) Recommended methods based on previous independent benchmarking studies, whose designs were limited to evaluating reference-based deconvolution methods.

Building on studies using the current benchmarking designs^{51–53,75,76} in balancing between real-data applicability and experimental feasibility, insights into the performance of deconvolution methods can be obtained across a spectrum of validation frameworks⁷⁷. For semi-reference-based methods, controlled cell-line mixtures serve as a reasonable benchmarking strategy for its ability to distinguish between cell-type proportions and cell-type-specific RNA transcript proportions (Fig. 3B). In DeMixT⁶¹ and its application to pan-cancer datasets⁷⁵, this approach was employed to accurately assess method performance with known cellular ratios and to measure tumor-cell transcript outputs. For reference-based methods, a reasonable and achievable approach involves generating matched bulk and single-cell data from the same dissociated cell suspension using identical library preparation protocols (Fig. 3C). The paired bulk and scRNA-seq design enables more accurate benchmarking with aligned ground truth and enhances deconvolution performance through the use of context-matched reference profiles. This approach, implemented by Guo et al.⁵³ (as part of reference data) in retina, by Hippen et al.⁵¹ in ovarian cancer tissues, and most recently in human prefrontal cortex⁷⁸, minimizes confounding effects from tissue digestion differences and technical biases between platforms. Finally, current benchmarking efforts reveal that reference-free methods underperform by 15-25% compared to reference-based approaches in cancer applications, while semi-reference-based methods lack sufficient independent validation studies⁷⁹. We therefore provide recommendations for reference-based methods where multiple independent benchmarking studies, including cancer-specific evaluations, have been performed (Fig. 3D).

More efforts on designing and generating cancer-specific benchmarking datasets is needed to gain further insights across all categories of methods⁵⁴. The ideal situation of having one perfect benchmarking study may not be attainable due to the exceptionally high transcriptional variability across cancer tissues (Box 1). Therefore, interpreting model output under a biological context, with or without the ideal benchmarking, becomes ever more important.

A framework for deconvolution in cancer

Different biological or clinical questions drive the diverse transcriptome deconvolution needs of researchers. Using our systematic decision path (Fig. 4), researchers can navigate key checkpoints to select appropriate deconvolution strategies based on data availability, model assumptions, and specific research objectives.

Figure 4. A guide to cancer transcriptomic deconvolution: from study design to method selection.

(A) A decision tree guiding the selection of an appropriate transcriptomic analysis strategy based on available data. When internal single-cell RNA sequencing (scRNA-seq) data are accessible, researchers can directly investigate novel cell states, assess cell-type composition across conditions, or identify cell-type-specific signature genes, optionally integrating with bulk RNA-seq data if statistical power is insufficient or for external validation. Without internal scRNA-seq data, hypotheses can be generated using differential expression analysis (DEA) or gene set enrichment analysis (GSEA) from bulk RNA-seq cohorts, potentially augmented by publicly available scRNA-seq datasets. Subsequently, bulk RNA-seq deconvolution can be performed, and the analysis can be directed towards either a comprehensive characterization of the tumor microenvironment (TME) or a tumor cell-specific characterization. To validate TME-specific signals, researchers often employ techniques like immunohistochemistry (IHC) and cytometry by time-of-flight (CyTOF). To validate tumor cell intrinsic properties, such as stem-like cell states, researchers often use in vitro assays (for example, genetically modified cell lines) and in vivo models such as patient-derived xenografts (PDX) or genetically engineered mouse models (GEMMs). (B) A decision tree for selecting appropriate deconvolution method once the need for deconvolution has been established. For TME cell analysis (blue shading), single-cell reference-based methods are recommended, with its accuracy dependent on the availability of tissue-matched bulk or single-cell data. For tumor cell analysis (red shading), reference-based methods, semi-reference-based methods, or reference-free approaches are available depending on data type:. Semi-reference-based deconvolution methods are advantageous to study tumor intrinsic activities as they impose no constraints with tumor cell expression profiles. Hence, even when scRNA-seq data are available, semi-reference-based methods should also be applied. Other more nuanced tumor-specific signals will likely be discovered through reference-free approaches. Finally, the availability of other measurements of the tumor samples to benchmark cell-type specific signals may be accounted for when prioritizing the deconvolution strategy.

After applying suitable deconvolution methods to bulk data, researchers can extract insights from the primary cellular or molecular data (for example, TME composition, tumor subtypes, and tumor-specific expression profiles) and then perform downstream analysis for clinical or biological investigations (e.g., prognostic biomarkers and treatment response predictors). The following sections explore these applications in detail, demonstrating how deconvolution bridges fundamental cancer biology and clinical decision-making.

Estimation of tumor microenvironment cellular composition

The TME is a complex interplay of immune cells, stromal cells, and extracellular matrix components that influence tumor progression, immune evasion, and response to therapy⁸⁰. Deconvolution methods offer powerful tools to dissect this complexity and quantify the cellular composition of the TME.

Immune profiling through deconvolution is one of the most advanced applications in TME studies. Immune cells are characterized by relatively small transcriptomes and distinct gene expression patterns, reflecting their specialized functions^81–83. For example, CD3D and CD3E are highly expressed in T cells, CD19 and membrane spanning 4-domains A1 (MS4A1, encoding CD20) in B cells, CD68 and CD163 in macrophages, and ITGAX (also known as CD11c) in dendritic cells. More detailed marker genes include CD8A and CD8B and granzyme B (GZMB) for cytotoxic T cells, CD4 and interferon gamma (IFNG) in helper T cells, forkhead box P3 (FOXP3) and cytotoxic T-lymphocyte associated protein 4 (CTLA4) for regulatory T cells, and CD38 and syndecan 1 (SDC1, also known as CD138) for plasma cells⁸⁴. The application of reference-based deconvolution (Table 1) for immune profiling has been successful since its inception in 2016, where bulk expression profiles of individual immune cell types extracted from PBMCs have been used as reference datasets^27,39. Dissecting specific immune cell types has provided critical insights into how various immune cells influence tumor behavior. For instance, using transcriptomic deconvolution approaches including CIBERSORT and immune gene expression signatures, six distinct immune subtypes across 10,000 tumors from 33 cancer types were identified, revealing the complexity of immune landscapes within tumors⁸⁵. Using CIBERSORT²⁷ to analyze tumor-associated leukocytes across a broad spectrum of human cancers, a pan-cancer map was constructed that identified specific immune cell populations⁸⁶. This study highlighted the distinct roles of tumor-associated neutrophils and plasma cells in the tumor microenvironment.

Understanding the dynamics between tumor, immune, and stromal components is of essential importance, but it is more difficult to quantify these components jointly. This complexity arises from the large transcriptome size and high transcriptional variability observed in fibroblasts, endothelial cells, and epithelial cells⁷⁵. Tumor cell plasticity further introduces considerable variation (Box 1) in tumor-cell-specific expression, often violating the core assumption of reference-based methods that a mean reference profile reliably represents cell-type specific transcriptional activity. In contrast, semi-reference-based and reference-free methods have provided a more adaptable strategy for deconvolving these complex components, as they allow the tumor cell component to remain unknown, offering greater flexibility in analyzing stromal elements (Table 1). For example, DeMixT⁶¹ was successfully applied to head-and-neck squamous cell carcinoma samples from the Cancer Genome Atlas projects (TCGA) to estimate proportions of tumor, stromal, and immune components using only normal tissues as reference for stromal cells, while keeping the tumor component unconstrained. Similarly, BayICE⁴⁷ was successfully applied to deconvolve non-small-cell lung cancer samples by using reference profiles for immune cells and normal lung tissue, while leaving the highly variable tumor component unconstrained. These approaches demonstrate how semi-reference-based methods can overcome the limitations of reference-based approaches when analyzing the full complexity of the TME.

With the increasing availability of scRNA-seq data, researchers can now employ a two-step strategy: first analyzing scRNA-seq data to identify and annotate the major cell types present in a specific tissue and then using these annotated cell types to create reference profiles for deconvolution of bulk RNA-seq data from the same tissue type (Table 1). This approach enables the identification of tissue- and biology-relevant cell subtypes and provides more reasonable reference profiles for the matched tumor cells. In a study on pancreatic ductal adenocarcinoma, scRNA-seq was leveraged to pinpoint a distinct interleukin 1β (IL-1β)⁺ macrophage subpopulation marked by elevated expression of inflammatory genes such as IL1B, tumor necrosis factor (TNF), and CXC motif chemokine ligand 8 (CXCL8)⁸⁷. By applying CIBERSORTx⁵⁵ to deconvolve bulk transcriptomic datasets from pancreatic adenocarcinoma using single-cell annotations as a reference, the prevalence of this macrophage subtype was found to substantially contribute to the inflammatory landscape of the tumors.

Spatial transcriptomic technologies provide an important dimension to TME profiling by offering spatially resolved gene expression, enabling the investigation of the spatial organization and interactions between diverse cell types within the complex tumor architecture (Box 3). Deconvolution approaches for spatial transcriptomic data are valuable as they can reveal cellular composition and functional states while preserving crucial information about physical locations and proximity relationships between different cell populations. Advances in this rapidly evolving field will further enhance our understanding of tumor heterogeneity, cell–cell communication networks, and response to therapies in a native spatial context.

Box 3.

Integrating spatial context: deconvolution in spatial transcriptomics

While bulk RNA-seq deconvolution supports dissecting TME components at the patient level, recent advances in spatial transcriptomics [G] provide powerful tools to investigate the spatially-defined transcriptome¹⁹⁷. Among the available platforms with high transcript throughput, several of the widely adopted spatial platforms (including the currently popular 10X Visium platform^198,199) have multicellular resolution, where each bulk spatial unit, or ´spot´, contains multiple cells. Thus, for these data, computational tools for spot-level deconvolution to resolve the cellular composition within each spot are needed to refine the analysis of spatial heterogeneity in cellular architecture and tumor ecosystems⁵⁴. These methods will facilitate the integration of spatially-resolved phenotypes with clinical outcomes, ultimately providing a more comprehensive understanding of the TME²⁰⁰.

Extending from bulk deconvolution methods, tools for spatial transcriptomics spot deconvolution require a bespoke design for sparse, spatially resolved data. These methods can be similarly categorized based on reference requirements²⁰¹ and share similar methodological strengths and weaknesses (Figure). The computational frameworks underlying these approaches include probabilistic models that use statistical distributions to quantify uncertainty in cell-type assignments, NMF that decomposes gene expression matrices while maintaining biological interpretability, linear regression approaches that model spot expression as weighted combinations of cell-type profiles, and deep learning methods that capture complex non-linear relationships between expression patterns and cellular composition.

Reference-based approaches utilize scRNA-seq data to deconvolve spots into fractions of different cell types²⁰² or map single cells to individual spots²⁰³. Semi-reference-based²⁰⁴ and reference-free²⁰⁵ approaches focus on cell fraction deconvolution when reference data are limited or incomplete[Au: Please expand the text a bit describing the “Applications” shown in the Figure. In the current version, it is not becoming clear whether ´Immune infiltrated´ is better assessed with reference-based approaches or if this can be assessed with either approach. Please specify. Given the nascency of this field, limited biological discoveries using these methods have been reported. In one notable example, Ma et al.³² uncovered the distribution of subtypes of cancer-associated fibroblasts and their role in shaping an immunosuppressive TME. However, most studies utilizing this approach remain exploratory with at least a 10-fold smaller sample size at the patient level, indicating that the field is still in the early stages of direct clinical translation^206,207. Integrating spatial transcriptomics data with data rich in sample size and clinical information, e.g., bulk RNA-seq, may offer broader applications. For a more comprehensive exploration of spot-deconvolution technologies, benchmarking, and applications, readers are directed to recent reviews^{151,200,208–211}.

Identifying subtype-specific features through deconvolution

Deconvolution techniques can be employed to stratify tumors into distinct subtypes, driven by either intrinsic tumor cell properties or by interactions between tumor cells and their microenvironment. While traditional clustering of molecular profiles across patients for subtyping has been very successful^88,89 — exemplified by methods like the well-established breast cancer subtyping method prediction analysis microarray 50 (PAM50)⁹⁰ and consensus molecular subtypes (CMS)⁹¹ for colorectal cancer — deconvolution offers the potential to uncover new subtypes through more granular analysis of cellular composition that is often masked in bulk expression profiles.

To subtype tumors based on the TME, or to identify tumor ecosystem subtypes, methods that accurately quantify the complex cellular interactions within the TME are essential. Reference-based methods, such as CIBERSORT²⁷ and its single-cell-based version, CIBERSORTx⁵⁵, are widely used for estimating cell-type proportions using cell-type-specific gene signatures. CIBERSORTx⁵⁵ offers additional capabilities, including the identification of multicellular patterns, referred to as ´ecotypes´ within tumors. These ecotypes are recurring combinations of cell types and states that represent unique cellular environments that correlate with specific tumor subtypes and biological behaviors.

For instance, applying CIBERSORTx⁵⁵ and DWLS⁴³ to breast cancer samples identified nine ecotypes, each defined by distinct combinations of tumor, immune, and stromal components⁹². Ecotype 3 was characterized by a predominance of basal-like and highly proliferative tumor cells, suggesting a more aggressive tumor phenotype, while Ecotype 4 featured substantial infiltration of anti-tumor immune cells, indicating a more immune-active tumor microenvironment⁹². [Au: I wonder if these results can be connected to PAM50. Does it improve breast cancer subtyping? These findings underscore how deconvolution can reveal tumor subtypes based on their cellular ecosystem.

While deconvolution can reveal novel tumor ecosystem subtypes, their identification typically serves as a steppingstone toward clinical applications. Most studies correlate these cellular composition-based subtypes with patient prognosis and treatment response^87,93, as tumor architecture directly influences biological behavior and therapeutic vulnerability.

Extraction of tumor cell RNA expression levels

Identifying cancer-intrinsic features requires tools that specifically output tumor-specific expression profiles, separating tumor signals from the complex mixture of stromal and immune components. Methods such as DeClust⁷¹, DeMixT⁶¹, ISOpureR⁶², CIBERSORTx⁵⁵, and BayesPrism⁴⁶ are particularly useful in this regard, as they separate cancer cell RNA expression levels from confounding non-cancerous cell expression levels (Table 1), thereby reducing inter-sample heterogeneity and increasing statistical power.

Deconvolution methods that output tumor-specific RNA expression typically produce three types of data: total tumor mRNA abundance, representing the overall transcriptional activity of malignant cells; patient-specific gene-level expression profiles, showing individual tumor expression patterns; and population-average gene-level expression profiles, representing the mean tumor expression patterns across samples. These different outputs serve complementary purposes in cancer research. While methods like DeMixT⁶¹ focus primarily on estimating tumor mRNA proportions (the fraction of total mRNA derived from tumor cells, rather than cellular proportions), with tumor-specific expression for each patient as a secondary output. ISOpureR⁶² is optimized for population-average tumor expression profiles only. CIBERSORTx⁵⁵ and BayesPrism⁴⁶ provide both patient-specific and population-average expression estimates across various cell types. Table 1 shows additional methods with diverse statistical approaches, each offering different advantages depending on the research question.

This capability enables identification of distinct molecular signatures corresponding to established tumor subtypes, providing a more accurate representation of tumor-intrinsic characteristics. For example, applying ISOpure⁶² to bulk gene expression data from breast cancer samples separated the gene expression profiles of tumor cells from those of non-tumor cells⁹⁴. The study discovered that the PAM50 genes⁹⁰ exhibited different expression patterns between tumor-specific and non-tumor mRNA profiles, with tumor-specific mRNA profiles providing clearer subtype separation.

A method to determine tumor-specific mRNA expression levels (TmS), an application extended from DeMixT⁶¹, revealed that higher tumor-specific transcript expression are associated with epithelial-to-mesenchymal transition, stemness signatures, hypoxia, active metabolic pathways, increased abundance of open-chromatin, and more frequent TP53 driver mutations, whereas tumor subtypes with low TmS scores display less aggressive features across cancers⁷⁵. By isolating tumor-specific expression profiles, the confounding effects of non-tumor components are minimized, offering a more precise representation of tumor biology that can reveal previously masked associations with genomic alterations, biological pathways, and clinical outcomes.

Predicting patient outcomes and treatment responses

Building upon the insights generated through deconvolution, approaches for predicting patient outcomes and treatment responses can be developed, enabling the translation of these insights into clinically actionable information. This represents one of the major applications that deconvolution is uniquely suited for.

Specific cell-type proportions within tumors, as determined by deconvolution, serve as robust prognostic markers. For example, using TmS⁷⁵, with expression data further adjusted for tumor purity and ploidy, high tumor-specific total mRNA expression is shown as a key marker associated with tumor cell plasticity and disease progression. Additionally, using CIBERSORT²⁷, higher proportions of CD8⁺ T cells were linked to improved survival across multiple cancer types, and using PREDE⁶³, higher macrophage proportions were found to correlate with worse survival in breast and bladder cancers⁹⁵. Moreover, the neutrophil-to-lymphocyte ratio, as calculated by CIBERSORT²⁷, has been consistently associated with poor prognosis across various cancer types⁸⁶. Some deconvolution tools also extend these analyses beyond cell proportions to include variations in phenotypic cell states. For instance, using BayesPrism, shifts in macrophage states that correlate with clinical outcomes in cutaneous melanoma and glioblastoma were identified⁴⁶.

The integration of single-cell discovery with bulk validation through deconvolution has emerged as a powerful translational approach that leverages the high resolution of single-cell data while overcoming its limited sample size. As mentioned earlier, a novel IL-1β⁺ macrophage population was first identified through scRNA-seq analysis, and its presence and clinical relevance was then validated in larger bulk cohorts using CIBERSORTx⁸⁷. This macrophage subtype, characterized by elevated expression of inflammatory genes (IL1B, TNF, CXCL8), was shown to significantly contribute to the pathogenic inflammatory landscape of pancreatic tumors, establishing it as a potential therapeutic target for reducing tumor-promoting inflammation. Similarly, in hepatocellular carcinoma, unique ´onco-fetal´ cell populations, including EPCAM⁺ onco-fetal hepatocytes and PDGFRA⁺ onco-fetal fibroblasts were identified⁹³. Through deconvolution of bulk hepatocellular carcinoma datasets, a higher proportion of these onco-fetal cells was found to correlate with an increased risk of relapse and poorer response to PD-1 blockade immunotherapy, emphasizing the potential of scRNA-seq-based deconvolution in uncovering clinically relevant tumor subtypes.

While immunotherapy has transformed cancer treatment with remarkable responses in some patients, identifying those who will benefit remains a critical clinical challenge, as current response rates vary widely across cancer types. Deconvolution tools are particularly valuable in monitoring treatment response and predicting therapy outcomes, especially for immunotherapies where the TME composition directly impacts efficacy. CIBERSORTx⁵⁵ has been used to define cell types that predict response to immunotherapy across different cancer types^96–100, with multiple studies demonstrating that pre-treatment immune profiles can stratify responders from non-responders. Other deconvolution tools such as MIXTURE¹⁰¹, Kassandra¹⁰², and ProM¹⁰³ have shown promise in predicting immunotherapy outcomes. A recent comprehensive transcriptome deconvolution analysis¹⁰⁴ using non-negative least squares regression across 1,486 tumors from multiple cancer datasets, including TCGA cohorts and immune checkpoint blockade-treated patients, demonstrated that stromal — rather than cancer — cell expression of PDL1, CXCL9, and CXCL13 served as key determinants of immune checkpoint inhibitor response. This finding challenges the conventional focus on tumor cell biomarkers and highlights how deconvolution can reveal cell-type-specific contributions to treatment response by separating cancer and stromal cell expression signatures from bulk tumor transcriptomes.

Future directions

Transcriptomic deconvolution holds immense potential to further advance both cancer research and clinical practice. Progress in the field will depend on improving two factors: the translation of deconvolution tools into clinical applications and the continued refinement of the underlying methodologies.

Clinical application of deconvolution

Deconvolution methods are increasingly being integrated into clinical research as exploratory tools to understand tumor microenvironment dynamics and treatment responses. Currently, several clinical trials employ these approaches as secondary or correlative analyses across three main applications: (i) monitoring TME dynamics in response to treatment, such as in triple-negative breast cancer (NCT03979508¹⁰⁵) and metastatic colorectal cancer (NCT06522919¹⁰⁶); (ii) comparing genotypes at diagnosis and at relapse, as in early-stage follicular lymphoma (NCT05929222¹⁰⁷); and (iii) profiling immune cell composition to inform treatment decisions, including a trial in colorectal cancer (NCT03827967¹⁰⁸) and a prospective analysis of acute myeloid leukemia and sarcoma (NCT06764589¹⁰⁹).

Among these trials, only CIBERSORT²⁷ and CIBERSORTx⁵⁵ were explicitly specified (NCT03979508, NCT05929222, NCT03827967^105,107,108). While currently serving as secondary measures, the clinical application of deconvolution tools demonstrates growing potential for enhancing clinical decision-making, such as prognostic biomarker discovery and treatment stratification^{55,105,107,108}), which likely reflects the nascent stage of clinical application of deconvolution tools. [Au: I wonder if you can expand a bit and provide details about the samples, methods and perhaps tumor types that are analysed as part of these clinical studies. I think the reader would also value your expert opinion on the challenges these trials face and how these might be addressed in the future (for which reason I recommend moving the discussion into the next main section entitled “Future directions”. Reply: Thank you for the suggestion. We have reorganized the section accordingly and revised to provide more details about methods and tumor types. However, the information available remains limited - for example, the sample type for deconvolution is not stated in the clinical trials, and only a subset of them specified the proposed deconvolution tool. We therefore added trial details to the best of our knowledge, provided a broad overview of challenges, and discussed the major one, applicable sample type (FFPE), in the next paragraph.] While currently serving as secondary outcome measures, the incorporation of deconvolution tools in these trials highlights both the growing recognition of their potential value and the existing barriers to clinical implementation. Key challenges include the lack of standardized protocols for different sample types, uncertainty regarding optimal method selection for specific tumor types and clinical questions, and the absence of validated clinical interpretation thresholds for deconvolution-derived metrics. The successful clinical translation of deconvolution tools will require addressing these implementation gaps through multi-disciplinary collaborative efforts: establishing standardized workflows, validating method performance across diverse clinical sample types, and establishing evidence-based guidelines for interpreting deconvolution results in clinical contexts. These efforts will be essential for transitioning these analytical tools from exploratory research to clinical decision support, enabling applications in prognostic biomarker discovery and treatment stratification.

One important ongoing step in adapting deconvolution methods for clinical use is extending their application to formalin-fixed paraffin-embedded (FFPE) samples, as this is the standard for storing clinical samples. Translating deconvolution approaches to clinical practice requires addressing practical tissue preservation challenges. FFPE samples constitute the vast majority of archived clinical specimens with long-term follow-up data, making them invaluable for translational cancer research. However, the fixation process causes RNA fragmentation, chemical modifications, and cross-linking, resulting in reduced fidelity of signals, decreased mapping ability, increased duplicate rates, 3’ bias in gene coverage, and significantly different transcriptomic profiles compared to fresh-frozen tissues^110–112, complicating the deconvolution process. The developers of several deconvolution methods like Kassandra¹⁰², CIBERSORT²⁷ and CIBERSORTx⁵⁵ emphasized the potential application of these techniques to FFPE samples. The performance of CIBERSORT and CIBERSORTx across tissue preservation techniques were evaluated in their original studies^27,55. These approaches could be useful when fresh-frozen tissues are not readily available, especially for retrospective bulk RNA-sequencing. [Au: Was Kassandra also tested for its ability to be used in FFPE? Please specify. The deconvolution resolution may remain limited, since success has been primarily reported in deconvolving major cell types (such as leukocytes) in FFPE samples. While initial attempts at deconvolving FFPE-derived bulk RNA sequencing data in a cancer setting have been made¹¹³, independent benchmarking of deconvolution methods for FFPE samples remains limited to non-cancer settings¹¹⁴. As FFPE-derived bulk RNA deconvolution is notably affected by sample quality and is still at an early stage of development, optimization strategies are primarily focused on experimental design. For FFPE transcriptomic profiling, a DV200 score, which is the percentage of RNA fragments >200 nucleotides of ≥30% has been shown to be a reliable quality control metric for accessing RNA integrity¹¹⁵, and protocols with extended proteinase K digestion improve extraction yields¹¹⁶. Researchers are advised to use either direct hybridization platforms (for example, NanoString nCounter^117,118) or FFPE-optimized library preparation kits (for example, Illumina TruSeq RNA Exome, Ion AmpliSeq¹¹⁹ and the 3’ RNA-seq method Lexogen QuantSeq¹²⁰) to accommodate degraded RNA and enable reliable expression profiling.

Once FFPE specimen transcriptomes are stably and accurately profiled, deconvolution algorithms will still need to be benchmarked for FFPE and optimized to mitigate the technical differences across platforms and tissue preservation techniques^52,53,55. Studies have demonstrated that batch-correction methods like ComBat¹²¹ and limma¹²² can successfully integrate data across both sequencing technologies and sample types¹²³, enabling meta-analysis of archival and fresh-frozen tissue data, while the effectiveness of batch correction may depend on FFPE sample quality metrics like DV200. Future work may involve building error models that are trained on paired fresh-frozen and FFPE samples¹²⁴, or integrating molecular barcoding techniques that mitigate degradation effects^114,125. Successfully adapting deconvolution for FFPE data would enable retrospective studies on the vast amount of available samples associated with long-term clinical outcomes and their incorporation into clinical workflows, offering deeper insights into disease progression and treatment response.

The incorporation of machine learning into deconvolution methodologies represents another promising direction for enhancing both accuracy and interpretability, particularly for challenging sample types like FFPE. Machine learning models can identify complex, nonlinear patterns in gene expression data that traditional approaches might not capture. Integrating machine learning into deconvolution workflows could substantially improve analytical precision and uncover new biomarkers or therapeutic targets. For example, when sufficient training data is available, machine learning can directly predict cell-type-specific outputs from raw mixed data, bypassing traditional deconvolution steps^49,102. Several existing deconvolution methods leverage machine learning techniques, including the supervised learning framework in Kassandra¹⁰², ensemble deep learning in Scaden⁴⁹, and tissue-specific neural networks like GBMPurity¹²⁶ for specialized applications. Emerging approaches include graph neural networks for cross-platform compatibility¹²⁷ and uncertainty quantification methods for improved reliability¹²⁸, with substantial opportunities remaining for novel architectures and training strategies. Additionally, such models could be particularly valuable for FFPE samples, where they might learn to correct degradation-specific artifacts. However, machine learning applications present inherent risks, including overfitting, reliance on poorly annotated training data, and the ‘black box [G]’ nature of complex models¹²⁹. Careful validation and development of transparent, explainable models will be essential to fully harness the potential of machine learning in cancer biology.

Methodological improvements

Advancing the technical capabilities of deconvolution methods is crucial, particularly in the context of temporal dynamics within tumors. Tumors evolve over time in response to environmental pressures, immune surveillance, and therapeutic interventions and traditional deconvolution methods provide only a static snapshot of these processes. By integrating time-series data, temporal deconvolution models could reveal how cell populations within tumors change over time, offering insights into the development of treatment resistance. However, this approach presents substantial methodological challenges, including the need to manage nonlinear dynamics and the inherent variability of longitudinal data. Addressing these challenges will be essential for accurately capturing the temporal evolution of tumors, thereby refining the precision of deconvolution analyses.

There is a long-standing interest in advancing deconvolution techniques to resolve cell-type specific gene expression for each patient sample. Most available methods provide mean expression profiles for cell types (Table 1), which can mask critical sample variation caused by intra-cell-type heterogeneity, a well-known feature of multiple cell types in tumor tissues. This limitation has already driven the development of advanced methods capable of estimating cell-type-specific expression at the patient-level and identifying statistically significant differentially expressed genes within these subpopulations, such as BayesPrism⁴⁶, CIBERSORTx⁵⁵, DeMix⁶⁰, and DeMixT⁶¹. Future efforts are needed to refine these methods, in order to uncover cell populations with variably expressed genes that play crucial roles in treatment resistance or metastasis. Achieving this level of precision in deconvolution analysis will require novel computational approaches that balance the need for detailed resolution with the necessity of producing reliable, biologically meaningful insights. As these techniques evolve, ensuring that they elude common pitfalls, such as overfitting or misinterpretation, will be essential to their successful implementation in cancer research.

Further potential for advancement of deconvolution methods lies in the incorporation of biological principles that have not yet been fully explored. Specifically for deconvolution of cancer samples, somatic copy number alterations are an important driver of cancer cell gene expression¹³⁰, and this information-rich signal is still relatively unexplored in existing deconvolution methods. Additionally, the emergence of atlases incorporating methylation or assay for transposase-accessible chromatin with sequencing (ATAC-seq) data presents an exciting opportunity to integrate transcriptomic deconvolution with epigenetic data¹³¹. Similar to tools like epiSCORE¹³², able to deconvolve bulk methylation from scRNA-seq, there is growing potential to leverage epigenetic data for the deconvolution of transcriptional profiles. Overall, approaches that integrate additional data types (such as DNA-seq or epigenetic data) with RNA-seq data to identify synergistic signals are likely an important avenue for further deconvolution method development^75,133.

Expanding the scope of deconvolution methods to include the full diversity of RNA species offers both technical challenges and clinical opportunities. Human cells contain tens of thousands of distinct RNA molecules, and current single-cell sequencing methods do not comprehensively capture the diversity of non-coding RNAs, such as long non-coding RNAs (lncRNAs), miRNAs, and circular RNAs (circRNAs)¹³¹. Hence, extending deconvolution techniques to these RNA species could provide a more comprehensive understanding of the regulatory networks driving tumor development. However, these RNA species present unique challenges, such as diverse expression patterns and stability, complicating their analysis. Recent technological advances such as Smart-seq-total¹³⁴ and parallel single-cell small RNA sequencing (PSCSR-seq)¹³⁵ can overcome some of these limitations and may be used to create robust reference sets for non-coding and small RNA species. Developing specialized reference datasets and analytical frameworks will be critical for accurately deconvolving these RNA types, thereby enhancing our understanding of their roles in tumor biology and translating these insights into clinical biomarkers and therapeutic targets.

Conclusions

With more than 40 methods published over the past 15 years, the progress in the field of bulk transcriptomic deconvolution shows no sign of slowing down. Building upon the notable amount of work in the field, we have developed a guide for cancer researchers to navigate through many possible routes of deconvolution and maximally utilize bulk RNA-seq data. In the context of the TME, researchers primarily interested in understanding the immune environment in a broad sense can readily enjoy advances made by many reference-based deconvolution methods. Those who are interested in the dynamic ecosystem of tumor-stromal-immune cells may utilize either single-cell reference-based or semi-reference-based methods, depending on the specific question, available prior knowledge, and experimental means for validation for the cancer type of interest. In the context of patient subtyping or prediction of prognosis or response to treatment, although a birds-eye-view ecosystem characterization can be informative, there remain significant gaps in knowledge to be filled to accurately characterize tumor and stromal (that is, non-immune) compartments through deconvolution. These questions are currently best addressed using semi-reference-based or specially designed single-cell reference-based methods. Reference-free methods are gaining popularity again, though their application to cancer research has yet to gain momentum.

The future of transcriptomic deconvolution in cancer research is full of potential and realizing this potential will require addressing substantial remaining technical and conceptual challenges, particularly in the integration of other data modalities in a question-specific fashion. Major efforts are needed to generate biologically meaningful reference and benchmark data. By refining existing methods, developing new approaches, and thoughtfully considering the balance between innovation and practicality, the field will continue to advance our understanding of cancer biology, ultimately leading to more effective and personalized cancer therapies.

Glossary

Deconvolution

A computational method that separates bulk tissue RNA sequencing data into individual cell type contributions by estimating the proportion and gene expression of each cell type present in the sample.

Expression matrix

A data structure where rows represent genes and columns represent samples, with each cell containing the measured expression level of a specific gene in a specific sample.

Non-negative least squares

A mathematical optimization technique that finds the best-fitting solution to a linear equation system while constraining all values to be zero or positive, commonly used in deconvolution to ensure biologically meaningful cell proportions.

Bayesian models

Statistical frameworks that incorporate prior knowledge or assumptions about parameters and update these beliefs based on observed data to make probabilistic inferences.

Pseudo-bulk

measurement data from summing over expression counts per cell across all cells for each gene and each sample.

Guided topic modeling approach:

A machine learning method adapted from natural language processing that identifies topics (cell types) in documents (samples) by discovering patterns of co-occurring words (genes).

Iterated Conditional Modes (ICM)

An optimization algorithm that iteratively updates each parameter set while keeping others fixed, commonly used to find maximum likelihood estimates in complex statistical models.

Dirichlet-multinomial mixture

A probabilistic model that combines Dirichlet distributions (which model proportions) with multinomial distributions (which model counts) to handle over-dispersed count data.

Non-negative Matrix Factorization

A mathematical technique that decomposes a matrix into two or more matrices with non-negative elements, often used to identify hidden patterns or reduce dimensionality in data.

Linear subspace identification

A mathematical approach that finds lower-dimensional linear spaces within high-dimensional data that capture the essential variation or structure.

Compressed sensing techniques

Mathematical methods that reconstruct signals or data from fewer measurements than traditionally required by exploiting sparsity or structure in the data.

Shift-invariance

A mathematical property where the output of a system remains unchanged when the input is shifted in time, space, or another dimension.

Convex analysis

A branch of mathematics dealing with convex sets and functions, providing theoretical foundations for optimization problems with guaranteed global solutions.

Black box

A computational method or algorithm where the internal workings are not transparent or easily interpretable, only the inputs and outputs are observable.

Phenotypic plasticity

The ability of cells to change their observable characteristics, gene expression patterns, or functional states in response to environmental conditions without altering their genetic code.

Capture efficiency

The proportion of RNA molecules from a cell that are successfully captured, reverse-transcribed, and sequenced during single-cell RNA sequencing experiments.

Dropout effects

The phenomenon in single-cell RNA sequencing where genes that are expressed in a cell fail to be detected, resulting in false zero counts in the expression data.

Multicollinearity

A statistical condition where two or more predictor variables in a model are highly correlated, making it difficult to determine the individual contribution of each variable.

LASSO regularization

A statistical technique that adds a penalty term to regression models to prevent overfitting by forcing some coefficients to become exactly zero, effectively performing feature selection.

Background noise

Random variation or systematic artifacts in measurement data that are not related to the biological signal of interest, providing positive baseline measurements when no biological signals are present.

Spatial transcriptomics

A technology that measures gene expression while preserving information about the physical location of cells or tissue regions, enabling analysis of spatial organization and cell-cell interactions.