Denoising single-cell RNA-seq data with a deep learning-embedded statistical framework

Abstract

Background

Single-cell RNA sequencing (scRNA-seq) provides extensive opportunities to explore cellular heterogeneity but is often limited by substantial technical noise and variability. The prevalence of zero counts, arising from both biological variation and technical dropout events, poses significant challenges for downstream analyses. Existing imputation methods face inherent trade-offs: statistical approaches maintain interpretability but exhibit limited capacity for capturing complex, non-linear gene expression relationships, whereas deep learning methods demonstrate superior flexibility but are prone to overfitting and lack mechanistic interpretability, particularly in settings with limited sample sizes.

Methods

We present ZILLNB (Zero-Inflated Latent factors Learning-based Negative Binomial), a novel computational framework that integrates zero-inflated negative binomial (ZINB) regression with deep generative modeling. ZILLNB employs an ensemble architecture combining Information Variational Autoencoder (InfoVAE) and Generative Adversarial Network (GAN) to learn latent representations at cellular and gene levels. These latent factors serve as dynamic covariates within a ZINB regression framework, with parameters iteratively optimized through an Expectation-Maximization algorithm. This approach enables systematic decomposition of technical variability from intrinsic biological heterogeneity.

Results

Comparative evaluations across multiple scRNA-seq datasets demonstrate ZILLNB’s superior performance. In cell type classification tasks using mouse cortex and human PBMC datasets, ZILLNB achieved the highest Adjusted Rand index (ARI) and Adjusted Mutual Information (AMI) among tested methods, with improvements ranging from 0.05 to 0.2 over VIPER, scImpute, DCA, DeepImpute, SAVER, scMultiGAN and ALRA. For differential expression analysis validated against matched bulk RNA-seq data, ZILLNB demonstrated improvements ranging from 0.05 to 0.3 for area under the Receiver Operating Characteristic curve (AUC-ROC) and the Precision-Recall curve (AUC-PR) compared to standard and other imputation methods, with consistently lower false discovery rates. Application to idiopathic pulmonary fibrosis (IPF) datasets revealed distinct fibroblast subpopulations undergoing fibroblast-to-myofibroblast transition, validated through marker gene expression and pathway enrichment analyses.

Conclusion

ZILLNB provides a principled framework for addressing technical artifacts in scRNA-seq data while preserving biological variation. The integration of statistical modeling with deep learning enables robust performance across diverse analytical tasks, including cell type identification, differential expression analysis, and rare cell population discovery, demonstrating utility across common single-cell analysis tasks.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12859-025-06296-w.

Introduction

Single-cell RNA sequencing (scRNA-seq) technology has significantly advanced our ability to explore diverse cell populations, facilitating the discovery of novel cell types and states [1], elucidating relationships between previously characterized and newly identified cellular phenotypes [2, 3], and improving our understanding of transcriptomic variation in health and disease contexts [4, 5]. However, despite these advantages, scRNA-seq data frequently contain substantial technical noise and variability arising from multiple sources [6–8]. Specifically, heterogeneity can be categorized into cell-specific, gene-specific, and experiment-specific variations. Cell-specific measurement errors, notably those related to variability in library sizes, pose a significant challenge. These variations primarily stem from differences in sequencing depth and can occur even among cells considered biologically similar [9]. Another prevalent issue is the high frequency of zero counts in scRNA-seq data, often resulting from technical dropout events with distinct patterns across different cell types [10]. Gene-specific errors add further complexity, resulting from interactions between genes or between genes and environmental factors that remain inadequately captured or modeled [11, 12]. Experiment-specific variability, on the other hand, often originates from biases inherent in experimental procedures, such as PCR amplification bias.

Addressing these diverse sources of variability is essential to ensure reliable downstream analyses. Unlike bulk RNA sequencing data, scRNA-seq datasets require specialized computational methods to manage their unique characteristics, particularly the abundance of zeros and associated artifacts. Existing methods generally fall into two main categories: statistical modeling approaches and deep learning-based methods. Statistical approaches, including scImpute [13], VIPER [14], SAVER [15], and ALRA [16], utilize probabilistic frameworks explicitly designed to accommodate the zero-inflation and count distributions inherent to scRNA-seq data. Deep learning-based techniques, such as DCA [17], DeepImpute [18] and, more recently, scMultiGAN [19], leverage neural network architectures, particularly autoencoders and generative adversarial networks (GANs), to capture complex nonlinear relationships among genes. Although these deep learning models provide significant flexibility and scalability, they can suffer from interpretability issues and susceptibility to overfitting, especially when sample sizes are limited.

To leverage the advantages of both statistical modeling and deep learning approaches, we introduce the zero-inflated latent factors learning-based negative binomial (ZILLNB) model. ZILLNB integrates zero-inflated negative binomial (ZINB) regression with deep latent factor models, providing a unified approach for simultaneously addressing various sources of technical variability (Fig. 1). By explicitly modeling latent structures at both cell and gene levels, ZILLNB accurately recovers gene expression signals while preserving biologically meaningful variation. Additionally, ZILLNB is designed for broad applicability, effectively handling datasets with or without explicit covariates. We validate ZILLNB’s effectiveness across multiple analytical tasks, including cell type identification, classification of cell subsets, and differential expression analysis. The combined statistical and deep learning approach of ZILLNB thus provides a robust, interpretable, and adaptable solution for analyzing scRNA-seq data.

Fig. 1.

Workflow of the ZILLNB Denoising Method. ZILLNB processes a count matrix using an InfoVAE-GAN model to generate latent factors for cells and genes, which are used to fit a ZINB model. The output is a dense, denoised matrix. The schematic diagram at the bottom shows the decomposition of the link function of the mean value in matrix form. ${\alpha }_{L\times M},{\beta }_{K\times N},{\gamma }_{S\times M}$ are regression parameters, ${\xi }_N,{\zeta }_M$ are intercept terms, $1_M,1_N$ are vectors with ones, $U_{K\times M},V_{L\times N}$ are latent factors and $W_{S\times N}$ is covariates matrix which is optional. Dimensions $L,K,S$ represent the respective sizes of matrices $U,V,W$

Methods

The ZILLNB model comprises three main steps (Fig. 1). First, an ensemble-based deep generative framework combining InfoVAE and GAN extracts latent features from both cellular and gene-level perspectives. Second, the derived latent factors are utilized to fit a ZINB model, refining both latent representations and regression coefficients through iterative optimization via the Expectation-Maximization (EM) algorithm. This iterative refinement distinctly decomposes variability in the raw data into cell-specific sampling noise and intrinsic gene-specific heterogeneity. Finally, adjusted mean parameters generate a denoised and complete expression matrix.

Latent factors learning

We use the ensemble InfoVAE-GAN model for manifold learning and identifying potentially missing cell- and gene-grouping in scRNA-seq datasets [20, 21]. To mitigate overfitting issues common in traditional VAEs, we integrate InfoVAE [22, 23] with GAN [24] and replace Kullback–Leibler (KL) divergence with Maximum Mean Discrepancy (MMD) as the regularizer for both cell-wise structure V and gene-wise structure U, while constraining the latent space with normal priors:

$$\begin{array}{c}\hfill MMD(p(x)\mid \mid q(x))={\mathbb{E}}_{p(x),p\left(x^{\prime }\right)}\left[k,\left(x,,,x^{\prime }\right)\right]+{\mathbb{E}}_{q(x),q\left(x^{\prime }\right)}\left[k,\left(x,,,x^{\prime }\right)\right]-2{\mathbb{E}}_{p(x),q\left(x^{\prime }\right)}\left[k,\left(x,,,x^{\prime }\right)\right],\end{array}$$ 1

where $p(x),q(x)$ denote probability densities, $k(x,x^{\prime })=exp\left(-|x-,x^{\prime },{|}^2,/,2\right)$ is the Gaussian kernel, $x^{\prime }$ follows a prior distribution in the latent space, and x are latent samples derived from observed data y. The InfoVAE-GAN architecture includes three interconnected neural networks: (1) an encoder $Enc(y)$ that maps a sample y (either a cell or gene vector from the expression matrix Y) to latent space as $x\sim Enc(y)=q(x|y)$; (2) a decoder $Dec(x)$ reconstructing input samples as $y^{\prime }\sim Dec(x)=p(y^{\prime }|x)$; and (3) a discriminator $Dis(y)$, distinguishing real data from generated samples with probabilities defined as $p=Dis(y)\in [0,1]$. Adaptive weighting parameters ${\gamma }_1$ and ${\gamma }_2$ balance the reconstruction loss (${\mathcal{L}}_{\mathit{like}}$), prior alignment (${\mathcal{L}}_{\mathit{prior}}$), and generative accuracy (${\mathcal{L}}_{\mathit{GAN}}$) by ${\mathcal{L}}_{InfoVAE-GAN}={\mathcal{L}}_{\mathit{like}}+{\gamma }_1{\mathcal{L}}_{\mathit{prior}}+{\gamma }_2{\mathcal{L}}_{\mathit{GAN}},$ with ${\mathcal{L}}_{\mathit{like}}=-{\mathbb{E}}_{q(x|y)}[logp(y|x)]$, ${\mathcal{L}}_{\mathit{prior}}$ representing the MMD loss, and ${\mathcal{L}}_{\mathit{GAN}}=log(Dis(y))+log(1-Dis(Dec(x)))+log(1-Dis(Dec(Enc(y))))$. Additional network details and hyperparameter selection are provided in Supplementary Sect. 1.1.

ZINB fitting

Consider a single-cell RNA sequencing dataset represented by an expression matrix Y with dimensions M (genes) by N (cells). Each element $Y_{\mathit{ij}}$ represents the observed expression count for gene i in cell j, modeled using a ZINB distribution. We introduce latent binary variables $Z_{\mathit{ij}}\sim \text{Bernoulli}({\varphi }_i)$ indicating whether a zero is generated by a dropout event:

$$\begin{array}{c}\hfill Y_{\mathit{ij}}\mid Z_{\mathit{ij}},{\mu }_{\mathit{ij}},{\theta }_i\sim \left\{\begin{array}{cc}\text{I}\{Y_{\mathit{ij}}=0\},\hfill & Z_{\mathit{ij}}=1;\hfill \\ NB(Y_{\mathit{ij}}\mid {\mu }_{\mathit{ij}},{\theta }_i),\hfill & Z_{\mathit{ij}}=0.\hfill \end{array}\right)\end{array}$$ 2

Here, ${\varphi }_i\in [0,1]$ denotes the gene-specific dropout probability, while ${\mu }_{\mathit{ij}}\in {\mathbb{R}}_{+}$ and ${\theta }_i\in {\mathbb{R}}_{+}$ are parameters representing the mean and dispersion of the negative binomial distribution, respectively. The indicator function $\text{I}(·)$ equals 1 if its argument is true and 0 otherwise. By expressing the mean parameter ${\mu }_{\mathit{ij}}$ through a log-link function, we incorporate latent cell- and gene-specific structures

$$\begin{array}{c}\hfill log{\mu }_{M\times N}={\mathbb{1}}_M{\xi }_N^{\top }+{\zeta }_M{\mathbb{1}}_N^{\top }+{\alpha }_{L\times M}^{\top }{V}_{L\times N}+U_{K\times M}^{\top }{\beta }_{K\times N},\end{array}$$ 3

where $U\in {\mathbb{R}}^{K\times M}$ and $V\in {\mathbb{R}}^{L\times N}$ represent latent factor matrices associated with genes and cells, respectively. Parameters $\xi \in {\mathbb{R}}^N$ and $\zeta \in {\mathbb{R}}^M$ account for cell- and gene-specific intercepts, while regression parameters are encapsulated in matrices $\alpha \in {\mathbb{R}}^{L\times M}$ and $\beta \in {\mathbb{R}}^{K\times N}$. To improve model fit, we iteratively update the latent factor matrix U (obtained from the deep generative model) together with the regression parameters via the EM algorithm, while holding the cell-wise factor V fixed because it already captures sufficient biological structure (Figs. 4 and S2). To prevent overfitting, we include regularization terms on U and on the intercepts $\xi$ and $\zeta$ in the objective function. Detailed derivations and computations for the latent-variable updates and the auxiliary function are provided in Supplementary Sect. 1.2. In practice, the model converges after a few iterations (see Fig. S1).

Fig. 4.

Latent factor learning preserves cell-wise and gene-wise information in the mouse brain dataset [40]. A, B UMAP projections of cell latent factors V and gene latent factors $\widehat{U}$, annotated by cell types and Seurat-predicted clusters. C Comparison of learned gene latent factor matrix $\widehat{U}$ with a standard Gaussian matrix using Wishart ensemble. The upper panel shows empirical spectral distributions versus the theoretical Marchenko-Pastur law (red curve); the lower panel illustrates the level-spacing distributions alongside the Wigner Surmise law (blue curve). D Heatmap of the top 20 marker genes per cell type based on $\widehat{U}$, with genes as columns and latent variables ($K=20$) as rows. E Heatmap illustrating the overlap between the top 20 marker genes and Seurat-predicted gene clusters. F AUCell scores for the predicted gene clusters across various cell types, reflecting activity levels within the original dataset. G UMAP projection of the mESC cell cycle dataset [43] based on $exp({\widehat{U}}^{\top }\widehat{\beta })$ matrix, annotated by predicted and true cell types. H Clustering performance assessment using Purity, F score, ARI, and AMI metrics of imputed $exp({\widehat{U}}^{\top }\widehat{\beta })$ matrix (“ZILLNB$\_$Ubeta”), compared with COMSE [44], M3Drop [45], Scran [46], Scran cyclone [47], default Seurat method and Seurat cell-cycle scoring [48]

Furthermore, external covariates can be included by extending equation (3) with an additional term, ${\gamma }_{S\times M}^{\top }{W}_{S\times N}$, where $W\in {\mathbb{R}}^{S\times N}$ represents external covariate data, $\gamma \in {\mathbb{R}}^{S\times M}$ are the corresponding regression coefficients, and S represents the number of covariates. During optimization, since V remains fixed, we can concatenate V and W into a combined matrix $\tilde{V}\in {\mathbb{R}}^{(L+S)\times N}$, and similarly merge $\alpha$ and $\gamma$ into $\tilde{\alpha }\in {\mathbb{R}}^{(L+S)\times M}$, without altering the algorithm.

Data imputation

Building on latent factors learning and ZINB fitting steps described above, we introduce the concept of adjusted mean as follows:

$$\begin{array}{c}\hfill {\widehat{\mu }}_{\mathit{ij}}^{\ast }=\left\{\begin{array}{cc}exp({\widehat{\alpha }}_i^{\top }{v}_j+{\widehat{u}}_i^{\top }{\widehat{\beta }}_j),\hfill & (\text{Output 1, without covariates});\hfill \\ exp({\widehat{\alpha }}_i^{\top }{v}_j+{\widehat{u}}_i^{\top }{\widehat{\beta }}_j+{\widehat{\gamma }}_i^{\top }{w}_j),\hfill & (\text{Output 2, with covariates}).\hfill \end{array}\right)\end{array}$$ 4

Here, $v_j$ and $w_j$ denote the jth columns of the fixed cell latent factor matrix $V_{L\times N}$ and the covariate matrix $W_{S\times N}$, respectively. The vectors ${\widehat{\alpha }}_i$ and ${\widehat{\gamma }}_i$ are the ith columns of ${\widehat{\alpha }}_{L\times M}$ and ${\widehat{\gamma }}_{S\times M}$; ${\widehat{\beta }}_j$ is the jth column of ${\widehat{\beta }}_{K\times N}$; and ${\widehat{u}}_i$ is the ith column of ${\widehat{U}}_{K\times M}$; all estimated by the EM algorithm. By intentionally omitting the intercept terms ${\zeta }_i$ and ${\xi }_j$, we normalize the original mean parameter ${\mu }_{\mathit{ij}}$. This normalization allows direct comparability of expression levels across samples while preserving intrinsic biological variability, and is consistent with established denoising practices used in bulk RNA-seq analyses [9]. Moreover, Output 1 is preferred when covariates primarily encode nuisance variation (e.g., batch effects). Omitting the covariate term ${\widehat{\gamma }}_i^{\top }{w}_j$ facilitates integration across datasets while preserving the biological signal of interest within a unified framework. Unless otherwise stated, we use Output 1 by default in all experiments.

Dimension reduction and clustering

To effectively visualize high-dimensional raw and imputed data (as shown in Figs. 2, 3, 4, 5 and 6), we adopted the standard workflow from Seurat [25] (https://satijalab.org/seurat/articles/pbmc3k_tutorial). Raw data were processed following the default procedures recommended by Seurat. For the imputed data, we initially applied column-wise log-normalization, followed by principal component analysis to identify principal components used for dimensionality reduction via Uniform Manifold Approximation and Projection (UMAP). In cases involving large-scale datasets, we additionally employed t-Distributed Stochastic Neighbor Embedding (t-SNE) to enhance visualization quality. Cluster assignments and validations were conducted using hierarchical clustering with average linkage. Notably, clustering analyses performed on the estimated latent matrix $\widehat{U}$ followed the identical approach used for the raw data.

Fig. 2.

ZILLNB improves cell classification accuracy and compactness in mouse brain cortex and subsampled human PBMC scRNA-seq datasets. A, B, D, E UMAP plots of the mouse brain and subsampled human PBMC scRNA-seq data using raw and default ZILLNB-imputed data, with points labeled by true cell types. C, F Four external clustering validations (Purity, F score, ARI, and AMI) demonstrate the performance and robustness of ZILLNB compared to other widely used imputation methods, evaluated on the mouse brain dataset and human PBMC dataset

Fig. 3.

ZILLNB improves resolution and clarity in cell classification within the MCA scRNA-seq dataset. A–E tSNE plots displaying cell-type annotations of the mouse cell atlas dataset, before and after denoising using various imputation methods: A raw data, B ZILLNB, C DeepImpute, D DCA, and E ALRA. F Four external clustering validations (Purity, F score, ARI, and AMI) confirm ZILLNB’s superior performance compared to other imputation methods

Fig. 5.

ZILLNB robustly identifies DEGs in bootstrap experiments using breast cancer cell line dataset [33] with tumor cell line BT474 versus non-tumor cell lines Jurkat (A–E) and Thp1 (F–J). A, F FDP, TPR, ACC, and MCC. B, G ROC curves; D, I PR curves with a fixed tDEG proportion of 20%. C, H AUC-ROC and E, J AUC-PR across varying tDEG proportions (0–50%). Dashed lines indicate the case with $\sim$3000 tDEGs (top 20%). Methods compared include Seurat log-normalized counts (“logNorm”), default ZILLNB denoised matrix (“ZILLNB”), imputed matrices of DCA and ALRA (“DCA”,“ALRA”). “t/Wilcox” denotes a two-sample t-test or Wilcoxon rank-sum test for DEGs selection

Fig. 6.

Identification of fibroblast subsets undergoing FMT. A UMAP plots of re-clustered fibroblasts from 3 patients with IPF, 3 patients with chronic obstructive pulmonary disease (COPD), and 3 healthy controls, displaying expression levels of myofibroblast markers, AUCell scores for myofibroblast-associated gene sets, and Seurat-predicted labels derived from ZILLNB-denoised data. B GO analysis of the top 200 DEGs in cluster 3 identified by ZILLNB compared to the Seurat pipeline. The bar plot illustrates the number of DEGs associated with each GO term across Biological Process (BP), Cellular Component (CC), and Molecular Function (MF). C Cumulative curve showing the top 200 DEGs in cluster 3 (ZILLNB) and cluster 4 (Seurat) that overlap with myofibroblast-related gene sets

To assess clustering accuracy, we calculated four metrics against true cell labels provided in each original dataset: Purity, F score, adjusted Rand index (ARI) [26], and adjusted mutual information (AMI) [27]. We selected ARI and AMI as our evaluation metrics because they offer complementary perspectives [28] on clustering validation: ARI favors equal-sized clusters while AMI is better suited for evaluating imbalanced clusters. The combination of these two metrics enables a more thorough and balanced evaluation of clustering results [27]. Each metric ranges from 0 to 1, with values closer to 1 indicating superior clustering performance. Detailed definitions and formulas for these metrics are available in Supplementary Section 1.4.

Random matrix comparison

To quantitatively evaluate the learned latent factor matrix ${\widehat{U}}_{K\times M}$ against random Gaussian noise, we employ spectral methods derived from random matrix theory [29]. Consider a matrix $X\in {\mathbb{R}}^{D\times N}$ whose entries are independently drawn from a Gaussian distribution $N(0,{\sigma }^2)$. We examine its sample covariance matrix $W_X=X{X}^{\top }/N$ through three eigenvalue-based metrics: (1) empirical spectral density ${\mu }_W$, which asymptotically follows the Marchenko-Pastur distribution; (2) empirical normalized level-spacing density ${\mu }_G$, approaching the Wigner Surmise distribution; and (3) the largest eigenvalue (spectral radius) $\lambda (D)$, which converges to the Tracy-Widom distribution. Detailed derivations and asymptotic distributions for these statistics are provided in Supplementary Section 1.5. For an accurate comparison, we standardized ${\widehat{U}}_{K\times M}$ by centering and scaling each column to zero mean and unit variance. Subsequently, we derived its corresponding covariance matrix $W_U$ and calculated the aforementioned spectral statistics. The goodness-of-fit between empirical and theoretical distributions was assessed using Kolmogorov-Smirnov and Anderson-Darling tests. The largest eigenvalue comparison utilized the RMTstat package in R [30], which directly computes p-values based on the Tracy-Widom distribution.

Gene sets evaluation via enrichment analysis and AUCell scores

We assessed gene set relevance through enrichment analysis and gene set activity measurements using the AUCell package [31] in R. Gene Ontology (GO) and Kyoto Encyclopedia of Genes and Genomes (KEGG) pathway analyses were conducted using the clusterProfiler package [32], with the top 10 pathways visualized (Fig. S4). To further evaluate biological significance, we calculated AUCell scores to quantify gene cluster activities, subsequently averaging these scores by cell type within each cluster. Results were presented using heatmaps (Fig. 4F).

Differentially expressed genes (DEGs) selection and simulation design

To evaluate ZILLNB in a downstream analysis task (DEGs selection), we constructed simulated experiments using the mixed scRNA-seq (Mixture 1 breast cell line dataset [33]) with bulk RNA-seq on pure cell samples used as a reference. From the original scRNA-seq data, we formed disease–control subsets by pairing one tumor line (BT474 or T47D) with one non-tumor line (Jurkat or Thp1) as the control, and generated 100 bootstrap replicates per pairing.

For DEGs calling, we performed two-sample tests (t-test or Wilcoxon rank-sum) on either the imputed expression matrices (ZILLNB, DCA, ALRA) or on log-normalized counts from Seurat. p-values were adjusted by the Benjamini-Hochberg procedure [34], and DEGs were declared by simultaneous thresholds on the adjusted p-value and the log2 fold change (logFC). For convenience, we summarize these criteria into a per-gene ranking score f (see Supplementary Section 1.3) and select predicted DEGs (pDEGs) by thresholding f on scRNA-seq data. For bulk RNA-seq, we define true DEGs (tDEGs) as the intersection of calls made independently by DESeq2, edgeR, and limma [35], yielding a stable reference set.

After generating the tDEG and pDEG sets, we evaluated performance using standard metrics–false discovery proportion (FDP), true positive rate (TPR), accuracy (ACC), and the Matthews correlation coefficient (MCC)–as detailed in Supplementary Section 1.4. We also constructed ROC and precision–recall (PR) curves by varying the number of genes selected as pDEGs and computed their areas under the curve (AUC-ROC and AUC-PR), using tDEGs as the reference set. In addition to the classical metrics, we highlight MCC [36, 37] and AUC-PR [38, 39] because they are less sensitive to class imbalance and thus provide a more informative assessment in our simulations. In practical terms, we considered the top 20% of bulk RNA-seq genes ($f=0.2$, approximately 3,000 genes) as the tDEGs for sub-figures A, B, D, F, G, and I in Figs. 5 and S7, replicating realistic scenarios. Robustness was further evaluated by systematically adjusting f values from 0 to 0.5, measuring corresponding AUC values, Pearson correlation, Wasserstein-2 distance, and cross-entropy, as described comprehensively in the Supplementary, Section 1.4.

Results

Enhanced cell classification after denoising

To evaluate the effectiveness of the ZILLNB framework, we compared its performance against several widely utilized imputation methods, including VIPER [14], scImpute [13], SAVER [15], ALRA [16], DCA [17], DeepImpute [18] and scMultiGAN [19]. Specifically, we assessed the accuracy of each method in recovering missing transcript counts and preserving biologically relevant signals critical for accurate cell-type classification.

Our evaluations involved two scRNA-seq datasets with verified cell-type annotations: (1) Mouse Brain Dataset: consisting of 3005 single cells obtained from the cerebral cortex of 33 male and 34 female mice (GSE60361) [40]; and (2) Human PBMC Dataset: comprising eight distinct immune cell types from purified human peripheral blood mononuclear cells (PBMCs), subsampled to 500 cells per cell type [41]. We compared predicted labels derived from each imputation method against true cell-type labels, employing four clustering performance metrics: Purity, F score, ARI, and AMI (see the Methods and the Supplementary Section 1.4).

The ZILLNB framework consistently outperformed standard imputation methods in clustering and cell type classification across all datasets and metrics, demonstrating improved Purity, F score, ARI, and AMI (Fig. 2C, F). Notably, ZILLNB demonstrated a significant improvement in ARI and AMI, highlighting its robustness in accurately identifying cell-type groupings. Methods such as VIPER and scImpute did not outperform when using the raw data directly, likely due to challenges in distinguishing subtle subgroup structures within highly similar cell populations, such as the T cells in the PBMC dataset (Fig. 2F).

To further illustrate the scalability of ZILLNB on large-scale scRNA-seq datasets, we employed the mouse cell atlas (MCA) dataset [42], which includes samples derived from more than 50 mouse tissues and cultures. For consistency with our main benchmark and to keep the MCA-scale evaluation focused [16], we compared against the baselines highlighted in Fig. 2C and F, namely DeepImpute, DCA, and ALRA. Compared with these three methods, ZILLNB showed a t-SNE representation with superior resolution of subcell types (Fig. 3). Specifically, gonadal-related cell subtypes (including luteal cells, cumulus cells, granulosa cells, Leydig cells, and Sertoli cells) exhibited clearer and more distinct boundaries after ZILLNB denoising, as indicated by the red dashed circles in Fig. 3B. Moreover, cluster evaluation metrics further confirmed the improved performance of ZILLNB (Fig. 3F).

By showing the accurate cell clustering results, ZILLNB effectively preserves latent biological characteristics while capturing meaningful underlying signals.

Latent factors preserve cell- and gene-wise information

To assess the biological and statistical significance of the latent factor matrices U and V produced by ZILLNB, we evaluated their ability to retain meaningful information from multiple perspectives. The matrix V effectively captured the underlying structure of cell populations, as evidenced by Fig. 4A. Additionally, we verified the capability of matrix V to accurately preserve biological signals despite significant batch effects present in the raw data. By integrating batch information encoded as a one-hot matrix into the V matrix training process using pancreatic scRNA-seq datasets generated from diverse sequencing technologies [49], V effectively mitigated sequencing biases and preserved more biological structure than Seurat-CCA [25] and Harmony [50] (Fig. S2B).

For the gene-embedding latent factor matrix $\widehat{U}$, estimated from the EM algorithm, we validated its capacity to capture biologically relevant gene structures using the mouse brain dataset [40], using both statistical and biological analyses. We compared $\widehat{U}$ against a random Gaussian matrix X of equivalent dimensions, sampled from N(0, 1), the prior distribution used during training. Spectral density analysis, normalized level-spacing distributions, and largest eigenvalue assessments revealed pronounced distinctions between $\widehat{U}$ and the random matrix X (Fig. 4C). Goodness-of-fit tests (Kolmogorov-Smirnov and Anderson-Darling tests, $p{10}^{-14}$) and the largest eigenvalue test ($p{10}^{-16}$) strongly indicated that $\widehat{U}$ encodes structured latent signals rather than random noise.

The biological relevance of $\widehat{U}$ was further supported by clustering and enrichment analyses. Using Seurat’s clustering pipeline, we identified eight gene clusters and examined their biological significance (Fig. 4D). The heatmap depicting the top 20 marker genes per cell type confirmed that $\widehat{U}$ retained critical marker gene information, with clearly identifiable clusters for microglia, oligodendrocytes, and pyramidal cells. Additional analyses indicated strong enrichment of marker genes within specific gene clusters (Figs. 4E, S3). GO and KEGG enrichment analyses validated cell-type-specific functions, highlighting immune-related processes in microglia and differentiation-related functions in oligodendrocytes (Fig. S4). AUCell analysis further confirmed high activity of gene clusters 6, 7, and 8 within microglia, oligodendrocytes, and neurons, respectively (Fig. 4F). External validation on two additional neural scRNA-seq datasets [51, 52] shows that the predicted clusters align with the annotated cell types (Fig. S5).

We further validated ZILLNB on a homogeneous mouse embryonic stem cell (mESC) dataset [43], in which subtle gene-level variation delineates the G1, S, and G2/M cell-cycle phases. The $exp({\widehat{U}}^{\top }\widehat{\beta })$ matrix of ZILLNB clearly differentiated these cell cycle phases (Figs. 4G and S6), demonstrating superior clustering performance compared to methods such as COMSE [44], Scran Cyclone [47], and Seurat cell-cycle scoring [48] (Fig. 4H) and even the imputed matrix $exp({\widehat{\alpha }}^{\top }V)$ (Fig. S6). This suggests that the latent factor $\widehat{U}$ captures high-dimensional intrinsic gene structure aligned with cell-cycle states. In addition, combining the two imputed components–the default ZILLNB output–further improves clustering accuracy (Fig. S6C).

ZILLNB enhances DEGs selection

To illustrate the utility of ZILLNB in downstream analyses, we evaluated its performance in detecting DEGs, a critical task in scRNA-seq data analysis. Since ground-truth DEG labels are typically unavailable for scRNA-seq datasets, we employed complementary scRNA-seq and bulk RNA-seq datasets. The latter provided benchmark DEG labels for robust validation. Performance stability was rigorously assessed through 100 bootstrap iterations for each experimental dataset (see Methods Section 2.7 and Supplementary Section 1.3 for details).

We analyzed a breast cancer cell line dataset containing both scRNA-seq and bulk RNA-seq data for each tumor and non-tumor cell line [33]. Semi-synthetic datasets were created by pairing one tumor (BT474 or T47D) with one non-tumor cell line (Jurkat or Thp1) to simulate disease and control groups. DEGs selection was performed using established criteria involving adjusted p-value and logFC. tDEGs were defined by intersecting results obtained independently from DESeq2, edgeR, and limma analyses on bulk RNA-seq data [9, 53, 54].

Using the top 20% ($\sim$3000 genes) of bulk RNA-seq data as benchmark tDEGs, ZILLNB consistently outperformed the log-normalization baseline and two strong imputation baselines, DCA and ALRA, selected based on their performance in Figs. 2 and 3 across multiple evaluation metrics. ZILLNB achieved significantly reduced FDP and enhanced TPR, clearly demonstrating its superior recall capabilities (Figs. 5A, F and S7). Furthermore, ZILLNB achieved stronger ROC/PR performance, with higher AUCs, than the log-normalization baseline and, in most datasets, than the two imputation baselines (DCA and ALRA) (Fig. 5B, D, G and I). When the proportion of tDEGs was systematically varied from 0% to 50% ($\sim$7,500 genes), ZILLNB maintained consistent and substantial improvements over the baseline methods across all scenarios (Figs. 5C, E, H, J, S7). Additional quantitative comparisons using Pearson correlation, Wasserstein-2 distance, and cross-entropy further supported ZILLNB’s superior performance in capturing genuine biological signals from scRNA-seq data (Fig. S8). These findings were consistently replicated across other cell-line comparisons (T47D-Jurkat and T47D-Thp1), underscoring the robustness and reliability of ZILLNB (Figs. S7 and S8).

Overall, ZILLNB demonstrated robust advantages over traditional log-normalization methods, achieving improved FDP control, higher TPR, and enhanced preservation of intrinsic data structure. Its consistently strong performance across diverse scenarios and perturbations highlights ZILLNB’s effectiveness and reliability for DEG identification in scRNA-seq analyses.

ZILLNB facilitates identification of fibroblast subsets

We next demonstrate ZILLNB’s capability in identifying specific fibroblast subsets undergoing fibroblast-to-myofibroblast transition (FMT), a crucial event in idiopathic Pulmonary Fibrosis (IPF). Utilizing default ZILLNB-denoised scRNA-seq data (from Eq. 4, Output 1), we identified a distinct cluster exhibiting robust myofibroblast-associated characteristics, validating its biological significance through marker gene expression, pathway enrichment, and gene set activity analyses.

IPF, the most common type of pulmonary fibrosis, is characterized by chronic progressive lung fibrosis [55]. The FMT drives myofibroblast accumulation, and inhibiting this transition can significantly mitigate or halt disease progression [56, 57]. Given the critical role of myofibroblasts in IPF pathogenesis, accurately identifying fibroblast subsets involved in FMT is essential for advancing therapeutic strategies.

To this end, we analyzed publicly available scRNA-seq datasets of IPF, focusing specifically on fibroblast populations [58]. Using ZILLNB, we identified a clearly defined fibroblast subset (cluster 3) characterized by elevated expression of myofibroblast markers, including MYO1E, H19, COL3A1, and COL1A1 (Fig. 6A). We further evaluated these cells by generating a curated myofibroblast-related gene list from the GeneCards database [59], applying a relevance score threshold of $⩾$ 1, resulting in a set of 517 genes. AUCell scoring analysis confirmed enriched expression of these myofibroblast-associated genes within cluster 3 (ZILLNB) (Fig. 6A).

In contrast, a parallel analysis using the raw data and the standard Seurat pipeline identified another cluster (cluster 4) with similar marker gene expression. However, this cluster exhibited less coherent signals, and AUCell scoring failed to distinctly identify a myofibroblast-enriched subset (Fig. S9A). These observations indicate improved delineation of the myofibroblast-enriched subset relative to the raw-data Seurat pipeline (Fig. S9B).

To further establish biological validity, we conducted GO enrichment analysis of the top 200 DEGs from cluster 3 identified by ZILLNB. Results revealed significant enrichment in wound healing processes, a characteristic functional signature of myofibroblasts (Fig. 6B). Additionally, cumulative curve analyses showed that ZILLNB-derived DEGs encompassed a higher proportion of myofibroblast-related genes compared with those identified via the raw Seurat analysis, further highlighting the biological significance of the identified subset (Fig. 6C).

Thus, these results affirm that ZILLNB provides clearer and more biologically accurate identification of fibroblast subsets undergoing FMT. By effectively denoising scRNA-seq data, ZILLNB emerges as a robust analytical tool with broad applicability for investigating complex biological phenomena such as IPF and similar multifactorial diseases.

Discussion

ZILLNB combines learned latent representations with ZINB regression, casting denoising as a regression problem with learned covariates rather than purely unsupervised modeling. By dynamically capturing complex multivariate structures at both gene and cell levels, ZILLNB merges the strengths of deep learning’s powerful representational capabilities with the interpretability and robustness inherent to ZINB models.

Previous ZINB-based methods have been extensively utilized for scRNA-seq and microbiome analyses. However, these approaches typically depend heavily on user-defined covariates or limit covariate dimensionality, restricting the complexity of captured group structures [60–62]. In contrast, ZILLNB employs neural network architectures to autonomously learn intricate gene-wise and cell-wise structures without predetermined covariate constraints. We assessed the biological contribution of each model component. The cell-wise latent factor V captures major cell-type structure (Figs. 4A; S2), whereas the gene-wise factor $\widehat{U}$ preserves intrinsic functional modules associated with specific cell types (Figs. 4D–F; S3-S5), enabling accurate subpopulation classification. Combining these two components yields better performance than using either alone (Fig. S6C). Consistent improvements in clustering and DEG analyses further validate the effectiveness of ZILLNB. For most applications, the default output (Output 1 in Eq. 4) is sufficient. Despite integrating deep learning with statistical modeling, ZILLNB maintains reasonable memory consumption and computational feasibility on standard multi-core systems (Fig. S10). Practical speed–memory trade-offs include highly variable gene selection, cell downsampling, and reducing the latent dimensions K and L, all while preserving biological interpretability. In summary, compared with stand-alone statistical methods or deep learning models, ZILLNB’s integrated design enhances representational capacity and extends the applicability of classical statistical models to settings where key biological signals are only indirectly observed.

Although initially designed to address heterogeneity in scRNA-seq data, ZILLNB demonstrates considerable adaptability across a broad spectrum of analytical applications. For instance, it can readily integrate user-specified covariates, making it highly effective in tasks such as batch-effect correction and cohort bias adjustment. Additionally, integration strategies based on domain-adversarial learning and variational approximations provide promising avenues to improve cross-dataset alignment in future work [63]. Users can also define particular gene and cell groups to specifically target variance sources or amplify relevant expression signals. As demonstrated in Fig. 5, defining tumor and normal cell groups significantly enhances robust DEG identification. Similarly, ZILLNB’s flexibility enables the modeling of diverse structures, including cell types, experimental conditions, and biological perturbations, to enhance downstream analyses such as precise cell subset characterization (Fig. 6).

ZILLNB represents a powerful analytical framework for interpreting complex scRNA-seq data, offering extensive potential applications. Future extensions and adaptations could further broaden its utility. Promising directions include applying ZILLNB to pseudo-time trajectory analyses [2], enabling detailed exploration of dynamic cellular processes across developmental timelines by integrating latent-space similarity measures. Additionally, extending ZINB with variance-decomposition-based approaches for gene clustering and regulatory network inference may yield deeper insights into gene function and regulatory interactions [64, 65]. Furthermore, adapting ZILLNB to bulk RNA-seq or scATAC-seq data would facilitate critical tasks such as data denoising and differential feature identification, underscoring its versatility in analyzing diverse genomic data types.

Conclusions

ZILLNB is a powerful method that leverages deep learning methods and statistical models to retain biological heterogeneity while reducing technical heterogeneity. ZILLNB automatically learns the cell-wise and gene-wise information and uses them as covariates for ZINB model fitting. Compared to other common methods for denoising, ZILLNB shows superior performance in cell-type clustering, cell subtype identification, and differentially expressed gene identification. Additionally, it can flexibly incorporate known covariates to enhance robustness and facilitate downstream analysis. Thus, ZILLNB provides a new perspective for modeling heterogeneity in scRNA-seq data by merging deep-learning-embedded data and a statistical framework, and can identify the variation introduced by different sources. We expect ZILLNB to play a crucial role in modeling the heterogeneity of scRNA-seq data in future research.

Author contributions

T.W., Q.L., and Y.Y. conceptualized and designed the model and computational framework. Q.L. and Y.Y. developed the algorithm, implemented the model, and conducted performance evaluations on real datasets. T.W. led the manuscript preparation. All authors provided substantial, critical feedback, which shaped the research, analysis, and manuscript preparation. T.W. supervised the overall project. All authors reviewed and approved the final manuscript.

Funding

Not applicable.

Data availability

All datasets utilized in this analysis are publicly available. The mouse brain scRNA-seq dataset comprising 3005 cells from the mouse cortex and hippocampus can be accessed through the Gene Expression Omnibus (GEO) SuperSeries GSE60361 [40]. The other two mouse neural scRNA-seq datasets can be accessed through GSE90806 [51] and GSE71585 [52]. The human PBMC dataset was subsampled to 4000 cells from an original 68k PBMC dataset available in the Short Read Archive under accession SRP073767, encompassing proportionally subsampled cells from eight distinct immune cell types [41]. The expression matrix for the MCA dataset is available at https://doi.org/10.6084/m9.figshare.5435866.v8. The human pancreatic scRNA-seq datasets generated using various sequencing methods [49] can be accessed under the following accession numbers: GSE81076, GSE85241, GSE86469, GSE84133, GSE81608, and E-MTAB-5061 (ArrayExpress) or at https://doi.org/10.6084/m9.figshare.12420968. The mESC scRNA-seq datasets, generated using full-length and UMI-based techniques, are available under accession numbers E-MTAB-2805 [43] and GSE54695 [66], respectively. The bulk RNA-seq dataset for mESCs is accessible under GSE78140 [67]. The breast cancer cell line dataset, comprising both bulk RNA-seq and scRNA-seq data, is publicly accessible under accession GSE220608 [33]. The IPF scRNA-seq dataset is accessible under GSE136831 [58]. The source code for ZILLNB, implemented in R and Python, is available on GitHub (https://github.com/tianyingw/ZILLNB).

Declarations

Conflict of interest

The authors declare no Conflict of interest.