iVAE: an interpretable representation learning framework enhances clustering performance for single-cell data

Abstract

Background

Variational autoencoders (VAEs) serve as essential components in large generative models for extracting latent representations and have gained widespread application in biological domains. Developing VAEs specifically tailored to the unique characteristics of biological data is crucial for advancing future large-scale biological models.

Results

Through systematic monitoring of VAE training processes across 31 public single-cell datasets spanning oncological and normal conditions, we discovered that reducing the $\beta$ value which corresponds to lower disentanglement of VAE significantly improves unsupervised clustering metrics in single-cell data analysis. Based on this finding, we innovatively developed iVAE with an irecon module that, when benchmarked against 8 established dimensionality reduction methods across 5 clustering performance metrics, exhibited superior capabilities in representing single-cell transcriptomic data.

Conclusions

The proposed iVAE architecture enhances the interpretability of single-cell data compared to conventional VAE architectures as measured by clustering metrics. Our work establishes a potential foundational VAE architecture for developing specialized large-scale generative models for biological applications.

Graphical abstract

Supplementary information

The online version contains supplementary material available at 10.1186/s12915-025-02315-7.

Background

Single-cell sequencing technology has transformed biological research by providing high-resolution molecular data at individual cell resolution, revealing population heterogeneity and rare cell types obscured in bulk analyses. However, these datasets present unique computational challenges: inherent sparsity, technical noise, and batch effects that introduce non-biological variations [1–3], necessitating specialized computational methods for reliable analysis [4, 5].

Deep learning has emerged as an effective paradigm for single-cell analysis [6], excelling in handling high-dimensional sparse data while automatically extracting relevant features and capturing complex nonlinear relationships [7, 8]. Various architectures address different analytical aspects: autoencoders for dimensionality reduction and denoising [9], VAEs (variational autoencoders) for probabilistic latent distributions that facilitate data generation and clustering [10], and generative adversarial networks for creating synthetic samples [11]. Specialized architectures—convolutional neural networks, recurrent neural networks, and graph neural networks—effectively process data with grid-topology, sequential, and non-Euclidean structures, respectively [12–15]. These neural approaches surpass traditional methods through their capacity for complex nonlinear feature extraction [16].

VAEs have garnered particular attention since their introduction by Kingma and Welling [17], establishing themselves as powerful tools for representation learning and generative modeling. Their key advantage lies in learning disentangled representations, mapping distinct features to independent latent dimensions [18]. The $\beta$-VAE framework enhanced this capability by introducing a hyperparameter $\beta$ to balance reconstruction error against KL (Kullback-Leibler) divergence, promoting more disentangled representations when $\beta >1$ [19]. Subsequent methodological advances—FactorVAE [20], $\beta$-TCVAE (Total Correlation Variational Autoencoder) [21], and DIP-VAE (Disentangled Inferred Prior Variational Autoencoder) [22]—further enhanced disentanglement through regularization strategies.

In single-cell analysis, feature disentanglement has seen widespread application. ContrastiveVI effectively decomposes variations into target cell-specific factors and control cell-shared factors, disentangling complex cellular states [23]. Similarly, Concerto employs contrastive learning to distinguish individual cells, promoting finer feature learning [24–26]. Its teacher-student asymmetric self-distillation enables complex feature representations while maintaining simplified, disentangled features—particularly suited for large-scale multi-modal data integration.

Despite these advantages, recent research has revealed limitations in highly disentangled representations. Higgins et al. identified a fundamental trade-off wherein increasing disentanglement deteriorates reconstruction quality [19], problematic for applications requiring high-fidelity results. Locatello et al. demonstrated that unsupervised disentanglement may be theoretically unidentifiable without additional inductive biases [27]. Dai and Wipf observed that strong KL regularization can cause models to ignore important but less variable data aspects [28], while Raffin et al. found overly disentangled representations can impair performance by neglecting task-relevant feature correlations [29].

Interestingly, reduced disentanglement offers several benefits in certain contexts. Alemi et al.’s information bottleneck framework demonstrates that relaxing disentanglement constraints achieves better balance between representation learning and reconstruction [30]. Rolinek et al. showed that allowing coupling better captures inherently correlated generative factors [31]. Esmaeili et al. observed that preserving natural correlations between features produces more realistic samples than forcing complete independence [32]. These findings suggest optimal disentanglement is context-dependent, requiring calibration to specific application requirements.

In single-cell data, interpretability is assessed by how effectively different cell subpopulations can be distinguished [33–35]. Cells from different lineages exhibit divergent gene expression patterns, while similar lineages show correlated patterns reflecting developmental relationships. Understanding how representation learning extracts gene expression patterns fundamentally involves distinguishing between cell subpopulations, linking interpretability to the correlations within feature patterns of these subpopulations.

In this study, we investigate the relationship between latent space interpretability and representation learning disentanglement in VAE models applied to single-cell data. We enhance standard VAE architecture by introducing a low-dimensional interpretation module that projects latent representations into a lower-dimensional space optimized using reconstruction loss. This approach encourages the model to better understand its learned latent space, improving clustering performance while maintaining biological relevance.

Results

Comprehensive workflow design for single-cell VAE architecture exploration

As shown in Fig. 1, our systematic optimization of VAE architectures for single-cell analysis encompassed 31 diverse scRNA-seq datasets (16 cancer-related, 15 developmental) from 28 independent studies, incorporating varied experimental conditions including human interventions, subpopulation enrichment strategies, and multi-batch designs to ensure robust generalizability. Following standardized preprocessing, we evaluated models using both internal clustering metrics—ASW (Average Silhouette Width), DB (Calinski-Harabasz), and DB (Davies-Bouldin indices) and external metrics—NMI (Normalized Mutual Information) and ARI (Adjusted Rand Index), supplemented by correlation analysis of latent variables. Key findings revealed superior performance at lower KL regularization coefficients ($\beta$), indicating reduced disentanglement benefits single-cell data representation—a counterintuitive discovery that guided subsequent architectural refinements. Our optimized VAE architecture was rigorously benchmarked against established dimensionality reduction methods and validated across multiple biological contexts, demonstrating how reduced disentanglement preserves biologically meaningful gene expression correlations that enhance interpretability and clustering performance in transcriptomic data.

Fig. 1.

Research design workflow: analysis of 31 datasets from 28 studies using iVAE architecture, with systematic monitoring, refinement, evaluation, and comparing using multiple internal and external clustering metrics

Minimal disentanglement constraints enhance unsupervised clustering performance

We systematically evaluated VAE performance across eight $\beta$ values (50, 10, 5, 1, 0.5, 0.1, 0.05, 0.01) using both internal (ASW, CH, DB) and external (NMI, ARI) clustering metrics on 16 cancer and 15 developmental datasets (Fig. 2A, C), finding that minimal disentanglement constraint ($\beta =0.01$) consistently demonstrated superior performance. As shown in Fig. 2B and D, final VAE models with $\beta =0.01$ achieved statistically significant improvements in most evaluation metrics compared to higher $\beta$ values (especially the internal clustering metrics which utilized the latent space as input). For cancer datasets, maximum improvements occurred against $\beta =50$ (NMI: +0.583, ARI: +0.584, ASW: +0.218, DB: +0.931, CH: +2235.26, overall: +0.963), with similar patterns in developmental datasets (NMI: +0.587, ARI: +0.588, ASW: +0.204, DB: +0.919, CH: +2120.25, overall: +0.962) (Tables 1 and 2). We validated these findings in hematopoiesis [36] and endocrinogenesis [37] datasets with ten technical replicates (Additional file 1: Fig. S1 A, B), where comparing $\beta =0.01$ to $\beta =5.0$ revealed substantial improvements for hematopoiesis (ARI: +0.523, NMI: +0.525, ASW: +0.250, DB: +1.228, CH: +1145.890, overall: +0.859) and endocrinogenesis (ARI: +0.357, NMI: +0.359, ASW: +0.251, DB: +1.352, CH: +1366.682, overall: +0.863) datasets (Additional file 2: Tables S1, S2). Even the modest transition from $\beta =0.05$ to $\beta =0.01$ yielded measurable improvements. Visualization of latent component intensities (Additional file 1: Fig. S1C) demonstrated that the $\beta =0.01$ model revealed more distinct biological patterns in cell population distributions, while cell type distribution visualization (Additional file 1: Fig. S1D) confirmed that developmental trajectories and lineage relationships were better preserved at lower $\beta$ values, aligning with established hierarchies.

Fig. 2.

Performance evaluation across 8 $\beta$ values. A Metrics monitoring across 16 cancer datasets using 8 $\beta$ values (50, 10, 5, 1, 0.5, 0.1, 0.05, 0.01), measuring external clustering metrics (NMI, ARI), internal metrics (ASW, CH, DB), and summed absolute Pearson correlations between latent components (lower DB indicates better performance). B Evaluation of final VAE models at 8 $\beta$ values across cancer datasets (n = 16) using 5 clustering metrics and overall score. Statistical significance relative to $\beta$ = 0.01 was determined using Bonferroni correction (7 comparisons). For cancer datasets: paired t-tests were employed for comparisons that satisfied normality assumptions, while non-parametric Wilcoxon signed-rank tests were applied for metrics violating normal distribution (ARI and NMI at $\beta$ = 0.1, and overall score at $\beta$ = 0.1, 0.05). Significance marked with $\ast$. C Metrics monitoring across 15 developmental datasets. D Evaluation of final VAE models across developmental datasets (n = 15). For developmental datasets: statistical analysis followed the same principle with paired t-tests used for normally distributed comparisons, while Wilcoxon signed-rank tests were applied due to violation of normality assumptions (CH at $\beta$ = 5, 0.05, and overall score at $\beta$ = 50, 0.5, 0.05)

Table 1.

Performance improvements of $\beta =0.01$ compared to other $\beta$ values on cancer datasets

$\mathit{\beta }$ comparison	NMI	ARI	ASW	DB	CH	Overall
0.01 vs 0.05	0.041	0.041	0.026	0.092	586.96	0.116
0.01 vs 0.1	0.063	0.063	0.028	0.072	628.66	0.140
0.01 vs 0.5	0.090	0.090	0.058	0.177	777.37	0.211
0.01 vs 1	0.103	0.103	0.092	0.384	1181.05	0.307
0.01 vs 5	0.240	0.240	0.185	1.003	1991.41	0.677
0.01 vs 10	0.365	0.366	0.202	0.974	2127.22	0.791
0.01 vs 50	0.583	0.584	0.218	0.931	2235.26	0.963

Table 2.

Performance improvements of $\beta =0.01$ compared to other $\beta$ values on developmental datasets

$\mathit{\beta }$ comparison	NMI	ARI	ASW	DB	CH	Overall
0.01 vs 0.05	0.026	0.027	0.020	0.047	478.97	0.086
0.01 vs 0.1	0.023	0.023	0.029	0.085	620.79	0.117
0.01 vs 0.5	0.048	0.048	0.054	0.188	788.53	0.189
0.01 vs 1	0.083	0.083	0.090	0.403	1121.13	0.321
0.01 vs 5	0.260	0.260	0.165	0.926	1824.33	0.683
0.01 vs 10	0.382	0.383	0.189	0.985	2009.04	0.813
0.01 vs 50	0.587	0.588	0.204	0.919	2120.25	0.962

Interpretation reconstruction module enhances VAE clustering performance

We integrated an irecon (interpretation reconstruction) module into the standard VAE framework, creating an enhanced architecture designated iVAE (Fig. 3A), which demonstrated substantial performance improvements across diverse biological contexts. For 16 cancer datasets (Fig. 3B, Table 3), iVAE achieved significant enhancements across all clustering metrics (NMI: +0.029, ARI: +0.029, ASW: +0.048, DB: +0.199, CH: +1838.660), with an impressive overall improvement score of 0.950 and 14 out of 16 datasets showing maximum performance gains. Similarly, across 15 developmental datasets (Fig. 3D, Table 4), iVAE improved all metrics with comparable magnitudes (NMI: +0.031, ARI: +0.031, ASW: +0.055, DB: +0.316, CH: +1707.172), yielding an overall improvement score of 0.947.

Fig. 3.

irecon-enhanced VAE architecture and evaluation. A Architecture of VAE with irecon module. B Performance comparison between standard VAE and iVAE across 16 cancer datasets measuring external (NMI, ARI) and internal (ASW, CH, DB) clustering metrics and latent variable correlations. C Evaluation of VAE versus iVAE final models across cancer datasets (n = 16). Statistical significance was assessed using paired t-tests for metrics that satisfied normality assumptions (ARI, NMI, ASW, CH, and overall), while Wilcoxon signed-rank test was applied for DB metric due to violation of normality distribution. Significance relative to iVAE marked with $\ast$. D Corresponding performance comparison across 15 developmental datasets. E Evaluation of final models across developmental datasets (n = 15). Statistical analysis followed identical methodology: paired t-tests were used for normally distributed metrics (ARI, NMI, ASW, CH, and overall), while non-parametric Wilcoxon signed-rank test was employed for DB metric due to non-normal distribution

Table 3.

Performance improvements of iVAE on cancer datasets

Dataset	NMI	ARI	ASW	DB	CH	Overall
GSE148218 bmALL	0.055	0.056	0.011	0.064	2225.151	1.000
GSE262288 breastMetasis	0.004	0.004	0.062	0.298	1674.175	1.000
GSE155109 bcEC	0.010	0.010	0.009	0.060	754.879	1.000
GSE123813 bcc	0.028	0.028	0.047	0.200	2342.487	1.000
GSE155109 bcStroma	0.069	0.069	0.148	0.486	2989.251	1.000
GSE143423 tnbcCancerBrain	0.022	0.022	0.027	0.013	543.104	1.000
GSE123813 scc	0.025	0.025	0.011	0.005	739.455	1.000
GSE123902 LungAdre	0.013	0.013	0.039	0.350	1517.017	1.000
GSE225600 breastCancer	0.019	0.019	0.009	−0.039	1364.171	0.600
GSE235787 bcellsALL	0.030	0.031	0.037	−0.008	2890.894	0.600
GSE222002 Tcells	0.048	0.048	0.141	0.599	3035.678	1.000
GSE98638 TcellLiver	0.040	0.039	0.054	0.234	688.648	1.000
GSE222369 NKsLymphoma	0.015	0.015	0.025	0.241	979.335	1.000
GSE183904 gastric	0.045	0.045	0.038	0.227	1915.451	1.000
GSE132509 acutelymluekPBMC	0.020	0.020	0.059	0.145	2030.651	1.000
GSE143423 lbmCancerBrain	0.018	0.018	0.059	0.303	3728.214	1.000
Average improvement	0.029	0.029	0.048	0.199	1838.660	0.950

Table 4.

Performance improvements of iVAE on developmental datasets

Dataset	NMI	ARI	ASW	DB	CH	Overall
GSE132188 endo	0.063	0.064	0.097	0.484	677.421	1.000
GSE253355 bmNicheHm	0.050	0.050	0.096	0.330	2867.454	1.000
GSE167597 spineMm	−0.037	−0.037	0.018	0.111	2268.167	0.200
GSE144024 hESC	0.018	0.018	0.065	0.312	2092.427	1.000
GSE192857 hESCHmTimes	0.018	0.018	0.052	0.215	3033.527	1.000
GSE255019 hemato	0.011	0.011	0.066	0.240	2563.423	1.000
GSE226131 HSCMmAged	0.030	0.030	0.063	0.404	1490.858	1.000
GSE165844 LSKMmBatch	0.004	0.004	0.036	0.392	780.993	1.000
GSE141259 lung	0.069	0.069	0.037	0.432	1297.674	1.000
GSE95753 dentate	0.027	0.027	0.061	0.331	1833.078	1.000
GSE226824 ifnHSPC	0.021	0.021	0.024	0.115	748.052	1.000
S-SUBS8 setty	0.052	0.052	0.101	0.645	1489.701	1.000
GSE148215 hESCHSPCD8Hm	0.094	0.094	0.054	0.312	878.721	1.000
GSE120446 bm	0.006	0.006	0.025	0.295	1792.557	1.000
GSE120505 bloodAged	0.041	0.041	0.024	0.117	1793.531	1.000
Average improvement	0.031	0.031	0.055	0.316	1707.172	0.947

Validation in hematopoiesis and endocrinogenesis systems with ten technical replicates confirmed consistent benefits (Additional file 1: Fig. S2A, D), with hematopoiesis dataset showing greater improvements (Additional file 2: Table S3: ARI: +0.054, NMI: +0.055, ASW: +0.054, DB: +0.330, CH: +524.042, overall: +0.586). Visualization of cell distributions and latent component intensities (Additional file 1: Fig. S2C, F) revealed improved separation between biological populations, with iVAE more accurately representing developmental hierarchies from stem cells to differentiated lineages in hematopoiesis and better delineating between pancreatic endocrine and exocrine lineages in endocrinogenesis.

Statistical testing confirmed significant improvements across internal evaluation metrics (Fig. 3C, E), which directly measure lineage and state separation and cohesion utilizing the latent space as input. External metrics also demonstrated improvements—as confirmed through quantitative assessment which evaluating concordance with known biological classifications—a capability previously lacking in unsupervised single-cell analysis tasks.

iVAE outperforms VAE variants and dimensionality reduction methods

We evaluated iVAE against established VAE variants and dimensionality reduction techniques across diverse biological datasets, finding that iVAE consistently outperformed four VAE variants across 16 cancer datasets (Fig. 4A), with the greatest significant improvement against DIP-VAE (overall: 0.815, ASW: +0.094, DB: +0.246, CH: +2822.474), followed by TC-VAE (overall: +0.751), beta-VAE (+0.742), and Info-VAE (Information Maximizing Variational Autoencoder) (+0.736) (Table 5, Fig. 4B). Similar patterns emerged for 15 developmental datasets (Fig. 4C, D, Table 6), where iVAE demonstrated superior performance versus DIP-VAE (overall: +0.807), TC-VAE (+0.733), Info-VAE (+0.717), and beta-VAE (+0.709). Technical validation using ten replicates of hematopoiesis and endocrinogenesis datasets (Additional file 1: Fig. S3A) showed iVAE’s greatest advantage over DIP-VAE in hematopoiesis (ARI: +0.077, NMI: +0.077, ASW: +0.132, DB: +0.481, CH: +1823.936, overall: +0.717) (Additional file 2: Table S4) and endocrinogenesis (ARI: +0.112, NMI: +0.112, ASW: +0.151, DB: +0.639, CH: +2103.070, overall: +0.739) (Additional file 2: Table S5). The final model evaluation results (Additional file 1: Fig. S3B) confirmed these advantages quantitatively, with iVAE achieving the highest composite scores in both biological systems with statistical significance. Visualization of latent components (Additional file 1: Fig. S3C) revealed iVAE provided better separation between stem, progenitor, and mature populations in hematopoiesis and more accurately preserved relationships between pancreatic lineages in endocrinogenesis. The cell type distribution visualization (Additional file 1: Fig. S3D) further highlighted how iVAE more effectively captured biologically meaningful transitions between cell states. Against dimensionality reduction techniques, iVAE demonstrated marked superiority on cancer datasets (Additional file 1: Fig. S3E, Table 7), with the most significant improvement versus scVI (Single-cell Variational Inference) (ASW: +0.153, DB: +0.690, CH: +1084.544, overall: +0.937), while for developmental datasets (Fig. 4F, Table 8), performance margins against ICA (independent component analysis) (overall: +0.446) and KPCA (kernel principal component analysis) (+0.180) were larger than in cancer datasets, suggesting iVAE provides particular advantages in capturing complex developmental trajectories.

Fig. 4.

Comparative evaluation of iVAE against VAE variants and dimensionality reduction techniques. A Performance comparison of VAE variants (DIP-VAE, TC-VAE, info-VAE, $\beta$-VAE) and iVAE across 16 cancer datasets. B Evaluation of final models across cancer datasets (n = 16). Statistical significance relative to iVAE was determined using Bonferroni correction (4 comparisons). Paired t-tests were used for most comparisons, while Wilcoxon signed-rank tests were applied due to violation of normality distribution for ARI (DIP-VAE, TC-VAE, $\beta$-VAE), NMI (no violations), ASW (DIP-VAE, TC-VAE, info-VAE, $\beta$-VAE), and CH (TC-VAE). Significance marked with $\ast$. C Corresponding performance comparison across 15 developmental datasets. D Evaluation of VAE variants against iVAE across developmental datasets (n = 15) with identical statistical methodology and normality test results. E Comparison of dimensionality reduction techniques (PCA, KPCA, ICA, scVI) against iVAE across cancer datasets (n = 16). Paired t-tests were used for most comparisons, while Wilcoxon signed-rank tests were applied for ARI (no violations), NMI (no violations), ASW (PCA, KPCA, ICA), and CH (scVI). F Corresponding comparison across developmental datasets (n = 15) with identical statistical methodology and normality test results

Table 5.

Performance improvements of iVAE compared to VAE cariants on cancer datasets

VAE variant	ASW	DB	CH	Overall
iVAE vs DIP-VAE	0.094	0.246	2822.474	0.815
iVAE vs TC-VAE	0.099	0.325	1768.179	0.751
iVAE vs beta-VAE	0.080	0.279	2376.782	0.742
iVAE vs Info-VAE	0.074	0.268	2411.667	0.736

Table 6.

Performance improvements of iVAE compared to VAE variants on developmental datasets

VAE variant	ASW	DB	CH	Overall
iVAE vs DIP-VAE	0.080	0.291	2727.340	0.807
iVAE vs TC-VAE	0.085	0.392	1607.164	0.733
iVAE vs Info-VAE	0.069	0.294	2303.742	0.717
iVAE vs beta-VAE	0.067	0.266	2289.249	0.709

Table 7.

Performance improvements of iVAE compared to benchmarking methods on cancer datasets

Benchmark	ASW	DB	CH	Overall
iVAE vs scVI	0.153	0.690	1084.544	0.937
iVAE vs ICA	0.030	0.074	847.981	0.361
iVAE vs KPCA	0.006	0.081	369.491	0.171
iVAE vs PCA	0.002	0.077	363.282	0.164

Table 8.

Performance improvements of iVAE compared to benchmarking methods on developmental datasets

Benchmark	ASW	DB	CH	Overall
iVAE vs scVI	0.158	0.696	1041.462	0.939
iVAE vs ICA	0.055	0.156	857.048	0.446
iVAE vs KPCA	0.027	0.110	240.196	0.180
iVAE vs PCA	0.018	0.090	246.107	0.154

Technical reproducibility and parameter robustness of iVAE

We evaluated iVAE reproducibility across ten technical replicates using two multi-batch datasets, finding consistent superior performance against all VAE variants in the GSE165844 multi-batch LSK (Lineage$_{}^{-}$ Sca-1$_{}^{+}$ c-kit$_{}^{+}$) dataset (Fig. 5A, Table 9), with greatest improvements versus DIP-VAE (ARI: +0.123, NMI: +0.123, ASW: +0.049, DB: +0.389, CH: +1525.887, overall: +0.670), followed by TC-VAE (overall: +0.604), info-VAE (+0.319), and beta-VAE (+0.291), all statistically significant (Fig. 5B). Similarly, in the GSE185993 multi-batch AML (acute myeloid leukemia) cancer dataset (Fig. 5C, Table 10), iVAE demonstrated substantial improvements versus DIP-VAE (ARI: +0.092, NMI: +0.092, ASW: +0.080, DB: +0.265, CH: +8331.314, overall: +0.774), followed by beta-VAE (+0.495), info-VAE (+0.461), and TC-VAE (+0.382), with statistical significance across multiple metrics (Fig. 5D). Notably, improvement magnitudes were larger in the complex AML dataset than in the LSK dataset, suggesting particular advantages in handling heterogeneous cancer transcriptomes.

Fig. 5.

Technical reproducibility assessment of iVAE in multi-batch data integration. A Performance comparison of VAE variants (DIP-VAE, TC-VAE, info-VAE, $\beta$-VAE) and iVAE across 10 technical replicates using GSE165844 (multi-condition LSK dataset) evaluated on 5 unsupervised clustering metrics: Adjusted Rand Index (ARI), Normalized Mutual Information (NMI), Average Silhouette Width (ASW), Calinski-Harabasz index (CH), and Davies-Bouldin index (DB), along with composite overall scores. B Final model evaluation on LSK dataset with statistical significance relative to iVAE marked with $\ast$. Statistical significance was assessed using Bonferroni correction (4 comparisons) with paired t-tests applied for most metric comparisons. Wilcoxon signed-rank tests were used for specific non-normally distributed comparisons: $\beta$-VAE vs. iVAE (ARI), info-VAE vs. iVAE (CH), $\beta$-VAE vs. iVAE (CH), and DIP-VAE vs. iVAE (DB), as determined by Shapiro-Wilk normality testing. C Corresponding performance comparison using GSE185993 (multi-batch AML cancer dataset) across 10 technical replicates with the same evaluation metrics. D Final model evaluation on AML dataset where predominantly paired t-tests were applied, with Wilcoxon signed-rank test used only for DIP-VAE vs. iVAE comparison on the overall metric due to normality violation

Table 9.

10 technical repeats performance improvements of iVAE compared to VAE variants on multi-batch LSK dataset

Benchmark	ARI	NMI	ASW	DB	CH	Overall
iVAE vs DIP-VAE	0.114	0.115	0.065	0.486	513.638	0.621
iVAE vs TC-VAE	0.084	0.084	0.055	0.406	392.045	0.487
iVAE vs beta-VAE	0.021	0.021	0.034	0.241	362.702	0.256
iVAE vs info-VAE	0.013	0.013	0.036	0.240	370.050	0.245

Table 10.

10 technical repeats performance improvements of iVAE compared to VAE variants on multi-batch AML cancer dataset

Comparison	ARI	NMI	ASW	DB	CH	Overall
iVAE vs DIP-VAE	0.117	0.118	0.078	0.297	603.155	0.663
iVAE vs info-VAE	0.046	0.046	0.037	0.199	469.667	0.370
iVAE vs beta-VAE	0.048	0.048	0.036	0.191	461.538	0.368
iVAE vs TC-VAE	0.027	0.027	0.059	0.175	270.590	0.304

Parameter sensitivity analysis across decreasing irecon component weights (1, 0.5, 0.1, 0.05, 0.01) revealed a consistent pattern of performance enhancement with lower values (Additional file 1: Fig. S4A, B). For the hematopoiesis dataset (Additional file 2: Table S6), increasing irecon values progressively improved performance across all metrics, with the most substantial improvements observed at irecon = 1 compared to irecon = 0.01 (ASW: +0.054, CH: +1001.394, DB: +0.099, overall: +0.625). Similarly, the endocrinogenesis dataset (Additional file 2: Table S7) exhibited consistent improvement in clustering metrics with increasing irecon values (irecon = 1 compared to irecon = 0.01: ASW: +0.036, CH: +1360.632, DB: +0.094, overall: +0.545). Two dimensional interpretation embedding showcasing a more clear differentiation trajectory as irecon value increasing (Additional file 1: Fig. S4C). Besides, the other parameters like number of latent (Additional file 1: Fig. S4D) and dimension of the interpretation embedding (Additional file 1: Fig. S4E) did not indicate any consistent trends of significant influence on the model performance.

iVAE latent components reveal clinically relevant gene signatures in AML

We applied iVAE to AML patient data using a systematic gene signature approach (Fig. 6A), analyzing the GSE185993 dataset projected onto iVAE’s interpretation embedding (Fig. 6B), which revealed 15 latent components (L0–L14) capturing distinct hematopoietic lineages and leukemic states. Components L8/L13 delineated the erythroid lineage through correlation with CNRIP1/HBA1 (Fig. 6C), where HBA1 serves as a canonical erythroid differentiation marker altered in erythroid-feature AML [38, 39]. The myeloid lineage was captured by L9/L11 through FCER1G/S100A4 (Fig. 6D), with S100A4 overexpressed in 83% of AML patients [40, 41]. Components L2–L5 highlighted myeloid maturation arrest genes including HLA-DRB1/HLA-DRA/RNASE6/CTSA (Fig. 6E), while fate determination genes emerged in L6/L7/L12 corresponding to PRKAR2B/FOXC1/RPS17 (Fig. 6F), with FOXC1 aberrantly expressed in 20% of AML cases [42] and serving as an independent predictor of decreased survival [43]. Stemness and proliferation signatures were captured by L0/L1/L10/L14 through SPINK2/TUBA1B/IGLL1/STMN1 (Fig. 6G), where SPINK2 represents an independent adverse biomarker for survival and elevated therapy resistance [44, 45].

Fig. 6.

iVAE identifies lineage and state specific signature genes in AML data. A Workflow for identifying signature genes using iVAE, based on Pearson correlation between gene expression and latent dimensions, extracting top correlated genes for each component. B AML patient cohort (GSE185993) visualized on iVAE’s two-dimensional interpretation embedding with latent components L0–14. C L8 and L13 correspond to erythroid lineage genes CNRIP1 and HBA1. D L9 and L11 correspond to myeloid lineage genes FCER1G and S100A4. E L2, L3, L4, and L5 correspond to myeloid maturation arrest genes HLA-DRB1, HLA-DRA, RNASE6, and CTSA. F L6, L7, and L12 correspond to fate determination genes PRKAR2B, FOXC1, and RPS17. G L0, L1, L10, and L14 correspond to stemness and proliferation genes SPINK2, TUBA1B, IGLL1, and STMN1

iVAE latent components reveal biological trajectories in hematopoietic recovery

We applied iVAE to investigate hematopoietic stem cell recovery following radiation-induced damage, wherein the two-dimensional latent embedding effectively captured distinct differentiation branches (Fig. 7A) and revealed the dynamic trajectories of LSK cells across multiple post-radiation timepoints (Fig. 7B). Key lineage markers—including the stemness factor Hlf, monocytic-dendritic regulator Irf8, myeloid factor Cebpa, and erythroid regulator Gata1—were localized to discrete regions in the latent space (Fig. 7C), while latent components associated with primitive-to-mature cellular transitions revealed disruption of progenitor cell status, displaying characteristic rise-and-fall patterns along the pseudotime axis (Fig. 7D). Gene Ontology Biological Process (GOBP) enrichment analysis of the top negatively correlated genes (Fig. 7E) further substantiated these findings by showing significant associations with hematopoiesis and cell differentiation pathways (Fig. 7F). Our analysis identified several key components tracking different cellular lineages: the dendritic cell differentiation component exhibited a gradual increase in LSK immune activity following radiation before eventually normalizing (Fig. 7G), with associated genes including well-established dendritic cell markers—Plbd1, Aif1, Lgals3, and Xcr1—(Fig. 7H) and functional enrichment confirming the involvement of mononuclear, lymphocyte and leukocyte differentiation, alongside immune and defense pathways linked to radiation injury response (Fig. 7I). Similarly, the myeloid lineage component revealed distinct temporal dynamics characterized by an initial intensification at day 2, followed by a reduction by day 8, and a secondary increase by day 11 post-radiation (Fig. 7J); the genes most correlated with this component, including Mpo, Ctsg, Anxa3, and Ms4a3 (Fig. 7K), were consistent with myeloid cell identity, with functional enrichment substantiating associations with myeloid homeostasis and activation of immune and defense responses pertinent to radiation-induced injury (Fig. 7L). Finally, the erythroid component captured pronounced shifts in erythroid lineage bias between days 8 and 11 post-radiation (Fig. 7M), with genes linked to this axis featuring established erythroid regulators such as Blvrb, Klf1, Aqp1, and Epor (Fig. 7N), and subsequent enrichment analysis confirming significant correlations with cell activation and hematopoiesis, as well as inflammatory and defense mechanisms implicated in the response to radiation damage (Fig. 7O).

Fig. 7.

iVAE reveals lineage bias in irradiated LSK cells. A Integrated radiation-injured LSK dataset from GSE278673 and GSE280305. B Two-dimensional iVAE interpretation embedding showing different time points post-radiation and 4-week recovery. C Embedding regions interpreted using lineage markers: Hlf (stemness), Cebpa (granulocytic), Irf8 (monocyte-dendritic), and Gata1 (erythro-megakaryocytic). D Pseudotime intensity visualization across embedding and distribution across batches. E Correlation heatmap between gene expression and pseudotime. F Enriched pathways for genes negatively correlated with pseudotime. Monocyte-dendritic lineage analysis: G intensity distribution across time points, H top correlated genes and marker expression, I functional enrichment of correlated genes. Myeloid lineage analysis: J intensity distribution across time points, K top correlated genes and marker expression, L functional enrichment of correlated genes. Erythroid lineage analysis: M intensity distribution across time points, N top correlated genes and marker expression, O functional enrichment of correlated genes

Discussion

The balance between disentanglement and other performance metrics presents a crucial challenge in representation learning like we have observed in our study that lower disentanglement of latent space may increase the clustering performance in single cell data area. While strong disentanglement enhances interpretability and controlled generation, it often compromises reconstruction quality and modeling capacity [46]. Recent approaches have addressed this trade-off through various strategies. Mathieu et al. [47] developed a framework that simultaneously controls disentanglement degree and information content. For the identifiability problem, Locatello et al. [48] demonstrated that minimal supervision can significantly improve disentangled representations. Rather than pursuing universal disentanglement, Adel et al. [49] proposed task-specific customization of disentanglement properties. As Bengio et al. [50] suggested, ideal representation learning balances disentanglement with task performance. In practice, due to the potential different hypothesis of data distribution, like in transcriptome biological data, the data is in the negative binomial distribution, appropriate degrees of disentanglement should be selected based on specific area of data types, enabling VAEs to fully realize their potential in both representation learning and generative modeling.

Single-cell RNA sequencing data fundamentally differs from conventional datasets used in computer vision or natural language processing. Unlike these domains, gene expression patterns in single-cell data are intrinsically coordinated, with genes functioning in modular networks that co-express in functionally related groups to collectively define cellular states or cell type signatures. Additionally, biological variables such as stemness versus differentiation state or metabolic activity versus cell cycle are inherently correlated rather than existing as independent features that can vary separately. Recognizing these unique characteristics, our iVAE architecture deliberately implements two key design principles: reducing the disentanglement degree through lower $\beta$ values and introducing a latent space interpretation module. This approach preserves and enhances naturally occurring feature correlations in single-cell data, allowing for more biologically meaningful representations. Specifically, our method better captures cellular lineage continuity by explicitly modeling biologically coordinated changes among latent variables, thereby more accurately representing cell differentiation trajectories and state transitions. Furthermore, by maintaining appropriate feature correlations rather than artificially separating them, our approach strengthens gene expression patterns associated with cell identity in the latent space.

As illustrated in Additional file 1: Fig. S5, our study is structured around a stepwise “From-to” conceptual framework, which systematically clarifies our research focus, rationale, and methodological flow. We begin by defining the origin (“From”) of our study: single-cell transcriptomic data, which represents a high-dimensional snapshot of the underlying biological system—fundamentally distinct from conventional modalities such as images or text. Addressing “How to process?” this data, we employ representation learning based on the variational autoencoder (VAE) architecture in response to the inherent unsupervised nature of most single-cell analysis tasks, providing a foundational model well-suited for capturing complex gene expression patterns without relying on prior class labels. For “How to evaluate?” these representations, we naturally adopt unsupervised internal clustering metrics that objectively quantify structural properties within the learned latent space. What these metrics assess (“What was evaluated?”) is the model’s ability to delineate cellular lineages and states—revealing how effectively biologically related cells are grouped and meaningful structures are preserved in the latent space. This evaluation approach works (“Why could this be evaluated?”) because it is grounded in the intrinsic property of single-cell data: gene expression patterns are inherently correlated due to shared lineage, state, and regulatory mechanisms, relationships that can be faithfully encoded by VAE latent components. Ultimately, our “From-to” logic culminates (“To”) in the overarching goal of producing representations that are not only robust in unsupervised settings but also biologically interpretable, thereby enhancing both the technical performance and scientific significance of representation learning in single-cell transcriptomics.

Further architectural analysis (Fig. 3A, Additional file 1: Fig. S6) highlights the essential roles of both the standard VAE encoder-decoder (“Forward”) and the irecon bottleneck module in the iVAE framework. When the “Forward” component is removed (oVAE), the model loses its capacity to capture correlations in the latent space, resulting in the model degenerating into a standard VAE variant. Empirically, we observe that only the complete iVAE—integrating both components—consistently achieves superior results across cancer (n = 16) and developmental (n = 15) datasets (p < 0.001), as shown by unsupervised internal clustering metrics. Moreover, incremental improvements with increasing irecon refinement values (Additional file 1: Fig. S4, Additional file 2: Tables S6, S7) further support the contribution of this synergy. Mechanistically, this combined architecture allows iVAE to model gene expression correlations, balance dimensionality reduction and signal preservation, and more effectively capture the continuous cellular variation in single-cell data. However, it is important to acknowledge that despite consistent and significant improvements in composite evaluation scores, our validation of the improved iVAE model across dataset replications revealed that external metrics like NMI and ARI did not consistently demonstrate statistically significant changes. We attribute this phenomenon to the inherent nature of single-cell data analysis as an unsupervised task without gold-standard clustering labels. The derivation of “true labels” involves computational processes with inherent randomness, potentially reducing the precision of external metrics. To address this limitation, we conducted technical replication experiments on individual datasets.

Further analysis of the projections of individual latent components into two-dimensional space revealed that latent components learned under lower KL divergence penalties were more likely to correspond to biologically meaningful cell subpopulation regions. This suggests that reduced disentanglement allows the model to more effectively capture correlations in gene expression patterns within single-cell data, thereby forming a latent space with clearer biological lineage or state distinctions. Such correlations are particularly prominent in single-cell datasets. For example, stem cells are typically in a quiescent cell cycle state, resulting in lower expression of cell cycle-related genes compared to mature cells. Additionally, stem cells do not express lineage-specific genes characteristic of mature cells from various lineages. These gene expression correlations provide a foundation for iVAE to construct a latent space that better distinguishes between different cell lineages and states. By leveraging the latent space generated by iVAE, where each latent component corresponds to specific data regions and marker genes, we can interpret lineage or state heterogeneity in an unsupervised manner. Compared to traditional decomposiiton methods, iVAE provides a quantitative measure of lineage ro state heterogeneity strength through its latent components. In the future, we aim to develop VAE framework tailored for multi-omics and multimodal data, balancing between the disentanglement and other performance to further enhance the model in single-cell associated data area.

Conclusions

Our study reveals that reducing disentanglement in VAEs significantly enhances clustering performance on single-cell transcriptomic data. The proposed iVAE architecture with its innovative irecon module consistently outperformed eight established dimensionality reduction methods across five clustering metrics when evaluated on 31 diverse datasets, demonstrating robust reproducibility across technical replicates and parameter settings. Biological validation through analyses of AML samples and hematopoietic recovery models confirmed that iVAE’s latent components capture meaningful cellular transitions and provide quantitative measures of subpopulation heterogeneity. Rather than maximizing disentanglement, preserving biologically relevant correlations in gene expression patterns yields more interpretable representations of cellular states.

Methods

The iVAE model architecture

The iVAE model comprises three main components: the encoder, the decoder, and the latent space interpretation module, structured as follows.

Encoder

The encoder maps input data $x\in {\mathbb{R}}^{d_x}$ to the latent space $z\in {\mathbb{R}}^{d_z}$, producing latent distribution parameters:

$$\begin{array}{c}\hfill q_{\mu },q_{\sigma }=\text{Encoder}(x)\end{array}$$ 1

The standard deviation is constrained to be positive using the softplus activation:

$$\begin{array}{c}\hfill q_{\sigma }=\text{Softplus}(q_{\sigma })+ϵ,ϵ={10}^{-6}\end{array}$$ 2

The latent variable z is sampled from a Gaussian distribution via the reparameterization trick:

$$\begin{array}{c}\hfill z\sim \mathcal{N}(q_{\mu },q_{\sigma }),z=q_{\mu }+q_{\sigma }·ϵ,ϵ\sim \mathcal{N}(0,I)\end{array}$$ 3

The encoder’s neural network structure consists of:

$$\begin{array}{c}\hfill \text{Encoder}(x)=\text{ReLU}(\text{Linear}(x))\to \text{ReLU}(\text{Linear}(·))\to \text{Linear}(·)\end{array}$$ 4

Decoder

The decoder maps the latent variable z back to the input space, generating reconstructed data $\widehat{x}\in {\mathbb{R}}^{d_x}$:

$$\begin{array}{c}\hfill \widehat{x}=\text{Decoder}(z)\end{array}$$ 5

The decoder output is normalized using softmax activation:

$$\begin{array}{c}\hfill \widehat{x}=\text{Softmax}(\text{Linear}(\text{ReLU}(\text{Linear}(\text{ReLU}(\text{Linear}(z))))))\end{array}$$ 6

The decoder includes a learnable dispersion parameter $\text{disp}\in {\mathbb{R}}^{d_x}$ to adjust the distribution of generated data.

Negative binomial log-likelihood for loss calculation

The iVAE model employs NB (negative binomial) log-likelihood for count-based data modeling. The NB distribution is parameterized by $\mu$: mean of the distribution (expected value of reconstructed data), $\theta$: dispersion parameter controlling distribution variance, and x: gene expression counts.

The probability mass function of the NB distribution is:

$$\begin{array}{c}\hfill P(x|\mu ,\theta )=\frac{\mathrm{\Gamma }(x+\theta )}{\mathrm{\Gamma }(\theta )·\mathrm{\Gamma }(x+1)}·{\left(\frac{\theta }{\theta +\mu }\right)}^{\theta }·{\left(\frac{\mu }{\theta +\mu }\right)}^x\end{array}$$ 7

The log-likelihood used for reconstruction loss is:

$$\begin{array}{cc}\hfill logP(x|\mu ,\theta )& =\theta ·(log\theta -log(\theta +\mu ))+x·(log\mu -log(\theta +\mu ))\hfill \\ \hfill & +log\mathrm{\Gamma }(x+\theta )-log\mathrm{\Gamma }(\theta )-log\mathrm{\Gamma }(x+1)\hfill \end{array}$$ 8

Latent space interpretation module

The latent space interpretation module consists of two linear layers that encode and decode the latent space z, producing an intermediate representation $l\in {\mathbb{R}}^{d_l}$ and a reconstructed latent variable $z^{\prime }\in {\mathbb{R}}^{d_z}$:

$$\begin{array}{c}\hfill l=\text{LatentEncoder}(z),l\in {\mathbb{R}}^{d_l}\end{array}$$ 9

$$\begin{array}{c}\hfill z^{\prime }=\text{LatentDecoder}(l),z^{\prime }\in {\mathbb{R}}^{d_z}\end{array}$$ 10

The module is defined as:

$$\begin{array}{c}\hfill \text{LatentEncoder}(z)=\text{Linear}(z),\text{LatentDecoder}(l)=\text{Linear}(l)\end{array}$$ 11

Overall workflow

The input data x is processed through the encoder to generate the latent variable z with its parameters:

$$\begin{array}{c}\hfill q_{\mu },q_{\sigma },z=\text{Encoder}(x)\end{array}$$ 12

The latent variable z passes through the interpretation module to produce l and $z^{\prime }$:

$$\begin{array}{c}\hfill l=\text{LatentEncoder}(z),z^{\prime }=\text{LatentDecoder}(l)\end{array}$$ 13

The decoder processes both z and $z^{\prime }$ to generate reconstructed data:

$$\begin{array}{c}\hfill \widehat{x}=\text{Decoder}(z),{\widehat{x}}^{\prime }=\text{Decoder}(z^{\prime })\end{array}$$ 14

Loss functions

Overall architecture of VAE

Our foundational VAE model employs variational inference to model latent variables for data generation. For input cell x, we aim to find its latent representation z by maximizing the evidence lower bound (ELBO):

$$\begin{array}{c}\hfill \mathcal{L}(\varphi ,\theta ;x)={\mathbb{E}}_{q_{\varphi }(z|x)}[log{p}_{\theta }(x|z)]-D_{\mathit{KL}}(q_{\varphi }(z|x)\Vert p(z))\end{array}$$ 15

Here, $\theta$ represents generative distribution parameters, $\varphi$ represents variational distribution parameters, ${\mathbb{E}}_{q_{\varphi }(z|x)}[log{p}_{\theta }(x|z)]$ denotes reconstruction loss, and $D_{\mathit{KL}}(q_{\varphi }(z|x)\Vert p(z))$ is the KL divergence between variational and prior distributions.

Irecon loss function of iVAE

The iVAE loss function extends $\beta$-VAE by incorporating an interpretation module. This module consists of two linear layers f that encode and reconstruct latent space z to obtain $z^{\prime }$, computing a secondary reconstruction loss. Parameter $\gamma$ tunes this irecon term:

$$\begin{array}{c}\hfill z^{\prime }=f(z)\end{array}$$ 16

$$\begin{array}{cc}\hfill {\mathcal{L}}_{\beta ,\gamma }(\varphi ,\theta ;x)& ={\mathbb{E}}_{q_{\varphi }(z|x)}[log{p}_{\theta }(x|z)]-\beta D_{\mathit{KL}}(q_{\varphi }(z|x)\Vert p(z))\hfill \\ \hfill & +\gamma {\mathbb{E}}_{q_{\varphi }(z^{\prime }|x)}[log{p}_{\theta }(x|z^{\prime })]\hfill \end{array}$$ 17

$$\begin{array}{cc}\hfill \underset{\varphi ,\theta }{max}{\mathcal{L}}_{\beta ,\gamma }(\varphi ,\theta ;x)& =\underset{\varphi ,\theta }{max}({\mathbb{E}}_{q_{\varphi }(z|x)}[log{p}_{\theta }(x|z)]\hfill \\ \hfill & -\beta D_{\mathit{KL}}(q_{\varphi }(z|x)\Vert p(z))\hfill \\ \hfill & +\gamma {\mathbb{E}}_{q_{\varphi }(z^{\prime }|x)}[log{p}_{\theta }(x|z^{\prime })])\hfill \end{array}$$ 18

Loss function of implemented $\beta$-VAE

The $\beta$-VAE enhances regularization by incorporating a scaling factor for the KL divergence term, improving latent representation disentanglement:

$$\begin{array}{c}\hfill \underset{\varphi ,\theta }{max}{\mathcal{L}}_{\beta }(\varphi ,\theta ;x)=\underset{\varphi ,\theta }{max}({\mathbb{E}}_{q_{\varphi }(z|x)}[log{p}_{\theta }(x|z)]-\beta D_{\mathit{KL}}(q_{\varphi }(z|x)\Vert p(z)))\end{array}$$ 19

When $\beta =1$, this reduces to the standard VAE.

Loss function of implemented TC-VAE

The TC-VAE loss incorporates total correlation (TC) to constrain latent representations, promoting disentanglement:

$$\begin{array}{c}\hfill q_{\varphi }(z)=\int q_{\varphi }(z|x)p(x)dx\end{array}$$ 20

$$\begin{array}{c}\hfill \text{TC}(z)=D_{\mathit{KL}}\left(q_{\varphi },(z),\Vert ,\prod _{j=1}^d,q_{\varphi }(z_j)\right)\end{array}$$ 21

$$\begin{array}{cc}\hfill {\mathcal{L}}_{\mathit{TC}}(\varphi ,\theta ;x)& ={\mathbb{E}}_{q_{\varphi }(z|x)}[log{p}_{\theta }(x|z)]-D_{\mathit{KL}}(q_{\varphi }(z|x)\Vert p(z))\hfill \\ \hfill & -\alpha ·\text{TC}(z)\hfill \end{array}$$ 22

Parameter $\alpha$ adjusts the weight of the total correlation term.

Loss function of implemented DIP-VAE

DIP-VAE applies regularization to encourage the joint distribution of latent variables to resemble the product of independent distributions:

$$\begin{array}{cc}\hfill {\text{Reg}}_{\text{DIP}}& ={\lambda }_{\text{od}}\sum _{i\ne j}{({\mathrm{\Sigma }}_{p,ij}-{\mathrm{\Sigma }}_{q,ij})}^2\hfill \\ \hfill & +{\lambda }_{\text{d}}\sum _i{({\mathrm{\Sigma }}_{p,ii}-{\mathrm{\Sigma }}_{q,ii})}^2\hfill \end{array}$$ 23

$$\begin{array}{c}\hfill {\mathcal{L}}_{\mathit{DIP}}(\varphi ,\theta ;x)=\mathcal{L}(\varphi ,\theta ;x)+{\text{Reg}}_{\text{DIP}}\end{array}$$ 24

Here, ${\mathrm{\Sigma }}_{p,ij}$ and ${\mathrm{\Sigma }}_{q,ij}$ denote covariance matrices of prior and latent variable distributions.

Loss function of implemented Info-VAE

Info-VAE regularizes the latent space via sample-wise KL divergence and marginal distribution KL divergence, while considering mutual information between input and latent variables:

$$\begin{array}{cc}\hfill {\mathcal{L}}_{\mathit{info}}(\varphi ,\theta ;x)& ={\mathbb{E}}_{q_{\varphi }(z|x)}[log{p}_{\theta }(x|z)]\hfill \\ \hfill & -(\alpha D_{\text{KL}}(q_{\varphi }(z|x)|p(z))\hfill \\ \hfill & +\beta D_{\text{KL}}(q_{\varphi }(z)|p(z))-\lambda I_q(x;z))\hfill \end{array}$$ 25

Analysis methods

Training process monitoring

During training, each data batch is used for both model weight updates and latent space interpretability evaluation, measuring metrics such as NMI, ARI, ASW, CH, DB, and total Pearson correlation.

Clustering metrics

Normalized Mutual Information (NMI) measures clustering consistency with original labels:

$$\begin{array}{c}\hfill \text{NMI}(U,V)=\frac{2·\text{MI}(U,V)}{H(U)+H(V)}\end{array}$$ 26

Adjusted Rand Index (ARI) measures similarity between clustering results:

$$\begin{array}{c}\hfill \text{ARI}=\frac{{\sum }_{\mathit{ij}}\left(\begin{array}{c}{n}_{\mathit{ij}}\\ 2\end{array}\right)-[{\sum }_i\left(\begin{array}{c}{a}_i\\ 2\end{array}\right){\sum }_j\left(\begin{array}{c}{b}_j\\ 2\end{array}\right)]/\left(\begin{array}{c}n\\ 2\end{array}\right)}{\frac{1}{2}[{\sum }_i\left(\begin{array}{c}{a}_i\\ 2\end{array}\right)+{\sum }_j\left(\begin{array}{c}{b}_j\\ 2\end{array}\right)]-[{\sum }_i\left(\begin{array}{c}{a}_i\\ 2\end{array}\right){\sum }_j\left(\begin{array}{c}{b}_j\\ 2\end{array}\right)]/\left(\begin{array}{c}n\\ 2\end{array}\right)}\end{array}$$ 27

Calinski-Harabasz score (CH) compares between-cluster and within-cluster covariance:

$$\begin{array}{c}\hfill \text{CH}=\frac{\text{tr}(B_k)}{\text{tr}(W_k)}\times \frac{n-k}{k-1}\end{array}$$ 28

Davies-Bouldin score (DB) evaluates subgroup compactness and separation:

$$\begin{array}{c}\hfill \text{DB}=\frac{1}{c}\sum _{i=1}^c\underset{i\ne j}{max}\left(\frac{S_i+S_j}{M_{\mathit{ij}}}\right)\end{array}$$ 29

Silhouette score measures clustering quality by comparing intra-cluster and nearest-cluster distances:

$$\begin{array}{c}\hfill s(i)=\frac{b(i)-a(i)}{max\{a(i),b(i)\}}\end{array}$$ 30

Correlation metric

We use the sum of absolute values of upper triangular elements in the Pearson correlation matrix as a linear correlation metric:

$$\begin{array}{c}\hfill S=\sum _{i=1}^n\sum _{j=i+1}^n|R_{\mathit{ij}}|\end{array}$$ 31

Aim, design, and setting of the study

The aim of this study was to develop and evaluate iVAE, a novel representation learning framework specifically designed for single-cell RNA sequencing data analysis. Unlike traditional VAEs, iVAE was designed to preserve the intrinsic correlations between gene expression patterns that reflect biological relationships between cell lineages and states. We conducted a comprehensive computational study comparing iVAE against standard VAE architectures and other representation learning methods. Our experimental design involved systematic evaluation of different algorithms, parameter settings, and representation learning approaches on identical datasets to ensure fair comparability. Unlike traditional wet-lab experiments, our computational approach allowed for controlled comparison of multiple methods on the same underlying data, eliminating biological variability as a confounding factor. The primary objective was to evaluate how different levels of disentanglement in VAE architectures affect the ability to capture biologically meaningful cell states and lineages in single-cell data. We specifically investigated whether reduced disentanglement, contrary to conventional machine learning wisdom, might better preserve the inherent biological relationships in single-cell transcriptomic data.

Characteristics of materials

This study analyzed two categories of single-cell RNA-seq datasets: cancer tissue-derived single-cell datasets ($n=16$) and normal tissue development-related single-cell datasets ($n=15$). All datasets were primarily obtained from the Gene Expression Omnibus (GEO) public repository, selected based on stringent quality criteria established in previously published high-impact studies. Our experimental design incorporated two complementary types of replication: biological replication, achieved by including multiple independent cancer and normal tissue biological datasets representing diverse tissue origins, cell states, and disease conditions; and technical replication, performed by conducting 10 independent technical replicate analyses on selected datasets (S-SUBS8, GSE132188, GSE165844, GSE185993) to verify result stability and reproducibility.

Statistical analysis

All comparative analyses employed a paired experimental design with rigorous statistical methodology to ensure robust evaluation of method performance. Prior to statistical testing, data normality was assessed using the Shapiro-Wilk test ($p>0.05$ indicating normal distribution) for each method and metric combination. For two-group comparisons, paired t-tests were applied when data met normality assumptions, while Wilcoxon signed-rank tests were employed for non-normally distributed data. Multiple comparison corrections were applied using Bonferroni correction. All error bars represent standard deviation (SD) with statistical significance indicated using standardized notation ($\ast p<0.05$, $\ast \ast p<0.01$, $\ast \ast \ast p<0.001$). All tests were two-tailed with $p<0.05$ considered statistically significant.

Abbreviations

iVAE

Interpretation Variational Autoencoder

irecon

Interpretation reconstruction

DIP-VAE

Disentangled Inferred Prior Variational Autoencoder

TC-VAE

Total Correlation Variational Autoencoder

Info-VAE

Information Maximizing Variational Autoencoder

NB

Negative binomial

PCA

Principal component analysis

KPCA

Kernel principal component analysis

ICA

Independent component analysis

scVI

Single-cell Variational Inference

AML

Acute myeloid leukemia

LSK

Lineage$_{}^{-}$Sca-1$_{}^{+}$c-Kit$_{}^{+}$

KL

Kullback-Leibler divergence

NMI

Normalized Mutual Information

ARI

Adjusted Rand Index

ASW

Average Silhouette Width

CH

Calinski-Harabasz index

DB

Davies-Bouldin index

Authors’ contributions

Z.F.: conceptualization, methodology, software, validation, formal analysis, investigation, data curation, resources, writing—original draft, writing—review and editing, visualization. C.C.: formal analysis, data curation, visualization, writing—original draft, writing—review and editing. S.W.: resources, supervision, project administration. J.W.: resources, supervision, project administration, funding acquisition. S.C.: writing—review and editing, resources, supervision, project administration, funding acquisition.

Funding

This research was funded by the National Natural Science Foundation of China of funder grant number 82222060, 82430103, 82473572, 81930090, 81725019, 82073487, and 81602790.

Data availability

All data generated or analyzed during this study are included in this published article, its supplementary information files, and publicly available repositories. The source code for this study is publicly available in the GitHub repository at https://github.com/PeterPonyu/iVAE and Zenodo at https://doi.org/10.5281/zenodo.15686686.

The following datasets were utilized for cancer-related analyses:

-Mehtonen, J., Teppo, S., Lahnalampi, M., Kokko, A., Kaukonen, R., Oksa, L., Laukkanen, S., Bouvy-Liivrand, M., Lönnberg, T., Lohi, O. and Heinäniemi, M. Single cell characterization of arrested B-lymphoid differentiation and leukemic cell states in ETV6-RUNX1-positive pediatric leukemia. GEO https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE148218 (2020); [51]

-Li, Z. and Luo, L. Single-Cell RNA Sequencing Identifies Molecular Biomarkers Predicting Response to CDK4/6 Inhibition in Metastatic HR+/HER2- Breast Cancer. GEO https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE262288 (2025); [52]

-Geldhof, V., de Rooij, L. P., Sokol, L., Amersfoort, J., De Schepper, M., Rohlenova, K., Hoste, G., Vanderstichele, A., Delsupehe, A., Isnaldi, E., Dai, N., Taverna, F., Khan, S., Truong, A. K., Teuwen, L., Richard, F., Treps, L., Smeets, A., Nevelsteen, I., Weynand, B., Vinckier, S., Schoonjans, L., Kalucka, J., Desmedt, C., Neven, P., Mazzone, M., Floris, G., Punie, K., Dewerchin, M., Eelen, G., Wildiers, H., Li, X., Luo, Y. and Carmeliet, P. Single cell atlas identifies lipid-processing and immunomodulatory endothelial cells in healthy and malignant breast. GEO https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE155109 (2022); [53]

-Yost, K. E., Satpathy, A. T., Wells, D. K., Qi, Y., Kageyama, R., Wang, C., Sarin, K. Y., Brown, R. A., Bucktrout, S. L., Davis, M. M., Chang, A. S. and Chang, H. Y. Clonal replacement of tumor-specific T cells following PD-1 blockade [single cells]. GEO https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE123813 (2019); [54]

-Wang, L., Dai, J., Han, R. and Jin, W. Single-Cell Map of Diverse Immune Phenotypes in the Metastatic Brain Tumor Microenvironment of Non Small Cell Lung Cancer and triple negative breast cancer. GEO https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE143423 (2025); [55]

-Laughney, A. M., Hu, J., Campbell, N. R., Bakhoum, S. F., Setty, M., Lavallée, V. P., Xie, Y., Masilonis, I., Carr, A. J., Allaj, V., Mattar, M., Xavier, J., Mazutis, L., Poirier, J. T., Rudin, C. M., Pe’er, D. and Massagué, J. The single cell transcriptional landscape of human lung adenocarcinoma (primary tumours and metastases). GEO https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE123902 (2020); [56]

-Liu, Y., Ge, J., Chen, Y. and Yu, K. Single-cell profiling and spatial transcriptome of primary tumors and paired metastatic lymph nodes in breast cancer patients. GEO https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE225600 (2023); [57]

-Yang, J., Yu, J., Huang, X. and Li, Y. Single-cell network pharmacology predicts total therapies targeting multiple developmental clones in B-cell acute lymphoblastic leukemia. GEO https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE235787 (2024); [58]

-Chen, X. and Li, Z. Targeting tumor cells toward the antigenic specificity of bystander T cells in tumor microenvironment potentiates cancer immunotherapy. GEO https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE222002 (2024); [59]

-Zheng, C., Zheng, L., Yoo, J., Guo, H., Zhang, Y., Guo, X., Kang, B., Hu, R., Huang, J., Zhang, Q., Liu, Z., Dong, M., Hu, X., Ouyang, W., Peng, J. and Zhang, Z. Landscape of infiltrating T cells in liver cancer revealed by single-cell sequencing. GEO https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE98638 (2019); [60]

-Cichocki, F. and Day, A. Nicotinamide enhances natural killer cell function and yields remissions in patients with non-Hodgkin lymphoma. GEO https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE222369 (2023); [61]

-Kumar, V., Ramnarayanan, K. and Tan, P. Single-cell atlas of lineage states, tumor microenvironment and subtype specific expression programs in gastric cancer. GEO https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE183904 (2024); [62]

-Caron, M., St-Onge, P., Sontag, T., Wang, Y. C., Richer, C., Ragoussis, I., Sinnett, D. and Bourque, G. Single-cell analysis of childhood leukemia reveals a link between developmental states and ribosomal protein expression as a source of intra-individual heterogeneity. GEO https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE132509 (2021); [63]

The following datasets were utilized for developmental and normal tissue analyses:

-Bastidas-Ponce, A., Tritschler, S., Leander, D., Scheibner, K., Tarquis-Medina, M., Salinno, C., Schirge, S., Burtscher, I., Böttcher, A., Theis, F. J., Lickert, H. and Bakhti, M. Comprehensive single-cell mRNA profiling reveals a detailed roadmap for pancreatic endocrinogenesis. GEO https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE132188 (2025); [37]

-Bandyopadhyay, S., Ahn, K. J., Duffy, M., Qin, L. and Tan, K. Mapping the Cellular Biogeography of Human Bone Marrow Niches Using Single-Cell Transcriptomics. GEO https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE253355 (2024); [64]

-Alkaslasi, M. R., Piccus, Z. E., Hareendran, S., Silberberg, H., Chen, L., Zhang, Y., Petros, T. J. and Le Pichon, C. E. Single nucleus RNA-sequencing defines unexpected diversity of cholinergic neuron types in the adult mouse spinal cord. GEO https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE167597 (2024); [65]

-Wang, H., He, J., Xu, C., Chen, X., Yang, H., Shi, L., Liu, B. and Zhou, J. Decoding Human Megakaryocyte Development. GEO https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE144024 (2020); [66]

-Spildrejorde, M., Samara, A., Sharma, A., Falck, M., Leithaug, M., Modafferi, S., Bjørnstad, P. M., Gervin, K., Lyle, R. and Eskeland, R. Gene expression analysis of hESCs undergoing neuronal differentiation [scRNA-seq]. GEO https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE192857 (2023); [67]

-Poscablo, D., Forsberg, C. and Medina, P. An age specific platelet differentiation path from hematopoietic stem cells causes thrombocytosis and exacerbated thrombosis. GEO https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE255019 (2024); [68]

-Cosgrove, J. and Perie, L. Single-cell RNA sequencing of hematopoietic stem and progenitor cells from young and aged mice. GEO https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE226131 (2023); [69]

-Fast, E. External signals regulate continuous transcriptional states in hematopoietic stem cells. GEO https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE165844 (2024); [70]

-Simon, L. M. and Schiller, H. B. Longitudinal single-cell transcriptomics analysis of mouse lung upon bleomycin-induced injury. GEO https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE141259 (2022); [71]

-Linnarsson, S. Transcriptome analysis of single cells from the mouse dentate gyrus. GEO https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE95753 (2020); [72]

-Demerdash, Y., Bouman, B. J., Haghverdi, L. and Essers, M. A. Time series single-cell RNA sequencing of murine hematopoetic stem and progenitors (HSPCs) following in vivo IFN$\mathrm{\alpha }$ treatment. GEO https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE226824 (2024); [73]

-Setty, M. Profiling of CD34+ cells from human bone marrow to understand hematopoiesis. ENA https://www.ebi.ac.uk/ena/browser/view/PRJEB37166 (2019); [36]

-Zhu, Y., Wang, T., Gu, J. and Pan, G. Characterization and generation of human definitive multipotent hematopoietic stem/progenitor cells. GEO https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE148215 (2020); [74]

-Dillon, L. Human Bone Marrow Assessment by Single Cell RNA Sequencing, Mass Cytometry and Flow Cytometry. GEO https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE120446 (2018); [75]

-Teo, Y. V., Hinthorn, S. J., Webb, A. E. and Nicola, N. Single-cell RNA seq of aging peripheral blood. GEO https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE120505 (2023). [76]

Declarations

Ethics approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.