Abstract
Clustering analysis is a fundamental step in scRNA-seq data analysis. However, its reliability is compromised by clustering inconsistency among trials due to stochastic processes in clustering algorithms. Despite efforts to obtain reliable and consensus clustering, existing methods cannot be applied to large scRNA-seq datasets due to high computational costs. Here, we develop the single-cell Inconsistency Clustering Estimator (scICE) to evaluate clustering consistency and provide consistent clustering results, achieving up to a 30-fold improvement in speed compared to conventional consensus clustering-based methods, such as multiK and chooseR. Application of scICE to 48 real and simulated scRNA-seq datasets, some with over 10,000 cells, successfully identifies all consistent clustering results, substantially narrowing the number of clusters to explore. By enabling the focus on a narrower set of more reliable candidate clusters, users can greatly reduce computational burden while generating more robust results.
Subject terms: Software, Functional clustering, Data processing, Statistical methods
Accurate identification of cell types in vast single-cell datasets is a major challenge. Here, authors deliver scICE, a computational tool that ensures clustering consistency with up to 30-fold speed improvement, empowering more robust and rapid insights.
Introduction
Single-cell RNA sequencing (scRNA-seq) technology has revolutionized the study of cellular heterogeneity by providing high-resolution scRNA-seq data that captures gene expression profiles at the single-cell level1–3. The vast amount of information in scRNA-seq data makes it suitable for a variety of analyses, including differentially expressed gene (DEG) analysis to elucidate the functionality of specific cell types, and ligand-receptor analysis to explore interactions between different cell types4–8. One essential step in these downstream analyses is identifying cell types using clustering algorithms, which group cells with similar gene expression profiles.
The most widely used algorithms for clustering analysis of scRNA-seq data are graph-based clustering methods, such as the Louvain and Leiden algorithms, due to their speed and efficiency9–12. However, these methods rely on stochastic processes, leading to variability in clustering results across different runs. Specifically, these algorithms search for the optimal partition of each cell in a random order, meaning the resulting cluster labels can vary from run to run depending on the chosen random seed; in the worst-case scenario, altering the seed can result in drastically different cell assignments13,14. For example, when clustering is performed in Seurat with its tutorial dataset, significant differences can arise in clustering outcomes depending on random seeds (Fig. 1a). This variation leads to the disappearance of previously detected clusters or the emergence of entirely new clusters across runs (the red-dashed boundary in Fig. 1a). When we applied the same approach to clustering analyses from two recently published studies15,16 using their publicly available code and data, again simply changing the random seed led to significant label variations, with some clusters disappearing entirely (Fig. 1b).
Fig. 1. Different random seeds in the clustering algorithm yield different clustering outcomes.
a For the Seurat tutorial dataset, using a random seed of 88 yields five T-cell sub-clusters. However, when the seed is changed to 863, only four clusters emerge, with one of the initially identified clusters no longer detected (red-dashed boundaries). b This inconsistency is also confirmed in recent studies. For instance, modifying random seeds in the codes used to cluster non-small cell lung cancer tumor cells considerably alters clustering results (UMAP embeddings, black boxes, top row). Specifically, using a random seed of 1275190282 reveals a green cluster (red dashed boundary, top row), whereas switching to a seed of 1360516437 causes this cluster to fragment into more than three sub-clusters (zoomed-in embeddings, red boxes, top row). Similarly, in the placental cells data, the cyan cluster is split into the light pink and light purple clusters after changing the random seed (zoomed-in embeddings, red boxes, bottom row). Source data are provided as a Source Data file.
The inconsistency in clustering results can significantly undermine the reliability of assigned labels, making it essential for users to assess whether labels remain stable across different runs. Thus, several algorithms have been developed to estimate clustering consistency, including SC3, SCENA, scCCESS, multiK, and chooseR17–21. These methods generate multiple clustering labels through different strategies: SC3 and SCENA achieve this by varying the number of principal components or genes used, respectively, while scCCESS repeatedly applies random projections to the data. Meanwhile, multiK and chooseR produce multiple labels by sampling sub-datasets from the original data. Therefore, these methods inherently require repeated execution of computationally intensive processes—including preprocessing, dimensionality reduction (DR), and clustering with varying user-defined parameters—making them impractical for large datasets exceeding 10,000 cells21–23. Additionally, their computational cost is further increased by the need to construct a consensus matrix, a computationally expensive process that evaluates clustering consistency by determining whether all pairs of cells are co-clustered across iterations24,25. Furthermore, the stability metrics based on this consensus matrix, such as the relative proportion of ambiguous clustering used in MultiK, depend on additional hyperparameters to define the boundary between clear and ambiguous consensus17, limiting reproducibility and interpretability. Besides these limitations related to algorithm speed and metric interpretability, the conventional algorithms primarily focus on finding the optimal label that achieves the best consensus, making it challenging to comprehensively evaluate clustering consistencies across different numbers of clusters17,18,20,21,23. These multifaceted shortcomings of the conventional algorithms hinder the systematic and scalable evaluation of clustering stability in large scRNA-seq data.
To comprehensively and efficiently evaluate clustering consistency, we developed the single-cell Inconsistency Clustering Estimator (scICE). Unlike conventional methods, scICE assesses the clustering consistency across multiple labels generated by simply varying the random seed in the Leiden algorithm, eliminating the need for repetitive data generation. To evaluate label consistency, scICE employs the inconsistency coefficient (IC)26, a metric that neither requires hyperparameters nor relies on the computationally expensive consensus matrix. This approach enables efficient parallel processing without intensive data transfer between processors. As a result, scICE achieves up to 30-fold speed improvement compared to multiK and chooseR. With its computational efficiency, scICE effectively identifies multiple sets of consistent cluster labels across different numbers of clusters. When applied to 48 real and simulated scRNA-seq datasets4,27–30, scICE revealed that only ~30% of clustering numbers between 1 and 20 were consistent. By providing a compact set of consistent cluster labels, scICE minimizes unnecessary exploration in selecting cluster labels, thereby enhancing the efficiency and reliability of clustering analysis. Moreover, incorporating scICE into sub-clustering workflows enhances the detection of rare cell subtypes that are difficult to identify using a single resolution parameter. By providing an efficient and scalable solution for clustering consistency evaluation, scICE improves the robustness and reliability of scRNA-seq analysis, facilitating more reliable biological discoveries.
Results
Efficient clustering consistency evaluation using the inconsistency coefficient (IC) and parallel processing
To demonstrate how IC is calculated, we used a graph composed of 50 cells. When the clustering algorithm is applied five times with different random seeds, two different cell clusters, c1 and c2, are obtained with the probability p1 = 0.4 and p2 = 0.6, respectively (Fig. 2a). Next, the similarity between the two different c1 and c2 cluster labels is quantified using element-centric similarity31 (ECS), which yields a more intuitive and unbiased comparison than other label similarity metrics31 (Fig. 2b). Computing ECS involves calculating the difference in affinity matrices ((i) and (ii) of Fig. 2b)—which capture the similarity structure between cells based on their shared cluster memberships—between c1 and c2 and summing row-wise to obtain the L1 vector, where each element represents the total affinity difference of each cell ((iii) of Fig. 2b). Next, the L1 vector is subtracted from one, yielding the ECS vector, with each element indicating the agreement of each cell’s membership across the pair of labels ((iv) of Fig. 2b). Therefore, the average of this ECS vector (S12 = 0.74) represents the overall similarity between c1 and c2 31 (Fig. 2c). After calculating the similarity between all possible unique pairs of labels, the similarity matrix S, where element Sij is the similarity of label ci and cj, is constructed. Finally, the IC is calculated using the inverse of pSpT, where p (p1, p2)), which, in this example, yielded an IC of 1.13 (Fig. 2d) (see the “Methods”). The IC close to 1 occurs when the cluster similarity () is close to 1, indicating high similarity between c1 and c1 ((i) and (iii) in Fig. 2d). It can also occur when p1 is nearly 0 or 1, indicating that one cluster label is dominant ((ii) and (iii) in Fig. 2d). Thus, IC close to 1 implies that the yielded labels are consistent, making them reliable (Fig. 2d). In contrast, an IC gradually rises above 1 as p1 approaches 0.5 and approaches 0, indicating substantial differences between c1 and c2 ((i)–(iii) in Fig. 2d). These conditions are associated with an increase in the number of cells exhibiting membership inconsistency. For example, when the proportion of cells with inconsistent membership is 0.5, 1, and 2%, IC is ~1.01, ~1.02, and ~1.04, respectively (Supplementary Fig. 1).
Fig. 2. Quantification of clustering consistency using Inconsistency Coefficients (IC).
a After five iterations of clustering with different random seeds, two different cluster labels, and , are obtained two times and three times, respectively. b To quantify the cell-to-cell relationship similarity between and , the element-wise absolute differences between the cell-to-cell affinity matrices of and labels are calculated (i and ii). After summing this absolute difference matrix row-wise, we obtain the L1 vector, where each element indicates the clustering inconsistency of a cell (iii). By subtracting the L1 vector from one, the clustering consistency vector, the element-centric similarity (ECS), is calculated (iv). c The mean of the ECS vector (S12 = 0.74) represents the overall similarity score between and . d Using this overall similarity score, the similarity matrix S is constructed. Then the weighted average of S, where the weight is occurrence probability vector p, is inversed to derive the inconsistency coefficient . As the cluster labels become more inconsistent (i and ii), IC becomes greater than 1 (iii). e After constructing the graph from the low-dimensional embedding obtained through preprocessing steps, including quality control and dimensionality reduction using scLENS (i), the constructed graph is transferred to multiple processors (ii). f On each processor, the Leiden algorithm, the graph-based clustering algorithm, is applied simultaneously to produce N clustering results at each resolution parameter. g–i Using this framework, the IC was calculated across various resolution parameters, for 6000 mouse brain cells. When these cells are clustered into six clusters, every clustering result is identical, resulting in an IC of 1, indicating consistent clustering (g). In contrast, when the cells are split into seven clusters, the IC increases to 1.11, indicating inconsistent clustering (h). IC decreases when the brain cells are grouped into 15 clusters despite the increased number of clusters (i). Source data are provided as a Source Data file.
Compared to the conventionally-used consensus matrix, this similarity matrix requires much lower computational cost, allowing us to more efficiently evaluate clustering consistency (see the “Discussion”). However, performing clustering multiple times to evaluate consistency can still be time-consuming. To reduce the time required for multiple clustering tasks, we utilized parallel processing. Specifically, we first applied typical quality control (QC) to filter out low-quality cells and genes and used the scLENS DR method32, which provides automatic signal selection to reduce the data size ((i) of Fig. 2e). Based on the distances between cells in the reduced data, we constructed a graph and distributed it to multiple processes running across multiple cores ((ii) of Fig. 2e). On each process, we then applied the Leiden algorithm to the distributed graph simultaneously (see the “Methods” for details). This parallel clustering approach enabled us to obtain multiple cluster labels at a single resolution with high-speed performance, facilitating quick acquisition of numerous labels across various resolution parameters (Fig. 2f).
We applied this process to existing mouse brain data containing ~6000 cells33. When a low-resolution parameter, yielding six clusters, was used, all labels were identical, and thus, IC was 1 (Fig. 2g). When a slightly increased resolution parameter, yielding seven clusters, was used, two different types of cluster labels occurred with a similar probability. As a result, IC was increased to 1.11, indicating high inconsistency of labels (Fig. 2h). When the resolution parameter was further increased, yielding 15 clusters, three different types of labels occurred (Fig. 2i). Since they are similar, the IC was decreased to 1.01, indicating that the labels with 15 clusters are more reliable than those with seven clusters (Fig. 2i). Thus, using this approach, we could determine that identifying seven groups from this data is unreliable due to their high inconsistency. By excluding such unreliable results, we can narrow down the reliable cluster label candidates for the analysis.
scICE: automated framework for estimating clustering consistency in scRNA-seq analysis
Using clustering inconsistency based on IC26, we can find and exclude unreliable clustering outputs. This approach can potentially resolve the current labor-intensive process in scRNA-seq clustering analysis when researchers manually check the suitability of the label from numerous candidates by coloring the clusters on low-dimensional embeddings such as tSNE or UMAP.
Thus, we developed a framework that evaluates the label consistency at a given number of clusters (Fig. 3a–d). To demonstrate the detailed procedure of this framework, we used a blood cell dataset containing ~6000 cells generated by mixing seven pre-sorted cell types28. To check whether the clustering results with seven clusters are consistent, we first used binary search (Supplementary Fig. 2 and Supplementary Fig. 3), which identified the resolution parameter range of [~1.38 × 10−4, 7.53 × 10−4] corresponding to seven clusters (the blue box in Fig. 3a) (See “Methods” for details). From this range, we uniformly sampled eleven resolution parameters (orange circles in Fig. 3a) and calculated IC at each resolution parameter after performing clustering 15 times (Fig. 3b). To make sure the calculated IC is optimal, we increased the iteration numbers of the Leiden algorithm until IC converges to its minimum (Fig. 3b and Supplementary Fig. 4) (See “Methods” for details). Then, we selected the resolution parameter with the minimum IC and the corresponding 15 cluster labels (orange boxes in Fig. 3b). Subsequently, to estimate label consistency more robustly, we bootstrapped these selected 15 labels 100 times and calculated 100 ICs. When the median of the calculated ICs is lower than the threshold of 1.005, corresponding to ~0.25% overall membership inconsistency (2–3 cells out of 1000 cells) (Supplementary Fig. 1), we determined that the label with that number of clusters is consistent. Therefore, the label with seven clusters is consistent as the median of IC at seven clusters is below the threshold (Fig. 3c). We then applied the same procedures for 1 to 20 clusters to assess the consistency at different cluster numbers. This yielded a set of cluster numbers with consistent clustering results, ranging from four to seven clusters (Fig. 3d), notably including the true number of cell types (n = 7). Combining all these procedures, we developed the scICE package, which provides a set of consistent clustering results.
Fig. 3. Overview of the scICE method for consistent clustering set determination.
a To estimate clustering consistency across different numbers of clusters, binary search is initially used to find the resolution parameter ranges corresponding to each number of clusters (Supplementary Fig. 2). For instance, the resolution parameter range for detecting seven clusters from blood cell data is narrowed down to [~1.38 × 10−4, ~7.53 × 10−4] (blue boxes). From this range, eleven resolution parameters are uniformly sampled (orange circles in the sub-graph). b The IC at each sampled resolution parameter is calculated after obtaining 15 cluster labels at each resolution parameter. Subsequently, the Leiden algorithm is applied to each cluster label iteratively until IC reaches one or the iteration number reaches 150 (Supplementary Fig. 4). Among the final IC values of eleven sampled resolution parameters, the minimum IC and corresponding resolution parameter is selected. c By bootstrapping 15 labels 100 times at the selected resolution parameter, 100 IC values are computed to calculate the median IC. When the median IC is lower than the threshold of 1.005 (red dashed line), indicating ~0.25% or less clustering inconsistency, the clustering result is determined as consistent. d After assessing the consistency of clustering for all numbers of clusters, the consistent clustering set (red arrow) is determined by excluding the cluster numbers where the median IC is higher than the threshold (shaded high inconsistency region). The box plot features a central line representing the median, a box defining the interquartile range (IQR) from the 25th to the 75th percentile, and whiskers extending to the furthest data points within 1.5 times the IQR from the box edges (Q1 and Q3). Data points beyond these whiskers are plotted individually as outliers. Source data are provided as a Source Data file.
scICE provides a reliable clustering set by excluding inconsistent labels
To assess the effectiveness of scICE, we applied it to 48 real and simulated scRNA-seq datasets4,27–30. The datasets have ground truth labels established independently of the scRNA-seq assay, thereby minimizing potential evaluation bias toward specific analysis tools27. Furthermore, they were generated by mixing presorted cells or created through the scDesign2 simulation framework30 to have varying data properties, including cluster size, imbalance ratio (i.e., the size ratio between the largest and smallest clusters), and presence of hierarchical cluster structure (Supplementary Table 1). For each dataset, scICE provided a consistent clustering set (black arrows in Fig. 4a)—a collection of cluster numbers that produces consistent labels within the initial set of candidate cluster numbers from 1 to 20 (Fig. 4a, gray arrow). On average, these consistent clustering sets included ~6 different cluster numbers, accounting for ~30% of the initial set (Fig. 4b). Depending on the dataset, scICE identified consistent clustering sets to include between 5% and 60% of the initial set (Fig. 4a). Notably, the larger consistent clustering sets were observed in the simulated immune cells and real blood cell datasets, both of which exhibit hierarchical cell-type relationships (Fig. 4a, red labels). In these cases, scICE identified multiple stable cluster labels at different hierarchical levels, leading to a larger consistent clustering set (Fig. 4c, red bar). This differs from conventional consensus clustering methods, which only provide a single optimal label, even for the dataset with a hierarchical structure17,18,20,21,23,34,35.
Fig. 4. Determining consistent clustering sets in 48 datasets using scICE.
a scICE is applied to 48 real and simulated datasets to determine the consistent clustering set (black arrows) within the number of clusters from 1 to 20 (the gray arrow titled the initial set). The consistent clustering sets include the optimal clustering results (blue stars), which most closely match the ground truth in all but two cases (the two datasets labeled in bold). b Across all datasets, the consistent clustering sets are obtained, on average, only ~30% of the initial set. (data are presented as mean ± SD; n = 48). c For datasets with a hierarchical structure (red labels in (e); n = 27), the consistent clustering sets are relatively larger due to the multi-level cluster structure, covering ~ 40% of the original set (red bar; data are presented as mean ± SD). In contrast, for datasets lacking a hierarchical structure (blue labels in (e); n = 21), the consistent clustering sets comprise only ~14% of the initial set, on average (blue bar; data are presented as mean ± SD). d When both the imbalance ratio and cluster sizes are small, indicating that the data consists of multiple small, similarly sized clusters, the size of the consistent clustering set increases. However, aside from this rare case in scRNA-seq data, scICE provides a substantially reduced clustering set, comprising only 5–15% of the initial set. e scICE completes the entire process in 20 min (green line), whereas multiK and chooseR require over five hours for the dataset with ~12,000 cells (blue and red lines). Source data are provided as a Source Data file.
In contrast, the consistent clustering sets of most simulated Tabula Muris datasets were substantially reduced in size (Fig. 4a, blue labels starting with sim_TMuris). These datasets were generated by mixing randomly selected cell types, thus lacking hierarchical cell-type relationships and having a few stable cluster labels. As a result, scICE achieved significant reductions in the size of consistent clustering sets, averaging ~14% of the initial set (blue bar in Fig. 4c).
Despite the lack of hierarchical structure, some datasets, such as ‘sim_TMuris_1019’, exhibited a large-sized consistent clustering set (Fig. 4a). To investigate this phenomenon, we analyzed all 19 simulated Tabula Muris datasets without hierarchical structure, examining how data properties like the number of cells per cluster and the imbalance ratio influence clustering performance20,36–38. Our analysis revealed that datasets with low imbalance ratios and small numbers of cells per cluster tended to exhibit a large consistent clustering set (Fig. 4d). In these cases, the data consists predominantly of small-sized clusters. In these small clusters, minor variations, such as noise, become more pronounced, potentially leading to overfitting39,40 (Supplementary Fig. 5). Specifically, noise can be misinterpreted as meaningful differences within small clusters, causing each cluster to erroneously split into multiple sub-clusters (Supplementary Fig. 5), leading to a larger consistent clustering set (Fig. 4d). In contrast, scICE identified a much smaller consistent clustering set, comprising only ~5 to 15% of the initial set when the data contained one or more large clusters due to a high imbalance ratio or a large number of cells per cluster (Fig. 4d). This indicates that as long as the data does not consist of only small clusters, scICE will identify a smaller consistent clustering set. This trend is common for scRNA-seq datasets, which contain large clusters due to their high heterogeneity and the large number of cells. Therefore, in most scRNA-seq datasets, scICE will provide a smaller consistent clustering set by effectively excluding unreliable clustering results.
Furthermore, even in these small, consistent clustering sets provided by scICE, the optimal clustering results that best match the ground truth labels are included in 46 out of 48 datasets (the blue stars in Fig. 4a). This demonstrates that scICE effectively enables users to identify optimal clusters from a limited set of reliable clustering results. Remarkably, this effectiveness is maintained despite variations in threshold settings. For example, even after significantly relaxing the IC threshold from 1.005 to 1.05, the consistent clustering set expanded only modestly, ensuring that the reliable cluster labels, including the optimal label, remained stable (Supplementary Fig. 6).
Moreover, these reliable cluster labels were obtained up to 30 times faster through scICE than multiK and chooseR (Fig. 4e). Notably, scICE completed the calculation within twenty minutes, whereas multiK and chooseR required more than five hours for the large data containing ~12,000 cells (Fig. 4e). In addition, for this data, scICE exhibited considerably lower peak memory usage of ~2 GB during the clustering process compared to chooseR (Supplementary Fig. 7). With such higher efficiency compared to the conventional methods, scICE can enhance the efficiency and accuracy of clustering analysis by providing limited number of reliable cluster labels.
scICE-based sub-clustering enhances cell subtype Identification
Although scICE proves its effectiveness by providing a compact consistent clustering set that includes the optimal clustering result, this inclusion is not always guaranteed. For example, using the Zheng data containing ~4800 blood cells (ZhengMix_4757) and simulated immune cell data containing ~3000 T-cells (sim_Tcell_10), the largest number of clusters in the consistent clustering set provided by scICE for each dataset is 6 and 11, respectively, despite the number of true cell types being higher, i.e., 8 and 13, respectively (Fig. 4a, bold red-labeled datasets). Specifically, attempting to identify all eight true cell types in the ZhengMix_4757 data induces the membership inconsistencies of cells located at the boundary between Naïve and Regulatory T-cell types, leading to a high IC of ~1.01 (yellow circle in Fig. 5a). Thus, within the consistent clustering set, naïve and regulatory T-cell types were not identified (Fig. 5b, c). Similarly, in the sim_Tcell_10 data, effector memory CD4 T-cells and tissue-resident memory CD8 T-cells were not classified within the consistent cluster labels (Fig. 5d–f). Indeed, increasing the resolution parameter to detect these smaller cell types caused unstable splitting of the cell population of naive central memory CD4 + T cells, leading to high clustering inconsistency. Therefore, the consistent cluster labels obtained using scICE may fail to include certain ground-truth cell subtypes, resulting in under-clustering where distinct cell subtypes are merged.
Fig. 5. scICE-based sub-clustering effectively identifies cell subtypes with high consistency.
a, b When attempting to identify all eight ground-truth clusters in ZhengMix_4757 data, high clustering inconsistency arises (IC ≈ 1.01; yellow circle in the shaded high inconsistency region of (a)), primarily due to the high membership inconsistency of cells located at the boundary between two subtypes, naïve and regulatory T cells (pink and purple cell points within the black border in (b)). c As a result, these subtypes are not identified in the consistent cluster label with the largest cluster number of six. d–f Similarly, attempting to detect all 13 ground-truth cell types in sim_Tcell_10 data leads to high clustering inconsistency (IC ≈ 1.01; yellow circle in the shaded high inconsistency region of (d)) due to the unstable population splitting of a naive central memory CD4 + T cell (dark purple cell in (e)). For this reason, effector memory CD4 and tissue-resident memory CD8 T cells are missed in the consistent cluster label (pink-colored cluster 6 within the black border in (f)). g–l To find these unidentified cell subtypes, scICE is applied for sub-clustering. g For the ZhengMix data, we initially extracted a sub-dataset containing the unidentified naïve and regulatory T cell types (green-colored cluster 4 within the black border in (c)). After applying scLENS and UMAP to this extracted sub-data, the graph is constructed. h–i From the constructed graph, scICE identifies two consistent sub-clusters (blue circle in (h)) corresponding to unidentified naïve and regulatory T-cells (i), respectively. j–l Similarly, for the simulated immune cell data, applying scICE to the sub-dataset (j) produces a single consistent cluster label (blue circle in (k)), allowing the identification of effector memory CD4 T cell and tissue-resident memory CD8 T cell types (l). In (a, d, h, and k), each box plot illustrates distribution of 100 IC values, with each set of 100 values generated by 100 iterations of bootstrapping 15 labels. These plots feature a central median line, an IQR box (25th–75th percentiles), whiskers extending 1.5 times the IQR to the furthest data points within this range, and individual outliers. Source data are provided as a Source Data file.
One potential reason for these under-clustering results is that while each true cell type has its optimal resolution, scICE estimates the label inconsistency at a single resolution. Furthermore, using a single resolution to identify multiple sub-clusters has a well-known limitation: it tends to favor clusters of a specific size38. For this reason, rare small-sized clusters could not be detected when we applied scICE to the entire graph with a single resolution.
Thus, sub-clustering is necessary to identify the cell subtypes at different resolutions more accurately. Specifically, to detect the naïve and regulatory T-cell types from the ZhengMix_4757 data with high clustering consistency, we initially extracted sub-data containing these cell subtypes (Cluster 4 in Fig. 5c). By applying scLENS and UMAP to the extracted sub-data, we constructed a sub-graph containing naïve and regulatory T-cells (Fig. 5g). After performing scICE on the constructed graph, a single consistent cluster label with two clusters was obtained (blue circle in Fig. 5h). Using this label, we successfully identified the naïve and regulatory T-cell types, which were not stably identified when clustering was performed at a single resolution parameter (Fig. 5i). By applying the same processes, involving scICE, we also identified effector memory CD4 T cell and tissue-resident memory CD8 T cell types with high consistency from the sim_Tcell_10 data (Fig. 5j–l).
Next, we applied the scICE-based sub-clustering analysis to the mouse gonadal white adipose tissue (GWAT) single-nucleus RNA sequencing (snRNA-seq) data41 and the mouse SARS-CoV-2 lung scRNA-seq data42 to test whether it can recover cell subtypes used for the pseudo-time analysis in previous studies. In the GWAT dataset, scICE initially identified twelve distinct clusters based on the consistent label (Fig. 6a, blue circle). We then performed scICE-based sub-clustering analysis on one of the twelve clusters (Cluster 3). This revealed sub-populations (Fig. 6b) that closely match previously identified cell subtypes for pseudo-time analysis, including monocytes and three distinct macrophage subtypes (Fig. 6c). Similarly, from the mouse SARS-CoV-2 lung scRNA-seq data, scICE-based sub-clustering also successfully identified key cell subtypes, including alveolar type 1 pneumocytes (AT1), alveolar type 2 pneumocytes (AT2), and primed AT2 cells (Fig. 6d–f). Finally, applying scICE-based sub-clustering to the spatial transcriptomics dataset43 revealed sub-structures within the brain region of the mouse embryo, such as the hippocampus (Supplementary Fig. 8). These results demonstrate that scICE-based sub-clustering analysis can consistently identify sub-clusters that may be difficult to detect with a single-resolution.
Fig. 6. scICE sub-clustering identifies cell subtypes in mouse gonadal white adipose tissue (GWAT) and SARS-CoV-2 infected lung data.
a scICE is applied to the mouse gonadal white adipose tissue (GWAT) snRNA-seq data and identifies up to twelve clusters (blue circle outside the shaded high inconsistency region). Among these identified clusters, scICE-based sub-clustering is applied to Cluster 3 (apricot-colored cell points within the black border). b, c. This analysis identifies three consistent labels defining two, three, and six clusters, respectively (IC distributions in (b)). The identified sub-clusters defined by the label yielding six clusters (six clusters in UMAP embedding of (b)) closely match the previously defined cell types, including monocytes and the three macrophage subtypes (c). d–f. Applying scICE-based sub-clustering to the mouse SARS-CoV-2 lung scRNA-seq data successfully uncovers pre-identified cell subtypes, including alveolar type 1 pneumocytes (AT1), alveolar type 2 pneumocytes (AT2), and primed AT2 cells. In (a, b, d, and e), each box plot displays distribution of 100 IC values derived from bootstrapped labels. These plots feature a central median line, an IQR box (25th–75th percentiles), whiskers extending 1.5 times the IQR to the furthest data points within this range, and individual outliers. Source data are provided as a Source Data file.
Discussion
Cluster labels obtained using clustering algorithms are fundamental for advanced downstream analyses of scRNA-seq data, such as differential expression analysis and cell-cell communication analysis, making their reliability and reproducibility crucial4–8. To enhance the reliability and reproducibility of cluster labels by identifying the most agreed-upon or consistent labels, many consensus clustering-based approaches have been proposed23. However, their performance in terms of speed is often sub-optimal22,23. For instance, methods that determine a consistent cluster label by estimating the consistency of labels obtained from different clustering algorithms, such as CHAI and scEFSC, are slow because all clustering algorithms must be executed to generate multiple cluster labels34,35. To yield multiple labels and find a consistent label among them, other approaches, such as multiK, chooseR, SC3, and scCCESS, perform multiple perturbations, such as repeated sub-sampling of cells or genes from the data or projecting data onto random matrices17–21. Thus, these approaches require repeated DR or graph-construction steps, slowing their speed17–21,23,34,35. In contrast, scICE evaluates only the label inconsistency caused by the stochastic processes in the Leiden algorithm. Therefore, scICE’s parallel processing requires only a graph and clustering algorithm for each processor, significantly reducing inter-processor data transfer (Fig. 2e). This streamlined approach enables more efficient implementation and ultimately achieves faster speeds than conventional methods.
Furthermore, some of these conventional approaches allow for the selective use of specific functions to evaluate label consistency. These functions rely on metrics derived from computationally expensive consensus matrices. Specifically, calculating these matrices requires checking whether each pair of cells out of total cells is co-clustered—equivalent to examining distinct pairs—and thus, results in complexity that quickly becomes prohibitive as grows. Thus, it is limiting to apply these conventional approaches to datasets containing more than 10,000 cells22. Specifically, for the data with ~12,000 cells, chooseR and multiK took over 5 h to analyze such datasets (Fig. 3i). On the other hand, scICE completed entire processes within several minutes by leveraging the ECS-based IC metric with parallel processing (Figs. 2a–e and 4e). When using ECS to compare two labels (with cells grouped into clusters of roughly equal sizes, respectively), the complexity for calculating affinity differences between each pair of clusters is ~ . Considering calculating all pairs of and clusters total complexity is . As and are typically much smaller than , the total complexity is far less than the cost of consensus matrices . Furthermore, scICE consumes much less memory than the conventional algorithms (Supplementary Fig. 7). Although it is considerably faster and more memory efficient than conventional methods, scICE can still be computationally intensive for extremely large datasets. For instance, when we applied scICE to a dataset of ~500,000 brain cells44 using eight processors, evaluating clustering consistency across a range of 1 to 20 clusters took over 35 h, with peak memory usage reaching ~23 GB (measured after the clustering process began). To reduce the computational cost, one can narrow the cluster search range, decrease the sampling frequency of resolution parameters, or limit the maximum number of Leiden iterations—though these optimizations may reduce accuracy.
Employing ECS enhances clarity in distinguishing clusters with high reliability compared to other similarity metrics, such as the adjusted Rand index (ARI) and normalized mutual information (NMI) (Supplementary Fig. 9). Specifically, ECS resulted in a larger difference between the IC values of reliable and unreliable labels than ARI and NMI. This finding indicates that calculating IC with ECS more effectively enables scICE to clearly separate reliable clusters from those with high clustering inconsistency.
Additionally, scICE was applied on UMAP graphs as UMAP graphs capture both global and local structures simultaneously45. Indeed, when scICE was applied to UMAP graphs, it provided larger consistent clustering sets on datasets with hierarchical structure compared to those obtained using graphs constructed with the k-nearest neighbor (kNN) and shared nearest neighbor (sNN)46 (Supplementary Fig. 10). For datasets without hierarchical structure, however, UMAP yielded sets that were comparable in size to those generated by kNN or even smaller than those obtained using sNN (Supplementary Fig. 10). These findings suggest that UMAP graphs more effectively capture hierarchical cluster structures than graphs constructed using kNN or sNN approaches. Nevertheless, various graphs, such as the structural shared nearest neighbor (ssNN), have been proposed for clustering47. Therefore, scICE can also serve as a means to assess which type of graph is more suitable for clustering in specific datasets.
scICE efficiently provides a consistent clustering set across various real and simulated scRNA-seq data. For datasets lacking a hierarchical structure, the consistent clustering set comprised only ~14% of the initial set, resulting in fewer consistent cluster labels (blue bar in Fig. 4c). Therefore, by allowing users to focus on a limited number of reliable labels, scICE enhances the efficiency of clustering analysis. Conversely, for datasets with a hierarchical structure, larger consistent clustering sets, with an average size of ~40% of the initial set, were observed (red bar in Fig. 4c) because a hierarchical structure inherently supports multi-level cluster labels. This demonstrates that scICE facilitates comprehensive hierarchical structure analysis by providing multiple consistent labels, whereas most conventional consensus clustering-based algorithms only provide a single optimal, consistent cluster label17,18,20,21,23,34,35. For example, from multiple consistent labels provided by scICE, parent-child relationships between clusters can be identified by connecting cluster pairs with the most shared memberships using the contingency matrix48. This matrix also allows for the identification and filtering of consistent labels that significantly violate parent-child relationships with other labels, thereby enhancing the accuracy of scICE’s clustering results.
Furthermore, scICE can provide the optimal clustering results as a consistent label in most cases (blue stars in Fig. 4a). However, in two cases, the consistent clustering set obtained by scICE did not include the optimal label due to the high clustering inconsistency caused by the single-resolution limit38 (two datasets labeled in bold in Figs. 4a and 5a–f). To circumvent this issue, we applied scICE-based sub-clustering to the two sub-datasets containing the unidentified cell types (Fig. 5g–l). As a result, scICE provided a single consistent label with two clusters corresponding to unidentified cell types in each data (Fig. 5i and l). Moreover, this scICE-based sub-clustering approach recapitulated the cell subtypes reported in previous studies but also uncovered additional hierarchical substructures in the spatial transcriptomic dataset (Figs. 6a–f and Supplementary Fig. 8). In this way, by providing a limited clustering set that accurately distinguished the true or pre-defined cell subtypes, scICE demonstrated its effectiveness even in the sub-clustering to identify the unidentified cell types due to the limitation of a single resolution.
scICE can also complement analyses performed by state-of-the-art clustering approaches. For example, the transfer learning method, like scArches, enables efficient label transfer from pre-trained reference models to query datasets49. However, when query datasets contain cell populations absent from the reference model, the transferred label may incorrectly annotate these undefined populations, resulting in labels that do not accurately reflect the underlying cluster structure50. As a result, this transferred label is often not reproducible by applying the Leiden algorithm to the graph derived from query data, resulting in a high IC value at the number of clusters specified by the transferred labels (Supplementary Fig. 11). Thus, providing an IC value for the transferred labels, scICE can assist users in deciding whether to trust these labels or to pursue alternative labeling strategies.
Although we demonstrated scICE’s effectiveness on 48 scRNA-seq datasets representing diverse clustering scenarios, it is necessary to evaluate its performance across a broader range of data types, including cases where the data exhibits modalities that make grouping cells particularly challenging. For example, in datasets containing cells undergoing differentiation, consistent clustering could not be easily achieved due to the presence of intermediate states. In such cases, consistent labels can be obtained by relaxing the IC threshold, which allows for the identification and removal of intermediate cells exhibiting low ECS scores due to discrepancies in the obtained labels. Removing these cells, which exhibit high membership inconsistency, facilitates the extraction of more robust clustering sets. In this way, scICE can be leveraged to identify robust and reliable cell groups using various data types.
scICE could encounter overfitting issues in datasets with small-sized clusters39,40, leading to the over-detection of clusters (Fig. 4d and Supplementary Fig. 5). While such instances are rare in scRNA-seq data, scICE may unnecessarily generate a larger clustering set in small-scale datasets or in cases where clusters are of similar sizes, potentially reducing efficiency (Fig. 4d). This issue could become more pronounced when performing sub-clustering on small clusters (Supplementary Fig. 5). Therefore, when applying a sub-clustering algorithm to a small cluster, users should exercise caution as the obtained results might represent artifacts due to overfitting rather than genuine biological variation. To address this challenge, biological validation through marker gene analysis or DEG analysis could be utilized to confirm the relevance of sub-clusters derived from small populations. Another potential solution for this is to exclude the consistent labels showing minimum statistical clustering significance as determined by clustering evaluation metrics such as the single-cell Significance of Hierarchical Clustering, Statistical Significance of Clustering (SigClust), and Clustering Deviation Index17,51,52.
In summary, scICE enables the evaluation of clustering consistency in large datasets by providing reasonable-speed performance and facilitating the exclusion of unreliable clustering results characterized by high inconsistency. Additionally, scICE enhances analysis efficiency and accuracy by allowing users to focus only on a limited set of reliable cluster labels. Therefore, scICE is anticipated to be widely adopted in various clustering analyses in future research.
Methods
QC and low-dimensional embedding
We performed QC to remove lowly expressed genes and low-quality cells from the scRNA-seq data. During this process, genes expressed in 15 or fewer cells and low-quality cells with 200 or fewer expressed genes were removed ((i) of Fig. 2e). After the QC, scLENS was applied to the filtered data to embed the cells in a low-dimensional space ((i) of Fig. 2e).
Graph construction using UMAP
The cosine distances between cells in a low-dimensional space were calculated. Based on these distances, a UMAP graph connecting neighboring cells was constructed using the UMAP.jl package ((ii) of Fig. 2e). The number of neighbors was set to 15 for the UMAP graph.
Clustering algorithm
To circumvent the limitations of the Louvain algorithm, which can lead to misclassification due to its inability to guarantee well-connected communities, we employed the Leiden algorithm with a randomness parameter set to 0.114. While this algorithm supports both Newman-Girvan (NG) modularity and the Constant Potts Model (CPM) as objective functions, we selected CPM to avoid non-monotonic relationships between the number of clusters and the resolution parameter53, which can reduce the efficiency of the binary search (Supplementary Fig. 12). Moreover, this choice circumvents the resolution limit inherent in NG modularity, which tends to merge smaller clusters into larger ones and hinders the accurate identification of distinct clusters36,37,54.
Multiple clustering with parallel computing
In our study, the constructed graph was distributed to 15 processes to execute the Leiden graph-based clustering algorithm on each processor simultaneously. This parallel computing enhances processing speed; however, increasing the number of processes raises memory usage, indicating a performance trade-off (Supplementary Fig. 13). Therefore, we allow users to set the number of processes based on their computer specifications to balance speed and memory usage during parallel computing.
Furthermore, the number of cluster labels, , obtained through this parallel clustering approach can be determined by the user as well. While a larger increases the accuracy of IC calculation, it increases the computation cost (Fig. 2f). To ensure a balance between computation speed and accuracy for IC calculation, we set to 10 ~ 20 for each resolution parameter.
Inconsistency coefficients (IC)
To quantify the overall consistency of different cluster labels in terms of IC26, we first compute the ECS31 for each cell. In order to calculate ECS, we construct an by affinity matrix Pa for a given cluster label , where is the total number of cells. Each element represents the connection strength between cells i and j, defined as follows:
| 1 |
where is the size of the cluster to which and belong, is the Kronecker delta (δij = 1 if i = j, and 0 otherwise), and is a parameter controlling intra-cluster weighting (here we use α = 0.9). If cells i and j belong to different clusters, then
| 2 |
For another cluster label, , we obtain affinity matrix Pb.
Using two affinity matrices of Pa and Pb, which are constructed from labels and , respectively, the ECS for each cell is defined as follows:
| 3 |
By averaging over all cells, we obtain the pairwise similarities between all unique pairs of cluster labels and :
| 4 |
Subsequently, we construct a similarity matrix S with elements .
Finally, the IC is calculated as the inverse of the weighted average similarity:
| 5 |
where is the number of unique labels and is the vector of relative frequencies for each label, satisfying . An IC value close to 1 indicates more consistent clustering, whereas an IC greater than 1 suggests increased inconsistency in clustering.
Binary search for determining resolution parameter ranges of each number of clusters
As the first step in evaluating clustering consistency for each number of clusters, a binary search determines the lower and upper bounds of the resolution parameter range for the target number of clusters.
Specifically, to search for the lower bound (LB), we initially set two resolution parameters corresponding to the number of clusters markedly higher and lower than the target number of clusters as the upper and lower search boundaries (USB and LSB), respectively (blue and purple boxes in step1 of Supplementary Fig. 2). The average of these two resolution parameters is set as the midpoint of the resolution parameter. At this midpoint of resolution parameter, the median number of clusters is calculated from ten cluster labels obtained by performing multiple clustering (yellow box in step1 of Supplementary Fig. 2). If the calculated median number of clusters at the midpoint is higher than the target number of clusters, the midpoint is set as the new USB; if lower, it is set as the new LSB, thus excluding half of the search range from the region of interest (step2 of Supplementary Fig. 2). From the new search boundaries, the midpoint of the resolution parameter and the corresponding median number of clusters are then recalculated (step2 of Supplementary Fig. 2). If the median number of clusters at the midpoint matches the target number of clusters, the midpoint is set to the USB to exclude half of the search area (step3 of Supplementary Fig. 2). This process is repeated until the difference between USB and LSB is less than a tolerance level of 1e–8 (step 4 of Supplementary Fig. 2), ensuring a high level of precision comparable to that obtained with a tolerance value of 1e−13 (Supplementary Fig. 3). When this condition is met, the final midpoint is set as the LB of the resolution parameter corresponding to the target number of clusters (brown box in step 4 of Supplementary Fig. 2).
To quickly find the UB of the resolution parameter for the target number of clusters, we set the previously found LB as the new LSB and performed a new binary search (step5 of Supplementary Fig. 2). In this binary search, unlike when finding the LB, if the median number of clusters of ten cluster labels at the midpoint matches the target number of clusters, the midpoint is set as the LSB (blue box in step6 of Supplementary Fig. 2). This process is repeated until the difference between LSB and USB is less than the tolerance level of 1e–8. Finally, after all repetitions stop, this midpoint is set as the UB of the resolution parameter corresponding to the target number of clusters (green box in step 7 of Supplementary Fig. 2). By combining the searched UB and LB of the resolution parameter, we determine the resolution parameter range corresponding to the target number of clusters (Fig. 3a and Supplementary Fig. 2). This process is repeated for all numbers of clusters of interest to obtain the resolution parameter ranges corresponding to each number of clusters (Fig. 3a and and Supplementary Fig. 3).
Evaluating the clustering consistency within resolution parameter ranges
The binary search method allows us to identify a range of resolution parameters corresponding to a single number of clusters. However, even within a single resolution range, the consistency of cluster labels can vary depending on the resolution parameter. Therefore, to scrutinize clustering consistency within a single resolution parameter range, we uniformly sample eleven resolution parameters within each range (the blue box in Fig. 3a).
To investigate the clustering consistency of uniformly sampled resolution parameters, we perform multiple clusterings for each resolution parameter to obtain a clustering ensemble containing 15 different cluster labels. Next, from each clustering ensemble, we compute the IC for each resolution parameter. If one of the computed ICs is 1, indicating that all clusters are identical, we select the cluster label of the resolution parameter with an IC of 1 as the representative clustering result for the number of clusters under investigation.
However, IC may not be 1 for all sampled resolution parameters. One reason for this may be insufficient iterations of the Leiden clustering algorithm, preventing the clustering results from reaching the maximum modularity of the UMAP graph (Supplementary Fig. 4a). To address this, we incrementally increased the modularity of the UMAP graph for the 15 cluster labels at each resolution parameter by iteratively using the previous cluster labels as new initial partitions for the clustering algorithm (left panel of Fig. 3b and Supplementary Fig. 4a). As the number of iterations increases, the cluster labels could become similar, potentially lowering the IC value (Fig. 3b and Supplementary Fig. 4b). If the IC reaches 1, shows no change after 100 iterations, or does not converge to 1 even after the maximum of 150 iterations, we stop the iterations for that sampled resolution parameter. Such IC trends suggest that further iterations are unlikely to improve the modularity of each cluster label. After iterations at all sampled resolution parameters have stopped, we select 15 cluster labels at the resolution parameter with the lowest IC as the most consistent cluster labels for the corresponding number of clusters (Fig. 3b). To evaluate the clustering consistency of the selected cluster labels more thoroughly, 100 bootstrapping iterations are performed on the cluster labels, creating 100 clustering ensembles. Using these 100 clustering ensembles, 100 IC values are calculated, and their distribution is obtained (Fig. 3c). Then, we select the median of the IC distribution as the representative IC of the corresponding number of clusters. If the median IC is less than 1.005, indicating less than approximately 0.25% membership discrepancy between cluster labels (Supplementary Fig. 1), we considered the clustering at the corresponding number of clusters to be consistent (Fig. 3c).
Reporting summary
Further information on research design is available in the Nature Portfolio Reporting Summary linked to this article.
Supplementary information
Source data
Acknowledgements
This work was supported by the Institute for Basic Science IBS-R029-C3 (to J.K. Kim (Jae Kyoung)) and the National Research Foundation of Korea (NRF) grant funded by the Korean government (MSIT) (No. 2021R1A5A8032895 to M.S.S. and No. RS-2023-00221112 to J.K. Kim (Jong Kyoung)).
Author contributions
H.K. and J.K. Kim (Jae Kyoung) conceived the study. H.K. developed the scICE Julia package. H.K. performed data analysis with assistance from I.P., J.E.P., J.K. Kim (Jong Kyoung), and M.S.S. H.K. and J.K. Kim (Jae Kyoung) discussed the scICE method. H.K. and J.K. Kim (Jae Kyoung) wrote the initial draft. All authors revised the manuscript.
Peer review
Peer review information
Nature Communications thanks Fuyi Li, Zhen Tian, and the other, anonymous, reviewer for their contribution to the peer review of this work. A peer review file is available.
Data availability
The real datasets used in this study are publicly available and can be accessed through the following sources: The mouse GWAT snRNA-seq data41 used in this study are available in the NCBI Sequence Read Archive (SRA) under accession code PRJNA942977. Similarly, the mouse SARS-CoV-2 lung scRNA-seq data42 used in this study are available in NCBI under accession code PRJNA884500. Koh, Kumar, and Trapnell datasets were obtained from the GitHub repository [https://github.com/markrobinsonuzh/scRNAseq_clustering_comparison]27. Zheng datasets28 were obtained from 10 × Genomics datasets [https://www.10xgenomics.com/resources/datasets]. Mouse brain33 and embryo datasets43 used for Fig. 1 and Supplementary Fig. 8, respectively, were obtained from 10 × Genomics datasets [https://www.10xgenomics.com/resources/datasets]. Additionally, the dataset containing ~500,000 cells, used to test the speed and memory performance of scICE, was generated from mouse brain scRNA-seq data44, which is publicly accessible from 10× Genomics datasets [https://www.10xgenomics.com/resources/datasets]. The simulated Tabula Muris and immune cell datasets were generated by fitting the original Tabula Muris dataset29, obtained from the Tabula Muris Project [https://tabula-muris.ds.czbiohub.org/], and the T-cell dataset4, obtained from the Cross-tissue Immune Cell Atlas [https://www.tissueimmunecellatlas.org], using scDesign230 with various sequencing depths. All benchmarking datasets generated and used in this study are available in the GitHub repository [https://github.com/Mathbiomed/scICE] and archived at Zenodo55 [10.5281/zenodo.15113898]. Source data for the figures are provided with this paper. Source data are provided with this paper.
Code availability
The Julia codes for the scICE used in the benchmark study are available on the Mathbio GitHub repository [https://github.com/Mathbiomed/scICE]. All original code developed by the authors within this repository, including scICE, is provided under the MIT license. This repository also contains R scripts used for the benchmark study, which utilize the external R packages multiK and chooseR. These R packages are subject to their respective licenses and were obtained from their distribution channels17,18. The specific version of the scICE code utilized for the analyses in the paper has been assigned a citable DOI through Zenodo56 [10.5281/zenodo.15425320].
Competing interests
The authors declare no competing interests.
Footnotes
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary information
The online version contains supplementary material available at 10.1038/s41467-025-60702-8.
References
- 1.Macosko, E. Z. et al. Highly parallel genome-wide expression profiling of individual. Cells Using Nanoliter Droplets. Cell161, 1202–1214 (2015). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Zilionis, R. et al. Single-cell barcoding and sequencing using droplet microfluidics. Nat. Protoc.12, 44–73 (2017). [DOI] [PubMed] [Google Scholar]
- 3.Klein, A. M. et al. Droplet barcoding for single-cell transcriptomics applied to embryonic stem cells. Cell161, 1187–1201 (2015). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Domínguez Conde, C. et al. Cross-tissue immune cell analysis reveals tissue-specific features in humans. Science376, eabl5197 (2022). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Saunders, A. et al. Molecular diversity and specializations among the cells of the adult mouse brain. Cell174, 1015–1030.e16 (2018). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Wen, S. et al. Spatiotemporal single-cell analysis of gene expression in the mouse suprachiasmatic nucleus. Nat. Neurosci.23, 456–467 (2020). [DOI] [PubMed] [Google Scholar]
- 7.Jin, S. et al. Inference and analysis of cell-cell communication using CellChat. Nat. Commun.12, 1088 (2021). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Van Den Berge, K. et al. Trajectory-based differential expression analysis for single-cell sequencing data. Nat. Commun.11, 1201 (2020). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Kiselev, V. Y., Andrews, T. S. & Hemberg, M. Challenges in unsupervised clustering of single-cell RNA-seq data. Nat. Rev. Genet.20, 273–282 (2019). [DOI] [PubMed] [Google Scholar]
- 10.Andrews, T. S., Kiselev, V. Y., McCarthy, D. & Hemberg, M. Tutorial: guidelines for the computational analysis of single-cell RNA sequencing data. Nat. Protoc.16, 1–9 (2021). [DOI] [PubMed] [Google Scholar]
- 11.Kharchenko, P. V. The triumphs and limitations of computational methods for scRNA-seq. Nat. Methods18, 723–732 (2021). [DOI] [PubMed] [Google Scholar]
- 12.Hao, Y. et al. Dictionary learning for integrative, multimodal and scalable single-cell analysis. Nat. Biotechnol.42, 293–304 (2024). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Blondel, V. D., Guillaume, J.-L., Lambiotte, R. & Lefebvre, E. Fast unfolding of communities in large networks. J. Stat. Mech. 2008, P10008 (2008).
- 14.Traag, V. A., Waltman, L. & Van Eck, N. J. From Louvain to Leiden: guaranteeing well-connected communities. Sci. Rep.9, 5233 (2019). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Van Buren, E. et al. Single-cell RNA sequencing reveals placental response under environmental stress. Nat. Commun.15, 6549 (2024). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.De Zuani, M. et al. Single-cell and spatial transcriptomics analysis of non-small cell lung cancer. Nat. Commun.15, 4388 (2024). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Liu, S., Thennavan, A., Garay, J. P., Marron, J. S. & Perou, C. M. MultiK: an automated tool to determine optimal cluster numbers in single-cell RNA sequencing data. Genome Biol.22, 232 (2021). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Patterson-Cross, R. B., Levine, A. J. & Menon, V. Selecting single cell clustering parameter values using subsampling-based robustness metrics. BMC Bioinforma.22, 39 (2021). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Kiselev, V. Y. et al. SC3: consensus clustering of single-cell RNA-seq data. Nat. Methods14, 483–486 (2017). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Yu, L., Cao, Y., Yang, J. Y. H. & Yang, P. Benchmarking clustering algorithms on estimating the number of cell types from single-cell RNA-sequencing data. Genome Biol.23, 49 (2022). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Cui, Y. et al. Consensus clustering of single-cell RNA-seq data by enhancing network affinity. Brief. Bioinforma.22, bbab236 (2021). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Shahsavari, A., Munteanu, A. & Mohorianu, I. ClustAssess: tools for assessing the robustness of single-cell clustering. Preprint at. 10.1101/2022.01.31.478592 (2022).
- 23.Nie, X. et al. Clustering ensemble in scRNA-seq data analysis: methods, applications and challenges. Comput. Biol. Med.159, 106939 (2023). [DOI] [PubMed] [Google Scholar]
- 24.Tandon, A., Albeshri, A., Thayananthan, V., Alhalabi, W. & Fortunato, S. Fast consensus clustering in complex networks. Phys. Rev. E99, 042301 (2019). [DOI] [PubMed] [Google Scholar]
- 25.Gan, L. & Allen, G. I. Fast and interpretable consensus clustering via minipatch learning. PLoS Comput. Biol.18, e1010577 (2022). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Lee, D., Lee, S. H., Kim, B. J. & Kim, H. Consistency landscape of network communities. Phys. Rev. E103, 052306 (2021). [DOI] [PubMed] [Google Scholar]
- 27.Duò, A., Robinson, M. D. & Soneson, C. A systematic performance evaluation of clustering methods for single-cell RNA-seq data. F1000Res7, 1141 (2020). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.Zheng, G. X. Y. et al. Massively parallel digital transcriptional profiling of single cells. Nat. Commun.8, 14049 (2017). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.The Tabula Muris Consortium et al. Single-cell transcriptomics of 20 mouse organs creates a Tabula Muris. Nature562, 367–372 (2018). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30.Sun, T., Song, D., Li, W. V. & Li, J. J. scDesign2: a transparent simulator that generates high-fidelity single-cell gene expression count data with gene correlations captured. Genome Biol.22, 163 (2021). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31.Gates, A. J., Wood, I. B., Hetrick, W. P. & Ahn, Y.-Y. Element-centric clustering comparison unifies overlaps and hierarchy. Sci. Rep.9, 8574 (2019). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32.Kim, H. et al. scLENS: data-driven signal detection for unbiased scRNA-seq data analysis. Nat. Commun.15, 3575 (2024). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33.10x Genomics. Visium CytAssist Gene Expression Libraries of Post-Xenium Mouse Brain FF (10x Genomics, 2023).
- 34.Lodi, M. K. et al. CHAI: consensus clustering through similarity matrix integration for cell-type identification. Brief. Bioinforma.25, bbae411 (2024). [DOI] [PMC free article] [PubMed]
- 35.Bian, C. et al. scEFSC: accurate single-cell RNA-seq data analysis via ensemble consensus clustering based on multiple feature selections. Comput. Struct. Biotechnol. J.20, 2181–2197 (2022). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 36.Fortunato, S. & Barthélemy, M. Resolution limit in community detection. Proc. Natl. Acad. Sci. USA104, 36–41 (2007). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 37.Lancichinetti, A. & Fortunato, S. Limits of modularity maximization in community detection. Phys. Rev. E84, 066122 (2011). [DOI] [PubMed] [Google Scholar]
- 38.Fortunato, S. & Hric, D. Community detection in networks: a user guide. Phys. Rep.659, 1–44 (2016). [Google Scholar]
- 39.Bolón-Canedo, V., Sánchez-Maroño, N., Alonso-Betanzos, A., Benítez, J. M. & Herrera, F. A review of microarray datasets and applied feature selection methods. Inf. Sci.282, 111–135 (2014). [Google Scholar]
- 40.Dougherty, E. R. Small sample issues for microarray-based classification. Comp. Funct. Genom.2, 28–34 (2001). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 41.Choi, C. et al. TM4SF19-mediated control of lysosomal activity in macrophages contributes to obesity-induced inflammation and metabolic dysfunction. Nat. Commun.15, 2779 (2024). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 42.Choi, S. et al. A longitudinal molecular and cellular lung atlas of lethal SARS-CoV-2 infection in K18-hACE2 transgenic mice. eBioMedicine99, 104932 (2024). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 43.10x Genomics. Mouse Embryo, 11 mm Capture Area (FFPE). https://www.10xgenomics.com/datasets/mouse-embryo-11-mm-capture-area-ffpe-2-standard (10x Genomics, 2022).
- 44.10x Genomics. 1.3 Million Brain Cells from E18 Mice. https://www.10xgenomics.com/datasets/1-3-million-brain-cells-from-e-18-mice-2-standard-1-3-0 (10x Genomics, 2017).
- 45.McInnes, L., Healy, J., Saul, N. & Großberger, L. UMAP: uniform manifold approximation and projection. JOSS3, 861 (2018). [Google Scholar]
- 46.Ertöz, L., Steinbach, M. & Kumar, V. Finding clusters of different sizes, shapes, and densities in noisy, high dimensional data. In Proceedings of the 2003 SIAM International Conference on Data Mining 47–58 10.1137/1.9781611972733.5 (Society for Industrial and Applied Mathematics, 2003).
- 47.Zhu, X. et al. Single-cell clustering based on shared nearest neighbor and graph partitioning. Interdiscip. Sci. Comput. Life Sci.12, 117–130 (2020). [DOI] [PubMed] [Google Scholar]
- 48.Everitt, B. S. The Analysis of Contingency Tables10.1201/b15072 (Chapman and Hall/CRC, 1992).
- 49.Lotfollahi, M. et al. Mapping single-cell data to reference atlases by transfer learning. Nat. Biotechnol.40, 121–130 (2022). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 50.Hu, J. et al. Iterative transfer learning with neural network for clustering and cell type classification in single-cell RNA-seq analysis. Nat. Mach. Intell.2, 607–618 (2020). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 51.Grabski, I. N., Street, K. & Irizarry, R. A. Significance analysis for clustering with single-cell RNA-sequencing data. Nat. Methods20, 1196–1202 (2023). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 52.Fang, J. et al. Clustering Deviation Index (CDI): a robust and accurate internal measure for evaluating scRNA-seq data clustering. Genome Biol.23, 269 (2022). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 53.Jeong, W., Lee, D., Kim, H. & Lee, S. H. Kaleidoscopic reorganization of network communities across different scales. Phys. Rev. E111, 014312 (2025). [DOI] [PubMed] [Google Scholar]
- 54.Traag, V. A., Van Dooren, P. & Nesterov, Y. Narrow scope for resolution-limit-free community detection. Phys. Rev. E84, 016114 (2011). [DOI] [PubMed] [Google Scholar]
- 55.Kim, H. Data supporting the publication: scICE - Enhancing clustering reliability and efficiency of single-cell RNA sequencing data with multi-cluster label consistency evaluation. Zenodo. 10.5281/zenodo.15113898 (2025). [DOI] [PMC free article] [PubMed]
- 56.Kim, H. scICE: an efficient clustering stability estimator for scRNA-seq data. (0.1.0). Zenodo. 10.5281/zenodo.15425320 (2025).
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Data Availability Statement
The real datasets used in this study are publicly available and can be accessed through the following sources: The mouse GWAT snRNA-seq data41 used in this study are available in the NCBI Sequence Read Archive (SRA) under accession code PRJNA942977. Similarly, the mouse SARS-CoV-2 lung scRNA-seq data42 used in this study are available in NCBI under accession code PRJNA884500. Koh, Kumar, and Trapnell datasets were obtained from the GitHub repository [https://github.com/markrobinsonuzh/scRNAseq_clustering_comparison]27. Zheng datasets28 were obtained from 10 × Genomics datasets [https://www.10xgenomics.com/resources/datasets]. Mouse brain33 and embryo datasets43 used for Fig. 1 and Supplementary Fig. 8, respectively, were obtained from 10 × Genomics datasets [https://www.10xgenomics.com/resources/datasets]. Additionally, the dataset containing ~500,000 cells, used to test the speed and memory performance of scICE, was generated from mouse brain scRNA-seq data44, which is publicly accessible from 10× Genomics datasets [https://www.10xgenomics.com/resources/datasets]. The simulated Tabula Muris and immune cell datasets were generated by fitting the original Tabula Muris dataset29, obtained from the Tabula Muris Project [https://tabula-muris.ds.czbiohub.org/], and the T-cell dataset4, obtained from the Cross-tissue Immune Cell Atlas [https://www.tissueimmunecellatlas.org], using scDesign230 with various sequencing depths. All benchmarking datasets generated and used in this study are available in the GitHub repository [https://github.com/Mathbiomed/scICE] and archived at Zenodo55 [10.5281/zenodo.15113898]. Source data for the figures are provided with this paper. Source data are provided with this paper.
The Julia codes for the scICE used in the benchmark study are available on the Mathbio GitHub repository [https://github.com/Mathbiomed/scICE]. All original code developed by the authors within this repository, including scICE, is provided under the MIT license. This repository also contains R scripts used for the benchmark study, which utilize the external R packages multiK and chooseR. These R packages are subject to their respective licenses and were obtained from their distribution channels17,18. The specific version of the scICE code utilized for the analyses in the paper has been assigned a citable DOI through Zenodo56 [10.5281/zenodo.15425320].






