Data-driven identification of cell subtypes for single-cell transcriptomic data with Subtypist

Abstract

Identification of new cell subtypes in single-cell RNA sequencing (scRNA-seq) data is critically crucial in accurately understanding the mechanisms of disease occurrence and development, which provides unprecedented insights into the development of therapeutic strategies, however, despite the abundance of existing methods, most heavily rely on the reference with the fixed cell labels, which fail to uncover new cell subtypes. Herein, we propose Subtypist, a fully data-driven multiscale iterative algorithm for cell subtype identification in scRNA-seq data without relying on external prior references. The benchmarking evaluation on both simulated and real datasets demonstrates its superior performance over other existing methods, and Subtypist was further applied to three disease scenarios, uncovering novel biologically meaningful cell subtypes and revealing their in-depth cell–cell communicative mechanisms underlying hepatocellular carcinoma, esophageal squamous cell carcinoma, and myocardial infarction. In summary, Subtypist enables the data-driven and reproducible identification of cell subtypes, providing an invaluable tool for accurately characterizing the cellular and molecular heterogeneity underlying disease pathogenesis.

Introduction

Cell subtypes are used to reveal the high heterogeneity within cell types and aim to achieve fine-grained cell identity characterization. Precise identification of cell subtypes is essential in single-cell RNA sequencing (scRNA-seq), as it facilitates a deeper understanding of the cellular and molecular mechanisms underlying disease pathogenesis and heterogeneity [1–6]. It also provides valuable support for the discovery of drug targets and the development of new therapeutics, particularly in cutting-edge areas such as immunotherapy and stem cell therapy [7]. For instance, subtype identification in cancer research has identified a specific dendritic cell subtype regulating anti-tumor immunity [8, 9] and a distinct NK cell subtype mediating therapeutic responses [10], highlighting the critical role of fine-grained subtype identification in uncovering disease mechanisms and therapeutic targets.

Currently, numerous automated identification approaches are available for cell type annotation [11–16], which significantly accelerate the annotation process and provide accurate identification of major cell types. These methods are built on predefined cell marker databases or reference datasets and assign cell type labels by matching known markers or reference expression profiles. For example, CellAssign [17] uses predefined marker genes to infer cell types through probabilistic models, while scDeepSort [13] trains models on high-quality annotated data to automate identification. CellTypist [11] classifies cells based on high-quality immune reference atlases, covering major immune cell types and 43 immune subtypes. However, the generalizability of these methods is often constrained by fixed, limited subtype labels and the poor reproducibility of reference data [18, 19]. For example, the intersection of gene sets across datasets frequently results in information loss, significantly reducing the number of available features and compromising performance. Consequently, these methods are typically restricted to identifying canonical subtypes with well-established marker genes (e.g. CD4⁺ T cells or conventional macrophages), and they lack the capability for the de novo discovery of novel or rare subtypes. Another widely used strategy for subtype identification is manual annotation, the standard pipeline of which typically involves unsupervised clustering, followed by assigning cell identities using cluster-specific marker genes [20]. The process is inherently time-consuming and subjective, often leading to low reproducibility. Alternatively, annotation workflows frequently rely on community detection algorithms. These methods require determining an optimal clustering resolution [21], a challenging task that leads to over-clustering and under-clustering [22–24]. These resulting clusters represent purely data-driven statistical partitions that often lack biological coherence. Thus, there is an urgent need for a data-driven and reproducible solution specifically designed for subtype discovery.

To achieve data-driven and robust subtype identification, we developed Subtypist, which combines a multiscale analytical strategy with a specificity-based scoring framework. Given a predefined cell type, our approach introduces a fully data-driven framework designed to uncover biologically meaningful subtypes, operating independently of external reference datasets or prior knowledge of canonical marker genes. We developed a multiscale analysis strategy to resolve the ambiguity inherent in resolution selection. By automatically determining the optimal clustering granularity, our model eliminates the dependency on manual parameter tuning and guarantees robust annotation, while also facilitating the convenient identification of cell subtypes through recommended marker genes. We present a data-driven, reproducible and automated tool designed specifically for novel cell subtype identification without relying on external prior references in the scRNA-seq analysis pipeline, benchmark our tool on simulated and real scRNA-seq datasets to prove the performance, and finally demonstrate the advantages within the human hepatocellular carcinoma (HCC) dataset [25], human esophageal squamous-cell carcinoma ecosystem dataset [26], and human myocardial infarction dataset [27].

Results

Overview of Subtypist

To address the challenges of identifying novel cell subtypes, specifically selecting the optimal clustering partitioning, we developed Subtypist (Fig. 1). To achieve fine-grained characterization of a predefined cellular identity, Subtypist refines initial clustering results using a multiscale analysis strategy that seamlessly integrates into the Seurat [28] analysis pipeline. The method emulates the manual annotation workflow, employing a specificity-driven iterative optimization to bridge computational clustering structures with biologically meaningful cell subtype annotations. First, the biological specificity of each cell subtype is quantified using specificity scores. A specificity validation step then evaluates the reliability of the current cluster partition and determines whether iteration should terminate; otherwise, a similarity-driven merging procedure is applied to further refine the partition. Finally, Subtypist produces cell subtype annotations across multiple resolutions, providing alternative views of the scRNA-seq data at different granularities, and identifies a representative resolution and its corresponding subtype partition marked with phenotypic molecules based on a biological specificity score. Here, phenotypic molecules correspond to the top-ranked genes identified by Subtypist and are used as potential marker genes for characterizing and distinguishing subtype identities.

Figure 1.

Overview of Subtypist. Subtypist utilizes a specificity-driven iterative optimization framework, emulating manual annotation, to bridge computational clustering structures with biological cell subtype annotations. Applied independently to clustering results from multiple resolutions, its core iterative loop comprises: (step A) specificity scores to quantify the biological rationale of subtypes; (step B) specificity validation to assess partition stability and determine iteration termination; and (step C) similarity-driven merging to optimize clustering if termination criteria are not met. Finally, Subtypist evaluates multi-resolution results via specificity scores to recommend the optimum and generate subtype annotations supported by phenotypic molecules.

Subtypist is capable of incorporating both positively and negatively regulated marker genes, enabling more accurate discrimination of closely related cell subtypes. In addition, it implements an accelerated mode designed to enhance computational efficiency while preserving analytical accuracy. To permit the evaluation of the specificity score for clusters that have already been identified, we developed a version adaptable to any given set of clusters. We highlight the critical importance of cell subtype annotation in scRNA-seq data analysis, describe the mathematical details of our approach, benchmark our approach on simulated and real data, and validate its performance in three disease contexts.

Evaluation on simulated datasets

To further validate the performance and robustness of Subtypist, we systematically benchmark it on a series of simulated datasets. We constructed a benchmark dataset comprising 40 simulated datasets using the ESCO [29] simulator, an advanced simulator that integrates features from existing frameworks and employs a copula model [30] to introduce controllable gene co-expression. This benchmark dataset systematically covers key characteristics, including the total number of cells, the predefined number of clusters, and the subtype-proportion structure, including both balanced and imbalanced cell subtypes (Supplementary Table S1). We evaluated both the default and elbow termination modes of Subtypist on each dataset and compared their performance with four widely used clustering approaches. The comparison tools included Louvain and Leiden algorithms [31] implemented in Seurat version 4 and version 5 as well as scSHC [23] and SC3 [32] (Fig. 2a–c). To ensure fair comparison across methods, all methods were evaluated under predefined method-specific parameter grids (Supplementary Fig. S1). We report the best-achieved performance under each method’s grid using the primary clustering metric.

Figure 2.

Subtypist results on simulated datasets. (a) Best ARI achieved by each clustering method across the predefined parameter grid on simulated datasets with varying cell numbers. (b) Best ARI achieved by each clustering method across the predefined parameter grid on simulated datasets with varying subtype numbers. (c) Best ARI achieved within the predefined parameter grid across simulated datasets with varying subtype balance conditions. (d) UMAP visualization of scRNA-seq datasets showing ground-truth annotations and tool-predicted labels from representative simulated data. The dataset includes 4500 cells across seven uneven clusters and was analyzed using multiple clustering approaches, including Subtypist, SeuratV4-Louvain, SeuratV4-Leiden, SeuratV5-Louvain, SeuratV5-Leiden, SC3, and scSHC. (e) UMAP visualization of a simulated scRNA-seq dataset containing 3000 cells across four uneven clusters, showing ground-truth annotations, Subtypist predictions under different resolution settings, and SeuratV4-Louvain predictions at resolution 1.2.

Across the 40 simulated datasets, Subtypist-default achieved strong and stable performance with a median adjusted Rand index (ARI) of 0.850 and a median Normalized Mutual Information (NMI) of 0.812 (Fig. 2a–c, Supplementary Figs S2, S3, S4, S5, S6, and S7 and Supplementary Table S2). Overall, Subtypist-default consistently outperformed graph-based clustering methods, achieving higher median ARI than Seurat-based Louvain and Leiden across both Seurat v4 (0.838 and 0.839, respectively) and Seurat v5 (0.839 and 0.844, respectively), and substantially exceeding SC3, which showed markedly lower accuracy (median ARI = 0.386). Among competing methods, scSHC achieved the highest median ARI and NMI on simulated datasets (0.861 and 0.841, respectively), but its performance was less consistent across real datasets. Subtypist achieved comparable performance to scSHC in simulated datasets, as reflected by similar median weighted-F1 scores.

In addition, the accelerated implementation of Subtypist (Subtypist-accelerated) preserved most clustering accuracy while substantially reducing runtime (Supplementary Fig. S7), further supporting its suitability for routine applications. The elbow-based mode showed reduced effectiveness on simulated datasets but demonstrated superior performance on real-world datasets, indicating complementary applicability under different data structures. Overall, these results demonstrate the robustness and generalizability of Subtypist for cell subtype identification across diverse settings.

In practical applications, exhaustive grid search is often not feasible; therefore, we further compared the performance of all methods under their default settings as representative cases. In the simulated dataset containing 4500 cells and seven uneven clusters (Fig. 2d), Subtypist accurately identified all seven clusters, achieving the highest accuracy (ARI = 0.685) and recommending corresponding phenotypic markers. In comparison, SeuratV4-Louvain and SeuratV4-Leiden identified eight clusters with lower accuracies (ARI = 0.598 and 0.563, respectively), while scSHC correctly identified seven clusters with an ARI of 0.665. In the simulated dataset containing 3000 cells and four uneven clusters, SeuratV4-Louvain and SeuratV4-Leiden showed differences in the number of clusters identified at different resolutions, while Subtypist consistently optimized to four clusters and maintained high accuracy at all resolutions (Fig. 2e). At a resolution of 1.2, the initial clustering yielded five clusters, but Subtypist effectively corrected the over-clustering issue, ultimately merging to the correct number of four clusters.

Subtypist demonstrates high performance in subtype annotation tasks

We further evaluated the annotation performance of Subtypist on 18 published scRNA-seq datasets from human and mouse, covering multiple tissues including liver [33–35], pancreas [36], intestine [37], and tonsil [38] (Fig. 3a and Supplementary Fig. S8). These datasets encompass a broad spectrum of major cell types, including epithelial, endothelial, myeloid, immune, and stromal cells, and further provide fine-grained subtype annotations, enabling systematic evaluation of the subtype resolution performance of Subtypist across diverse biological contexts. To evaluate subtype interpretation based on phenotypic molecules, we manually curated subtype-specific marker genes from the original literature (Supplementary Table S3) to construct an independent reference panel, which was used to assess whether the predicted phenotypic molecules are supported by external marker evidence.

Figure 3.

Subtypist demonstrates high performance in subtype annotation tasks. (a) UMAP visualization of scRNA-seq datasets, showing ground-truth annotation labels. (b) ARI comparison of different clustering tools, including Subtypist, Seurat-Louvain, Seurat-Leiden, SC3, and scSHC, across real datasets. (c) ARI between predicted clusters and reference cell subtype annotations was used to evaluate clustering performance. Boxplots summarize ARI values across resolution settings for each dataset, with points indicating the best-performing resolution. Subtypist was evaluated under two convergence modes.

To evaluate the accuracy of annotation for this complex multi-classification problem, we used the following metrics: (i) ARI, a statistical measure used to assess the similarity between two clustering results; (ii) NMI, which quantifies the shared information between predicted and reference partitions; (iii) macro-F1 score, representing the unweighted average of the F1-score across all phenotypic molecules; (iv) weighted F1-score, which calculates the F1-score while accounting for the proportion of each subtype in the dataset, and (v) marker concordance, an auxiliary interpretability metric defined as the proportion of predicted clusters for which the Subtypist-inferred phenotypic molecules show at least one overlap with the corresponding curated subtype marker set. Together, these metrics provide a comprehensive evaluation of clustering and annotation performance. For all metrics, higher values indicate greater concordance with the reference annotations, with values closer to 1 reflecting better overall performance.

We evaluated Subtypist under both elbow and default termination criteria across real-world datasets. The predicted phenotypic molecule annotations were compared with ground-truth marker genes, and clustering accuracy was further benchmarked against Seurat-Louvain, Seurat-Leiden, SC3, and scSHC. Subtypist achieved superior clustering performance to any other clustering method across multiple evaluation metrics (Fig. 3b, Supplementary Figs S9, S10, and S11 and Supplementary Table S4). In comparison, other annotation tools, including CellTypist [11], scType [39], and scGPT [40] exhibit reduced performance on the test datasets, likely due to incomplete training datasets or limited reference markers (Supplementary Figs S12, S13, and S14), resulting in lower overall accuracy and multiple misassigned or unannotated subtypes.

Subtypist with both elbow and default termination criteria demonstrated robust performance across datasets derived from diverse tissue origins (Fig. 3c and Supplementary Fig. S15). Subtypist-elbow achieved an average ARI of 0.599, an average NMI of 0.657, an average macro-F1 score of 0.605, and an average weighted F1 score of 0.714, while Subtypist-default achieved 0.571, 0.644, 0.710, and 0.670, respectively.

At the dataset level, Subtypist resolved cell subtypes across datasets of varying complexity. It achieved an ARI of 0.891 on dataset 10 (human liver-transplant myeloid cells with five reference subtypes) and performed well on dataset 14 (mouse NAFLD liver lymphocytes with 12 subtypes), yielding an ARI of 0.873, an NMI of 0.831, and a weighted F1 score of 0.838. It also identified rare cell populations; in a pancreatic cancer dataset (dataset 4), a minor stromal subtype was detected (ARI = 0.798). Marker concordance analysis revealed agreement between inferred phenotypic molecules and validated marker genes (Supplementary Fig. S15e).

Subtypist-elbow and Subtypist-default showed largely comparable performance across most datasets, with dataset-dependent differences under specific settings. Notably, Subtypist-elbow performed better in datasets characterized by continuous or transitional cellular states, particularly datasets 16–18 derived from human tonsil tissue. In dataset 16, clustering accuracy improved markedly, with ARI increasing from 0.442 to 0.702 (Supplementary Fig. S16). Both methods showed comparable performance in datasets containing rare subtypes (cell proportion < 1%). In other datasets, Subtypist-default showed slightly higher performance in some cases, such as dataset 11, where Subtypist-default achieved higher ARI than Subtypist-elbow. Overall, the two methods showed consistent performance across diverse dataset settings.

In the human liver macrophage dataset (dataset 10), Subtypist identified seven subtypes compared to the five reported in the original study. Five of these subtypes demonstrated high concordance with the original annotations. Beyond these consistent groups, Subtypist revealed additional heterogeneity, identifying Cluster-6, characterized by high expression of CD5L, CETP, and SLC40A1, and Cluster-3, which exhibited elevated levels of VCAN, S100A12, and FCN1 (Supplementary Fig. S19). CETP is recognized as a Kupffer cell marker that connects hepatic inflammation to atherogenic dyslipidemia [41], and Kupffer cells with high expression of CETP and SLC40A1 may potentially influence immune response, liver repair, transplant rejection, and chronic liver injury after transplantation by regulating lipid metabolism and iron balance [8]. Notably, while the cells in Subtypist’s subtype Cluster-0 and Cluster-6 were collectively identified as “IFI27⁺ Kupffer cells” in the original study, our tool effectively separated them into two distinct subtypes. Similarly, although Cluster-3 shares the marker VCAN with the monocyte-derived macrophage (MDM) population (Cluster-1), it exhibits significantly higher expression of monocyte-associated marker genes, such as S100A12 and FCN1. Given the high expression of genes such as S100A12 and FCN1, which are closely associated with monocytes [42, 43], this suggests that Cluster-3 may represent early-stage differentiated MDM cells. The results showed that this method can not only accurately identify the main subtypes but also reveal subtypes with potential biological significance.

Subtypist verified a spatially resolved macrophage subtype involved in the tumor immune barrier in HCC

We applied Subtypist to a publicly available scRNA-seq dataset of human HCC (Fig. 4a). Using a recommended resolution of 0.3, Subtypist partitioned macrophages into seven distinct subtypes characterized by specific markers (Fig. 4b and c, Supplementary Fig. S20). Notably, Subtypist identified the Secreted Phosphoprotein 1-positive macrophage (SPP1), a subtype enriched in tumor tissues with significant differences in expression levels between tumor and normal tissues (P = .01; Fig. 4d and e and Supplementary Fig. S20b). The enrichment analysis showed that this macrophage subtype was significantly enriched for the NABA MATRISOME ASSOCIATED gene set and pathways involved in extracellular matrix organization (Fig. 4f and Supplementary Table S5), suggesting a pivotal role in tissue remodeling and the modulation of the tumor microenvironment (TME). These findings indicate that our tool has identified a macrophage subtype with potential biological significance in tumors, likely contributing to the tumor microenvironment and influencing immune response dynamics.

Figure 4.

Subtypist verified a spatially resolved macrophage subtype involved in the tumor immune barrier in HCC. (a) Scheme of the HCC datasets. (b) UMAP visualization of myeloid cells from scRNA-seq, illustrating phenotypic molecule subtypes annotated by Subtypist. (c) Vlnplot depicting the highly expressed genes for the seven subtypes. The expression level is the normalized data (log1p value). (d) Proportion of seven myeloid subtypes in normal and HCC tumor samples. (e) Box plot of SPP1 expression levels in normal and HCC tumor samples. The two-tailed Wilcoxon rank-sum test was employed to compare the continuous gene expression levels between the two groups. (f) Enriched pathways and GO biological processes based on the top 100 significantly highly expressed genes for SPP1⁺ macrophage subtype. Statistical significance is expressed as , and dot size reflects the enrichment degree. (g) Correlation analysis of gene expression profiles between the SPP1⁺ macrophages reported in the original study and the SPP1⁺ macrophage subtypes identified by Subtypist in our study, along with the Spearman correlation coefficient (ρ) and the P-value derived from the two-tailed Spearman’s rank correlation test. (h) H&E staining of the tumor tissue of an ICB non-responder of ST spots (left) and definition of cell types of each spot from the original research (right). (i) Mapping of SPP1⁺ macrophages identified by Subtypist and original studies onto spatial transcriptome data. (j) Distances to HCC spots were compared between SPP1⁺ macrophages and other subtypes identified by Subtypist. Each dot represents a single cell, with horizontal jitter applied to reduce overlap. Error bars represent mean ± standard error, and statistical significance was determined by a two-tailed Wilcoxon rank-sum test. (k) Spatial mapping of SPP1⁺ macrophages from Subtypist and the original study on ST data, with SPP1⁺ spots highlighted (left). Boxplots show the comparison of their distances to SPP1⁺ ST spots, with statistical significance determined by a two-tailed Wilcoxon rank-sum test (right). (l) Spatial mapping of SPP1⁺ macrophages from Subtypist and the original study on ST data (left). Boxplots show the comparison of their distances to HCC spots, with statistical significance determined by a two-tailed Wilcoxon rank-sum test (right).

Critically, the SPP1⁺ macrophages identified by the Subtypist exhibited an exceptional correlation (r = 0.97) with the subtype annotated in the original study (Fig. 4g). This highly consistent mapping validates Subtypist’s ability to recover biologically critical subtypes. In the original research, these SPP1⁺ macrophages were shown to interact with cancer-associated fibroblasts (CAF) to form a tumor immune barrier, which significantly impairs immunotherapy efficacy.

To evaluate whether Subtypist provides a more biologically refined classification compared to the original study, we examined the spatial distribution of the identified SPP1⁺ macrophages. Specifically, we reconstructed the spatial coordinates of the SPP1⁺ macrophages from the original study and the tool separately onto spatial transcriptomics using CellTrek [44] (Fig. 4h and i and Supplementary Fig. S21). After mapping the spatial coordinates, we calculated the Euclidean distances from the SPP1⁺ macrophages to the HCC tumor regions in the spatial transcriptomics slices. The SPP1⁺ macrophages were significantly more proximal to the tumor boundaries than other subtypes (P = 3.7 × 10⁻³, Fig. 4j), suggesting their potential role in forming an immune barrier. Subsequently, while both methods identified SPP1⁺ macrophages, the subtype defined by Subtypist exhibited superior spatial specificity. Compared to the original study, the SPP1⁺ macrophages identified by Subtypist were significantly closer to the SPP1⁺ macrophage/CAF spots (P = 1.0 × 10⁻⁵, Fig. 4k). Additionally, these cells demonstrated a potential trend of closer proximity to HCC tumor spots, (P = .003, Fig. 4l). These spatial metrics suggest that Subtypist provides a more functionally refined classification by effectively isolating the specific cell states involved in the tumor immune barrier. This enhanced spatial specificity underscores the potential relevance of our tool in dissecting the complex dynamics of the tumor microenvironment.

Subtypist identifies HES1+ epithelial tumor cells associated with poor prognosis in ESCC

We applied Subtypist to scRNA-seq datasets profiling the human esophageal squamous cell carcinoma ecosystem (ESCC) (Fig. 5a). Subtypist identified 11 subtypes at the recommended resolution of 0.5 (Fig. 5b and c). Notably, the hairy and enhancer of split-1-positive (HES1) epithelial cell subtype exhibited a significant increase in proportion in the second and third stages of cancer compared to the first stage (Fig. 5d and Supplementary Fig. S22). HES1 is a key regulator of the Notch signaling pathway, balancing cell differentiation and stem cell maintenance, while also contributing to proliferation, apoptosis, and cancer progression. Numerous studies have demonstrated that HES1 is closely associated with tumor invasion and poor prognosis across various cancer types [45]. For instance, in colorectal cancer, HES1 promotes metastasis by inducing epithelial-mesenchymal transition (EMT) [46]. Similarly, in renal cell carcinoma, HES1 enhances tumor progression by modulating NOTCH1 signaling and interacting with miR-138-2 [47]. Importantly, HES1 has also been identified as an independent prognostic marker in ESCC, where its elevated expression is strongly linked to disease progression and adverse clinical outcomes [48].

Figure 5.

Subtypist identifies HES1+ epithelial tumor cells associated with poor prognosis in ESCC. (a) Scheme of the ESCC datasets. (b) UMAP visualization of epithelial cells from scRNA-seq, illustrating phenotypic molecule subtypes annotated by Subtypist. (c) Expression of markers for HES1⁺ epithelial cell subtype. (d) Proportion of 11 epithelial cell subtypes in tumor stages I, II, and III. (e) Volcano plot of marker genes for the HES1⁺ epithelial cell subtype. (f) Enriched GO biological processes (BP) and molecular functions (MF) based on the top 100 most significantly upregulated genes in the HES1⁺ epithelial cells subtype, with statistical significance adjusted by FDR. (g) Schematic overview of TCGA bulk RNA-seq data (N = 88 samples). (h) Calculation of the HES1⁺ Epi module score based on the top 100 significantly highly expressed genes using Seurat. Samples were stratified into high (top 25%) and low (bottom 75%) score groups. (i) Kaplan–Meier curves of the TCGA bulk RNA-seq cohort comparing high and low HES1⁺ Epi module score groups, with statistical significance determined by the two-tailed log-rank test. (j) Characterization of bulk RNA-seq data using cell-type deconvolution, stratified by the proportion of HES1⁺ epithelial cells into high (top 25%) and low (bottom 75%) proportion groups. (k) Kaplan–Meier survival curves of the TCGA bulk RNA-seq cohort comparing patients with high and low proportions of HES1⁺ epithelial cells, with statistical significance determined by the two-tailed log-rank test, as estimated by deconvolution analysis.

Next, to further characterize the molecular features of the HES1⁺ epithelial cell subtype, we conducted a differential expression analysis, which is visualized in a volcano plot (Fig. 5e). The analysis revealed significant upregulation of nuclear receptor subfamily 4 group A member 1 (NR4A1) and AP-1 Transcription Factor Subunit (FOSB). Previous studies have shown that NR4A1 upregulation is implicated in T cell dysfunction and immune evasion [49], while elevated FOSB expression enhances cellular proliferation and differentiation [50]. Enrichment analysis indicated that upregulated genes are associated with epithelial proliferation and response to xenobiotic stimulus, suggesting this subtype may promote tumor growth and stress adaptation (Fig. 5f and Supplementary Table S6). These findings indicate that this subtype may drive tumor progression by promoting cancer cell growth and adapting to metabolic or therapeutic stress, further associating it with a poor prognosis in ESCC.

To evaluate the clinical relevance, we obtained an independent cohort of 88 human ESCC specimens from the TCGA database, generated using bulk RNA-seq technology (Fig. 5g). To evaluate the clinical significance of the HES1⁺ epithelial subtype identified by our tool, we quantified the expression levels of bulk RNA-seq specimens and then stratified the samples into high- and low-expression groups (Fig. 5h). Survival analysis was then conducted to assess the association between this classification and patient prognosis, revealing a significant association with poorer outcomes (P = .032, Fig. 5i). We further inferred the HES1⁺ epithelial proportion in bulk RNA-seq data through a deconvolution algorithm. Specimens were then stratified into high- and low-proportion groups based on the quartiles of HES1⁺ Epi proportion (Fig. 5j). Survival analysis demonstrated a significant association between a higher HES1⁺ Epi proportion and poorer prognosis, with a correlation of 0.044 (Fig. 5k), confirming the association between this subtype and adverse clinical outcomes.

Subtypist uncovers a pro-remodeling macrophage subtype spatially colocalized with cardiomyocytes in the ischemic zone of myocardial infarction

We employed the Subtypist tool to conduct a comprehensive analysis of human myocardial infarction (MI) datasets, including single-nucleus RNA sequencing (snRNA-seq) data and spatial transcriptomics data derived from necrotic regions (including ischemic and border zones), remote areas, and fibrotic tissue samples (Fig. 6a). We identified and characterized eight distinct subtypes in snRNA-seq data (Fig. 6b and c and Supplementary Fig. S22). Our analysis revealed a marked enrichment of subtype Fibronectin 1 (FN1) positive macrophages in the ischemic region, with its relative abundance progressively increasing as the distance to the ischemic region decreases (Fig. 6d and Supplementary Fig. S23a). We first assessed the expression of myeloid cell marker genes in spatial transcriptomics data. We then performed deconvolution analysis on sections from different regions to characterize the cellular composition of each spot (Fig. 6e and Supplementary Figs S24, S25). FN1⁺ macrophages and major macrophages exhibit spatial co-occurrence, indicating a close spatial association. Consistently, the proportional distribution of FN1⁺ macrophages across different regions in ST data mirrored the regional variation observed in the snRNA-seq.

Figure 6.

Subtypist uncovers a pro-remodeling macrophage subtype spatially colocalized with cardiomyocytes in the ischemic zone of myocardial infarction. (a) Scheme of the myocardial infarction datasets. (b) UMAP visualization of myeloid cells from snRNA-seq, illustrating phenotypic molecule subtypes annotated by Subtypist. (c) Heatmap depicting the highly expressed genes for the eight subtypes. The expression level is the normalized data (log1p value). (d) Proportion of eight myeloid cell subtypes for control, ischemic zone, border zone, remote zone, and fibrotic tissue samples. (e) Characterization of spatial transcriptomics data based on the macrophage module score, derived from the top 100 significantly upregulated genes using Seurat (top). The proportion of FN1⁺ macrophages in each region is estimated by deconvolution analysis (bottom). (f) Ligand-receptor pairs mediating cell–cell communication involving FN1⁺ macrophages. (g) Top ligand-receptor pairs mediating cell–cell communication across all cell types and FN1⁺ macrophage subtypes. (h) Distribution and directional flow of the IGF interaction pathway, with the background heatmap representing the FN1⁺ macrophages deconvolution proportion. (i) Regional comparison of IGF signaling activity, based on aggregated pathway scores from spots enriched for FN1⁺ macrophages (deconvoluted proportion > 0.4) across control, ischemic, border, remote, and fibrotic tissue regions.

The FN1⁺ macrophages exhibit an elevated expression of Stabilin-1 (STAB1) and Myosin heavy chain, α isoform (MYH6), alongside its characteristic FN1 expression. STAB1 is implicated in tissue remodeling and has been shown to facilitate immune modulation and tissue regeneration during the remodeling phase of myocardial infarction [51]. FN1 encodes fibronectin, a multifunctional glycoprotein that plays a pivotal role in cell adhesion, migration, and extracellular matrix remodeling, which is essential for wound healing and tissue remodeling. Gene ontology (GO) enrichment analysis of this subset revealed significant associations with biological processes related to muscle cell development, muscle tissue morphogenesis, and muscle system processes, supporting its involvement in cardiac repair (Supplementary Fig. S23b).

Subsequently, we performed cell–cell communication analysis using CellChat [52] to elucidate the signaling pathways through which FN1⁺ macrophages contribute to tissue repair. In the snRNA-seq data, cell–cell communication analysis revealed that FN1⁺ macrophages interact with cardiomyocytes via the GAS pathway and engage in midkine and insulin-like growth factor (IGF) signaling with both fibroblasts and adipocytes (Fig. 6f and g). Notably, the insulin-like growth factor 1 (IGF1)–IGF1 receptor (IGF1R) interaction plays a crucial role in fibroblast–FN1⁺ macrophage communication. This signaling pathway is well-documented for its cardioprotective effects following MI [53, 54], facilitating tissue remodeling through inflammatory modulation and the promotion of an M2-like macrophage phenotype [55]. Similarly, the Growth arrest-specific 6 (GAS6)—Proto-oncogene tyrosine-protein kinase MER (MERTK) signaling axis has been shown to play a crucial role in anti-inflammatory responses and efferocytosis, facilitating apoptotic cell clearance and preserving tissue homeostasis, as evidenced in studies on hepatic ischemia–reperfusion injury (HIRI) [56]. This suggests that FN1⁺ macrophages may represent a macrophage subset influenced by both IGF1 and GAS6 signaling, actively contributing to cardiac tissue remodeling and immune resolution.

To further map the spatial distribution and directionality of IGF signaling, we applied COMMOT [57] to spatial transcriptomic slices (Fig. 6h). IGF signaling colocalized with FN1⁺ macrophages, with IGF signals predominantly enriched in regions with a high proportion of FN1⁺ macrophages. Furthermore, we quantified IGF signaling strength across different regions and found that its distribution closely mirrored the proportional changes in FN1⁺ macrophage abundance (Fig. 6i). These observations suggest a strong spatial association between IGF signaling and FN1⁺ macrophages, reinforcing the notion that FN1⁺ macrophages may be a key recipient of IGF-mediated signaling in the context of tissue repair and immune modulation.

Discussion

Cell subtype identification is a critical step in scRNA-seq analysis. However, existing annotation tools are still insufficient to meet the demands of analysis, with the whole process remaining heavily dependent on manual annotation. Here, we introduce Subtypist, a data-driven tool designed to identify new cell subtypes characterized by phenotypic molecules, providing a systematic and efficient approach to improve annotation accuracy while enabling quantitative evaluation through specificity scores. Furthermore, validation on a HCC dataset showed that our tool achieves more precise segmentation of continuous cell states while preserving biological relevance. Its application to human ESCC and myocardial infarction (MI) datasets successfully identified biologically meaningful cell subtypes, further highlighting its applicability and potential value across diverse disease contexts.

Our tool’s outstanding performance is due to its use of a specificity-driven iterative optimization that systematically guides users, which emulates the manual annotation process in a more standardized and reproducible manner. The data-driven cell subtype identification tool operates without relying on well-annotated reference datasets, enabling the discovery of novel and potentially important cell subtypes. As shown in the application of ESCC, Subtypist identified a prognosis-associated subtype without reference datasets or prior knowledge.

Cell subtype identification within the same cell type is often determined by multiple continuous axes of variation, which makes it extremely difficult to perform discrete divisions on these continua [1, 58]. Under this, statistical tests in scSHC may become overly conservative under such continuous structures, reducing their sensitivity. These challenges highlight the importance of method robustness across different data structures. Accordingly, Subtypist-elbow performs better in datasets characterized by continuous transcriptional gradients, as exemplified by the tonsil dataset, whereas Subtypist-default shows relatively more stable performance in datasets with less pronounced non-continuous structural features (Supplementary Fig. S17). The multiscale strategy provides a quantitative and comparative methodological framework across different granularities for determining cell states characterized by continuity. Thus, the tool achieves improved cellular classification. Besides, our approach exhibits strong flexibility across modalities. It can be applied to snRNA-seq datasets, as shown in MI dataset, where we identified a remodeling-associated subtype. Our approach is also compatible with spatial transcriptomics, enabling accurate identification of cell subtypes from gene expression profiles and providing the foundation for uncovering their spatial organization and associated functional states. We further evaluated the impact of data preprocessing strategies on performance and found that Subtypist exhibits higher sensitivity to the choice of integration method than to normalization strategy [59] (Supplementary Fig. S18). This observation suggests that selecting an appropriate integration approach is critical in downstream subtype analysis.

Transcriptomic data only provide the mRNA abundance at a given time point, representing the transcriptional activity of cells. However, the biological interpretation is complex, as mRNA levels often do not correlate perfectly with functional protein abundance due to multi-layered post-transcriptional regulation. Furthermore, scRNA-seq data alone lack crucial contextual information, such as cellular spatial context, signaling activity, or chromatin state, all essential for a comprehensive characterization of cell identity. These data are also susceptible to transient biological states, including cell cycle and microenvironmental factors, resulting in expression differences even within the same cell type. In addition, technical noise, such as dropout events and insufficient sequencing depth, severely affects the reliable detection of key marker genes and increases uncertainty in subtype identification. These factors represent common challenges for all identification methods.

Consequently, integrating other omics data with the increasing availability of multi-modal datasets, such as proteomics, epigenomics, or spatial transcriptomics (e.g. MERFISH [60], seqFISH [61]), is potentially beneficial. By integrating spatial information based on scRNA-seq, the tool can enable more accurate identification of cell subtypes and further enhance the ability to analyze their spatial organization and functional states. Despite the strengths of computational approaches, further experimental validation remains essential for confirming the robustness and biological relevance of the results.

In summary, Subtypist achieves identification of cell subtypes without relying on prior reference. By standardizing a emulates manual annotation process, it improves the accuracy and consistency of subtype annotation. Moreover, the use of a data-driven approach to assist in the discovery of new disease-related subtypes not only deepens our understanding of the mechanisms of disease pathogenesis, but also provides important guidance for exploring more targeted treatment strategies.

Methods

Specificity-guided cluster merging

We developed a computational method, Subtypist, which is designed to robustly and automatically refine cell clusters and annotate cells based on phenotypic molecules derived from scRNA-seq data. Subtypist seamlessly integrates into the standard Seurat analysis pipeline, operating directly on Seurat objects that have already undergone upstream processing (including quality control, data normalization and scaling, principal component analysis (PCA), and shared nearest neighbor (SNN) graph construction). This design enables it to be seamlessly embedded into existing analysis workflows.

The core algorithm of Subtypist is a multiscale analysis strategy that is designed to comprehensively identify the more biologically meaningful cell subtype divisions from a series of clustering results of different granularities. The strategy systematically evaluates a set of candidate clustering solutions pre-generated by Seurat’s graph-based community detection algorithm across a recommended resolution range of 0.1 to 1.5. For each initial clustering result at each resolution, Subtypist independently performs an iterative optimization process with conditional termination to correct potential over-segmentation and validate the specificity of the resulting clusters at that resolution.

This iterative optimization process involves three key stages: (i) Specificity score calculation, (ii) Specificity validation, and (iii) Similarity-driven merging.

Specificity score calculation

To quantify marker support during iterative cluster refinement, we define a gene-level specificity score. For each cluster i, candidate marker genes are identified through differential expression analysis, and the top N genes (default: N = 300, selected to ensure sufficient coverage of candidate marker genes) are retained based on their absolute average log2 fold change, . The directionality of retained markers (upregulated, downregulated, or both) is configurable.

For a gene j in cluster i, let represent the detection rate difference, where and denote the fractions of cells expressing gene j within cluster i and across all other clusters, respectively. The base score is defined as:

The gene receives a non-zero specificity score only when it satisfies directional consistency and at least one of two gating conditions. First, the fold change direction must match the regulation mode: when searching for upregulated markers, only genes with are considered; when searching for downregulated markers, only genes with are considered; in bidirectional mode, both are retained.

Second, the gene must pass at least one of the following criteria: (i) sufficient detection specificity, defined as (default: τ =0.4, selected to capture moderate detection-rate differences); or (ii) sufficiently high differential expression magnitude, defined as (default: θ =0.5, corresponding to moderate differential expression magnitude). Genes failing both criteria receive Si,j=0, and this product formulation ensures positive specificity scores when directional evidence is consistent.

For each cluster, marker support is aggregated from the top-scoring genes. The cluster-level specificity score is defined as , where denotes the set of top k markers (default: k = 3, representing a compact set of representative candidate marker genes).

Specificity validation

The merging procedure terminates based on two complementary criteria: (i) rule-based specificity validation, and (ii) elbow detection with optional subsampling stability refinement.

Rule-based specificity validation

This step quantitatively validates all cluster-level specificity scores to determine if the iterative process should terminate. Termination occurs under two distinct conditions: first, if all clusters in the partition satisfy the predefined threshold (), indicating complete convergence; or second, if exactly one non-specific cluster () remains that has already been merged once. This latter case designates the cluster as “un-optimizable,” halting the process to prevent further futile merging. If neither termination condition is met, the algorithm proceeds to the Similarity-Driven Merging stage for further refinement.

Elbow-based specificity validation

For each resolution r, Subtypist records the merge trajectory across steps . Let denote the clustering result after merge step t, and let denote the mean cluster specificity score at that step.

The elbow point is estimated from the complete merge trajectory using a two-segment linear model with a hinge term. For each candidate breakpoint t, we fit:

where , is the intercept, is the initial slope, and represents the change in slope after breakpoint t.

The estimated elbow step is defined as the breakpoint that minimizes the residual sum of squares:

To avoid premature termination, elbow detection is activated only when predefined minimum thresholds for data volume and merge depth are satisfied.

Optionally, a local stability validation via subsampling is performed to improve robustness. To balance computational efficiency and robustness, stability is evaluated only within a local neighborhood of the estimated elbow. The candidate window is defined as:

where \varDelta controls the window size around the estimated elbow. For each candidate step t , B subsampling replicates are performed. In replicate b, a random fraction ρ of cells is sampled to form subset . The merge procedure is rerun on , producing partition . Stability is quantified by comparing this to the full-data partition restricted to the same cells, , using the Adjusted Rand Index:

The final termination step is selected as the stable candidate closest to the estimated elbow:

where δ denotes the stability threshold (default: 0.85).

Similarity-driven merging

When specificity validation fails, we merge the most similar pair of clusters based on marker-gene overlap. For each cluster i, let denote its marker-gene set from the specificity score calculation. Similarity between clusters i and j is quantified using a weighted Jaccard index:

where represents the shared marker genes, and denotes the size of the union set. The numerator sums the absolute fold changes of shared markers from both clusters, weighting the overlap by differential expression magnitude. If no markers are shared, .

Alternatively, the similarity can optionally be computed separately for upregulated and downregulated markers. Each marker set is decomposed by fold change sign: and . The directional similarities and are computed using the same weighted Jaccard formula applied to the respective subsets, and the final similarity is their average: .

Next, we identify the two closest clusters based on the similarity matrix and merge them into a single cluster.

Inter-resolution comparison

To evaluate the partitioning quality across multiple granularities, a resolution-level specificity score is computed for each candidate resolution r within a predefined set . This metric standardizes the comparison by calculating the arithmetic mean of the absolute cluster-level specificity scores across all detected clusters:

All candidate resolutions are ranked in descending order based on their scores. The default optimal resolution is identified as the partitioning granularity that maximizes this metric, providing a quantitative reference to guide downstream analyses:

Optionally, an inter-resolution consensus analysis framework can be deployed to quantify the topological stability of clusters across neighboring partitioning granularities (Supplementary Fig. S27). To assess local stability under continuous resolution shifting, a neighbor evaluation window of radius h (by default, h=1) is constructed by ordering candidate resolutions sequentially and excluding the resolution itself:

For a reference cluster at resolution r, its optimal topological counterpart within a neighboring resolution is identified by maximizing their Jaccard overlap: , where . To evaluate the topological robustness of localized partitions with an emphasis on low-abundance lineages, two complementary stability metrics, and , are evaluated simultaneously for each cluster:

Comparison with other clustering tools on simulated datasets

To re-establish the representation of subtypes within a cell type, we simulated scRNA-seq data using ESCO. In greater detail, we configured varying library sizes at the default, set distinct base means for the genes, selected “group” as the cell structure, and adjusted the mean–variance relationship for genes across different cells, based on the characteristic patterns of each subtype (Supplementary Table S1). Additionally, the effects of real-world technical factors, like downsampling noise, were effectively integrated during the simulation process using default parameter settings.

The simulated data were then processed using Seurat’s standard analysis pipeline with default settings, including the following functions: NormalizeData, ScaleData, RunPCA, RunUMAP, and FindNeighbors. And add the simulated grouping labels as ground-truth.

Using single-cell RNA-seq datasets simulated with ESCO, we systematically compared Subtypist with several commonly used clustering methods, including the Louvain and Leiden algorithms implemented in Seurat (denoted as Seurat-Louvain and Seurat-Leiden), scSHC, SC3, and the Presto-based accelerated implementation of Subtypist. All methods were executed with default parameters and predefined method-specific parameter grids. Evaluation metrics included clustering accuracy, runtime, and computational resource usage. For SC3, analyses were performed using the number of clusters recommended by its built-in estimator.

Comparison with other annotation tools on real datasets

Baseline scRNA-seq data analysis

The data underwent normalization via LogNormalize or SCTransform [62] and were integrated with the different integration methods according to the original study, including canonical correlation analysis (CCA) of Seurat and Harmony [63]. Then, conducted PCAs, followed by dimensional reduction techniques like uniform manifold approximation and projection (UMAP) analysis, and constructed a nearest neighbor graph.

Human pancreatic ductal adenocarcinoma (datasets 1–4)

We obtained annotated scRNA-seq data from a primary research study which is available at NCBI GEO at accession number GSE231535, and selected four distinct cell types—myeloid cells, endothelial cells, lymphocytes, and stromal cells—as the major cell types for subsequent benchmarking. Each dataset was processed using the default custom scripts employed in the original assays. Data were normalized using SCTransform and batch-corrected using Harmony to ensure consistency across datasets. The parameter settings for each step were strictly aligned with those outlined in the original research. The final scRNA-seq datasets comprised the following cellular and genomic dimensions: myeloid cells (14 686 cells and 15 010 genes), endothelial cells (19 264 cells and 15 189 genes), lymphocytes (33 037 cells and 15 398 genes), and stromal cells (21 469 cells and 15 549 genes). The subtype annotation information from the original datasets was utilized as the ground truth for benchmarking and performance evaluation (Supplementary Table S7).

Human liver (datasets 5–6)

We obtained two cell subtypes from NCBI GEO at accession number GSE192742: lymphoid cells (26 536 genes, 101 075 cells, with 10 subtypes), and myeloid cells (23 092 genes, 40 821 cells, with 10 subtypes). These datasets were normalized using LogNormalize and batch-corrected using Harmony to ensure consistency across datasets.

Human inflammatory bowel (datasets 7–9)

We acquired the raw count matrices derived from scRNA-seq data, which are available at NCBI GEO at accession number GSE214695, then filtered out low-quality cells and performed data preprocessing following the methodology outlined in the original study. The final scRNA-seq datasets include plasma cells with 22 873 genes, 15 076 cells, and six subtypes; myeloid cells with 19 405 genes, 3733 cells, and 10 subtypes; and epithelial cells with 23 933 genes, 10 812 cells, and 11 subtypes. These datasets were normalized using LogNormalize and batch-corrected using Harmony to ensure consistency across datasets.

Human liver transplantation (datasets 10–12)

Following the same analytical approach, we obtained three subtypes: myeloid cells (22 556 genes, 6941 cells, five subtypes), neutrophils (19 800 genes, 11 452 cells, five subtypes), and T/NK cells (23 499 genes, 44 799 cells, eight subtypes), which can be accessed through figshare. Each dataset was processed using the standard protocol described in the original study. Data were normalized using LogNormalize and integrated using CCA–based batch correction to ensure consistency across datasets.

Mouse NAFLD liver (datasets 13–15)

We obtained three cell populations from NCBI GEO at accession number GSE156059: fibroblasts (19 024 genes, 4025 cells, four subtypes), lymphocytes (20 156 genes, 21 322 cells, 12 subtypes), and myeloid cells (21 429 genes, 42 922 cells, 12 subtypes). These datasets were normalized using LogNormalize and batch-corrected using Harmony to ensure consistency across datasets.

Human tonsil (datasets 16–18)

We acquired the raw count matrices derived from the Human Cell Atlas tonsil atlas Zenodo. We extracted three subtypes (37 378 genes each): germinal-center B cells (92 218 cells, 14 subtypes), plasma cells (14 256 cells, 21 subtypes), and naive/memory B cells (211 269 cells, 15 subtypes). These datasets were normalized using LogNormalize and batch-corrected using Harmony to ensure consistency across datasets.

We evaluated these 18 publicly available single-cell RNA-seq datasets to systematically compare multiple clustering methods, including the Louvain and Leiden algorithms implemented in Seurat version 4 and version 5 (denoted as Seurat-Louvain and Seurat-Leiden), scSHC, and SC3. All methods were run using their default parameter settings, and clustering accuracy was assessed across datasets.

In addition, we applied scGPT, scType, and CellTypist to evaluate their performance in cell-type identification tasks. For scType, we used the built-in database of cell-type marker genes for human datasets, and constructed a mouse-specific reference marker set based on mouse markers curated from CellMarker 2.0. For CellTypist, we selected the official reference models according to the species, organ, and tissue type of each dataset (Supplementary Table S7). For analyses requiring fine-grained annotation of immune cell subtypes, we additionally employed the high-resolution “Immune_All_Low.pkl” reference model provided by CellTypist to obtain more detailed annotations.

ARI. The Adjusted Rand Index (ARI) is a widely used metric for evaluating the similarity between two clustering results while correcting for random chance. Given a set of n samples, let X and Y denote two cluster assignments. The ARI is computed as:

where RI represents the Rand Index, is its expected value under random clustering, and \max (RI) is the maximum possible RI.

NMI. The Normalized Mutual Information (NMI) measures the agreement between two clustering assignments by quantifying the amount of shared information between them, normalized by their individual information content. Given a set of n samples, let X and Y denote two cluster assignments. The NMI is computed as:

where is the mutual information between X and Y, and H(X) and H(Y) are the entropies of the two cluster assignments. Specifically,

where and are the numbers of clusters in Xand Y, is the number of samples assigned to cluster i in X and cluster j in Y, and are the corresponding marginal counts, and n is the total number of samples. NMI ranges from 0 to 1, with higher values indicating greater agreement between the two clustering assignments.

F1 score. Because clustering labels are permutation-invariant, predicted cluster assignments were aligned to ground-truth labels prior to computing F1-based metrics and marker concordance.

Let and denote the ground-truth labels and predicted cluster assignments for n cells, respectively. A contingency matrix was defined as

To establish an optimal cluster-to-label correspondence, we formulated a linear assignment problem and solved it using the Hungarian algorithm:

where K and L denote the numbers of ground-truth classes and predicted clusters, respectively, and . When K≠ L, the contingency matrix was zero-padded to ensure a valid assignment formulation.

To calculate the F1-score, we used the following formula:

where TP, FP, FN, and TN represent true positives, false positives, false negatives, and true negatives, respectively.

In order to assess the unbiased performance of Subtypist in the context of a multi-class classification scenario, we computed two additional metrics: the Macro-average F1 score and the Weighted Average F1-score, which provide a more comprehensive assessment across all classes. The Macro-average F1-score is calculated as:

Where refers to the F1 score of the i- th class, and N is the total number of classes. This calculation provides an unweighted average of the F1-scores across all classes, treating each class equally regardless of its size.

On the other hand, the Weighted Average F1 score is calculated using the following formula:

where represents the number of instances in the i- th class. This metric takes into account the class distribution, weighting the F1 scores of each class by its percentage in the dataset.

Marker concordance. Marker concordance quantifies whether Subtypist-derived cluster markers overlap with independently curated subtype marker panels. Let denote the marker set of predicted cluster c, and let denote the curated marker set of reference subtype k. Predicted clusters are first aligned to reference subtypes using the optimal cluster-to-label assignment obtained via the Hungarian algorithm. The marker concordance is defined as:

where denotes the set of evaluated clusters in dataset d. The indicator function equals 1 if at least one marker gene overlaps between the predicted cluster and its matched reference subtype, and 0 otherwise. Higher values indicate stronger agreement between Subtypist-derived markers and curated subtype marker panels.

Human hepatocellular carcinoma dataset analysis

Single-cell transcriptomic data of human HCC, measured through 10× Genomics technology, revealed six major cell types. We subset the macrophage cell type with 4812 cells and 27 948 genes, which were thoroughly categorized into FOLR2 Macrophages, CCL3L1 Macrophages, SPP1 Macrophages, and Chemotactic Macrophages, as the input for Subtypist analysis. Subsequently, the macrophage subset was further subjected to thorough analysis, which involved variable feature selection, PCA with 40 principal components, integration (Harmony), and dimensional reduction techniques for each cell type individually. Subtypist with default parameters effectively recommends the optimal resolution and the corresponding phenotypic molecule annotation results of eight subtypes through specificity scores. A separate cluster was excluded due to its limited cell count and was characterized by robust expression of neutrophil markers (CSF3R, FCGR3B) (Supplementary Fig. S26), yielding a final set of seven macrophage subtypes. For the 10× Visium ST data, the original annotations from the ST data were retained as a reference, which included tumor sections from eight patients with HCC who received anti-PD-1 treatment. These patients (N = 8) were categorized into two groups based on their response to treatment: five non-responders and three responders. Differentially expressed genes shared between the original annotations of the ST data and the Subtypist result labels from the single-cell transcriptome data were retained for subsequent analysis.

Inference of spatial distribution of single-cell transcriptomes data

We employed CellTrek to directly assign single cells to spatial coordinates to map single-cell transcriptomic data onto spatial transcriptomics data. The parameters were set for each spot to be assigned to a single cell, with a maximum of five spots per cell.

Deconvolution of spatial and bulk transcriptome cellular components

To infer the cell type composition of bulk RNA data and spatial transcriptomics data, we used scRNA-seq data as a reference for deconvolution by SpaTalk [64]. We calculated the average expression profile of each subtype to ensure that the reference was more representative and robust. All parameters were kept in the analysis default settings.

Calculation of nearest neighbor points

To identify the nearest neighbors, the Euclidean distance between the target point and all other points was calculated. The distance to the point itself was excluded, and the minimum distance to the closest 10 neighboring points was retained.

Human esophageal squamous-cell carcinoma ecosystem dataset analysis

Single-cell transcriptomic data of the human esophageal squamous-cell carcinoma ecosystem, measured through 10× Genomics technology, revealed five major cell types. We subset the Epithelial cells with 44 730 cells as the input for Subtypist analysis. Subsequently, the epithelial cells underwent comprehensive analysis, including variable feature selection, RunPCA (50 components), integration using Harmony, dimensionality reduction, and the application of the FindNeighbors function for each cell type individually. Subtypist efficiently identified subtypes with phenotypic molecules across multiple resolutions and recommended the optimal resolution (0.5) based on the specificity score.

Verification in the human ESCC bulk RNA sequencing (RNA-seq) dataset

Bulk RNA-seq was conducted in samples obtained from the Cancer Genome Atlas (TCGA) of 94 ESCC patients for survival analysis. The 94 samples were further screened to exclude adjacent tissue samples, remove outliers with abnormal expression patterns, and remove samples with substandard RNA quality, ultimately obtaining 88 samples. We analyzed the correlation between gene expression levels in epithelial expression programs and survival time in patients with ESCC. The gene expression data from bulk RNA-seq were normalized using counts per million (CPM) and log2-transformed.

Survival analysis

1. In the gene expression-based strategy, we first quantified the expression profile of marker genes by identifying the top 50 DEGs in this subtype (Supplementary Table S8). The AddModuleScore function was then applied to compute a composite expression score for each specimen. Based on this score, specimens were stratified into high-expression (≥ upper quartile) and low-expression (< upper quartile) groups.

2. In the cell proportion-based strategy, to estimate the proportional composition of specific subtypes, we employed SpaTalk to deconvolute bulk RNA-seq data using scRNA-seq counts as a reference (see Methods: SpaTalk). Samples were then classified into high-proportion (≥ upper quartile of subtype proportion) and low-proportion (< upper quartile of subtype proportion) groups.

Survival analysis was assessed using the Kaplan–Meier method, and the significance of the differences was evaluated using the log-rank test.

Human myocardial infarction dataset analysis

Single-cell transcriptomic data of the human myocardial infarction, measured through 10× Genomics technology, revealed 11 major cell types. We subset the macrophage with 28 975 genes and 21 082 cells as the input for Subtypist analysis. The data underwent the same preprocessing protocol, and using its default parameters, Subtypist efficiently identified subtypes with phenotypic molecules across multiple resolutions and recommended the optimal resolution (0.5) based on the specificity score. In ST data measured using 10× Visium, genes shared between the original ST annotations and the Subtypist results labels from scRNA-seq data were retained. Additionally, the expression profile of DEGs in cell subtypes was quantified using the AddModuleScore function.

Cell–cell communication analysis

We applied CellChat to infer cell–cell communication in scRNA-seq data. The analysis was based on highly expressed ligands in sender cells and receptors in receiver cells. We selected the CellChat database for humans and filtered out cell communications with fewer than five cells. Other analyses followed the pipeline with default settings.

Inference of cell–cell communication from spatial transcriptomic data

We applied the COMMOT to infer the cell–cell communication in spatial transcriptomics data, summarizing directions of spatial signaling. We infer interactions referring to the CellChat ligands and receptors database for each cell type and then filter the significantly highly expressed ligands and receptors with the percentage of expressed cells >5%. Then we constructed cell–cell communication networks for the ligand-receptor pairs with a spatial distance constraint of 500 μm.

Pathway and biological process enrichment analysis

To evaluate pathway enrichment across different cell types, the FindAllMarkers function was employed to identify differentially expressed genes (DEGs) in each cell group relative to others. The analysis was conducted with only.pos = TRUE and min.pct = 0.2, focusing on genes with an adjusted P-value (computed using the Wilcoxon test) below 0.05. Then we selected the top 100 most highly expressed genes, ranked by their average fold change. This analysis was conducted using the clusterProfiler package (version 4.6.2) in R, focusing on the biological process (BP) and molecular function (MF) categories, with a q-value cutoff of 0.05. In addition, the Metascape web tool was also employed to perform pathway and biological process enrichment analysis.

Statistical analysis

R version 4.2.1 was used for all statistical analyses. All statistical tests are specified in the corresponding figure legends. Wilcoxon rank-sum tests were used for group comparisons, Spearman correlation for association analyses, and log-rank tests for survival analyses. Multiple testing correction was applied using FDR where appropriate. Results with P < .05 were considered statistically significant.

Key Points

• Subtypist moves beyond fixed-label annotation by discovering de novo cell subtypes and assigning identities using subtype-specific phenotypic molecules.

• Subtypist integrates resolution selection into subtype annotation, automatically determining clustering granularity from multiscale results and reducing manual tuning.

• Subtypist uses specificity scores to prioritize representative marker genes for each subtype and provide interpretable evidence for annotation.

• Simulated benchmarks demonstrate the robustness and accuracy of Subtypist, while real-world analyses identify disease-relevant subtypes.

Author contributions

X.S. and X.F. conceived and designed the study. Y.Y. and X.S. implemented the Subtypist methods. R.C. and J.Q. collected the datasets used in this article and performed benchmarking experiments. J.G., Y.Y., Y.P., and H.G. conducted experimental analyses on biological datasets. Y.Y. and X.S. wrote the manuscript. Y.Y., X.S., and X.F. revised the manuscript. All the authors read and approved the final manuscript.

Conflicts of interest

The authors declare no competing interests.

Funding

This work was supported by the “Pioneer” and “Leading Goose” R&D Program of Zhejiang (2024C03106, X.F.), the National Natural Science Foundation of China (82474160, X.S.).

Data and code availability

The source codes used for this study and the results of comparisons with other methods are freely available at GitHub (https://github.com/ZJUFanLab/Subtypist).

The original data used in this paper can be accessed through the following links: (1) Human pancreatic ductal adenocarcinoma (datasets 1–4): NCBI GEO, accession GSE231535; (2) Human liver (datasets 8–9): NCBI GEO, accession GSE192742; (3) Human inflammatory bowel (datasets 5–7): NCBI GEO, accession GSE214695. (4) Human liver transplantation (datasets 10–12): figshare; (5) Mouse NAFLD liver (datasets 13–15): NCBI GEO, accession GSE156059; (6) Human tonsil (datasets 16–18): Human Cell Atlas tonsil atlas, Zenodo; (7) Human hepatocellular carcinoma data: Mendeley Data (skrx2fz79n); (8) Human esophageal squamous-cell carcinoma ecosystem data: GEO accession: GSE160269; (9) Human myocardial infarction scRNA-seq and 10× Visium data: https://cellxgene.cziscience.com/collections/8191c283-0816-424b-9b61-c3e1d6258a77.

Resource availability

Lead contact

Further information and requests for resources should be directed to and will be fulfilled by the lead contact, Xiaohui Fan (fanxh@zju.edu.cn).

Materials availability

This study did not generate new unique materials.