Specificity-driven cell-gene graph learning identifies rare cell states in single-cell and spatial transcriptomic data

Abstract

Detecting rare cell populations that drive development, differentiation, and disease-associated transformation remains a central challenge in biology and medicine. Although these populations often represent promising targets for intervention, they are difficult to resolve from single-cell transcriptomic data because most methods rely on homophily-based cell–cell similarity, which can merge rare cells into dominant populations and mask their subtle transcriptional signatures. The challenge is further amplified in multi-sample analyses, where batch correction can dilute rare-cell-specific signals. Here, we present scFormer, a heterogeneous graph transformer (HGT) framework for sensitive and robust rare-cell discovery. scFormer constructs a Z-score-guided cell-gene heterogeneous graph in which highly specific marker genes serve as informational bridges, embedding rare-cell features directly into the graph topology rather than inferring them from global neighbors. This design provides a clear biological rationale for rare-cell recovery, as low-abundance cells can remain connected through shared high-specificity genes even when local cell–cell neighborhoods are sparse. An integrated optimization strategy jointly performs representation learning, clustering, and optional batch correction, enabling rare-cell discovery while preserving biological structure. Across 125 simulated and 18 real datasets, scFormer consistently achieved competitive or superior performance relative to existing approaches. Applied to diverse multi-sample single-cell and spatial transcriptomics datasets, scFormer recovered known but weakly represented populations and revealed previously obscured cell states, including proliferative club cells in the airway epithelium, revival stem cells during intestinal regeneration, and rare embryonic cell states from spatial transcriptomics. Overall, scFormer provides a unified framework for identifying biologically meaningful rare populations while mitigating batch effects in multi-sample datasets.

Supplementary Information

The online version contains supplementary material available at 10.1007/s44307-026-00121-y.

Introduction

Multicellular tissues are composed of a spectrum of cell types and dynamic states, including low-frequency populations that are often transient, spatially restricted, or activated only under stress or injury. These rare populations are of particular translational interest in oncology: even when they represent only a small fraction of cells (< 1%–5%), they can disproportionately influence tumor progression, immune evasion, therapeutic resistance, and relapse (Chu et al. 2024; Galassi et al. 2024; Goddard et al. 2024; Shaffer et al. 2017). Beyond cancer, rare populations also mark lineage commitment, regenerative trajectories, and specialized functional niches across normal and diseased tissues (Belarif et al. 2018; Ross et al. 1993; Wijnands et al. 2023). Resolving these populations is therefore central not only to understanding tissue development and homeostasis, but also to improving disease stratification and precision therapeutic intervention.

Single-cell RNA sequencing (scRNA-seq) now enables comprehensive profiling of cellular heterogeneity at unprecedented resolution (Lu et al. 2024; Liao et al. 2025; Potter 2018; Choi and Kim 2019; Jaitin et al. 2014; Hu et al. 2024; Sang and Kong 2024). However, rare-cell identification faces two key technical challenges. First, transcriptional signals from rare cells are easily masked by abundant cell types, limiting feature extraction (Zhou et al. 2024). Second, batch effects arising from differences in experimental protocols, sequencing depth, and sample handling introduce systematic variation that can confound downstream analyses, especially in multi-sample studies (Argelaguet et al. 2021).

Numerous computational strategies have been proposed to address rare-cell detection, yet fundamental limitations persist. Rarity-based approaches (for example, FiRE (Jindal et al. 2018) and GapClust (Fa et al. 2021)) and feature-selection strategies such as GiniClust3 (Dong and Yuan 2020) quantify cellular uniqueness but remain sensitive to noise and subtle expression shifts. Clustering-based methods (RaceID3 (Herman et al. 2018), CellSIUS (Wegmann et al. 2019), scCAD (Xu et al. 2024)) and feature-learning approaches such as SCMER (Liang et al. 2021) depend primarily on global cell–cell similarity. As a consequence, when rare cells lack sufficient neighbors to form distinct clusters, their signatures are often over-smoothed and absorbed into dominant lineages.

A critical and largely unresolved issue concerns the interaction between rare-cell detection and batch correction. Dedicated integration tools such as Harmony (Korsunsky et al. 2019), BBKNN (Polański et al. 2020), ComBat (Zhang et al. 2020) and MNN Correct (Haghverdi et al. 2018) are optimized to align global distributions across samples, not to preserve rare signals. Over-correction may inadvertently merge true rare populations into transcriptionally similar abundant types. Conversely, applying rare-cell detection without integration risks inflating false positives driven by technical variation rather than biology (Yong et al. 2021). This tension motivates the need for methods that remove batch effects while preserving weak but biologically meaningful signals.

Graph neural network and transformer-based models have recently expanded the methodological toolkit for single-cell analysis. For example, scMGCA uses a graph-embedding autoencoder to learn cell–cell topology and cluster assignments across single-cell platforms (Yu et al. 2023), whereas scHeteroNet introduces a heterophily-aware graph neural network for cell-type annotation and novel-cell detection (Liu et al. 2025). Beyond cell–cell graph modeling, heterogeneous graph transformer (HGT)-based frameworks have shown that distinct biological entities and relation types can be modeled explicitly in single-cell data. DeepMAPS represents cells and genes as distinct node types and uses HGT-based message passing for single-cell biological network inference (Ma et al. 2023), while MarsGT applies graph transformer learning to rare-population inference from single-cell multi-omics data (Wang et al. 2024). These studies demonstrate the value of GNNs, graph transformers, and cell-gene heterogeneous graphs in single-cell analysis. However, existing graph-based frameworks still leave an important challenge for rare-cell discovery. Rare populations are often too sparse to form robust cell–cell neighborhoods, and batch correction may further dilute weak but biologically meaningful variation. A method designed for this setting should therefore not only represent cells and genes in a heterogeneous graph, but also construct a rare-cell-oriented cell-gene topology and learning objective that preserve cell-specific transcriptional signals before they are absorbed into abundant populations.

Here, we present scFormer, a unified computational framework that uses HGTs for rare-cell identification with optional batch effect correction. Rather than relying on conventional cell–cell similarity networks that can obscure rare signals, scFormer converts cell-specific marker evidence into the topology of a cell-gene heterogeneous graph. Each cell is connected to a compact set of highly informative genes, allowing rare transcriptional programs to be represented by direct cell-gene edges before graph propagation. In this topology, highly specific marker genes serve as informational bridges, encoding biological priors directly into graph structure. This design is intended to preserve minority transcriptional programs that are often smoothed away in homophily-driven cell–cell graphs. The HGT then propagates information through node- and relation-aware attention, enabling cell and gene nodes to exchange information while retaining their distinct biological roles. scFormer further integrates representation learning, clustering regularization and optional batch correction within a unified objective, allowing rare-cell discovery across both single-sample and multi-sample settings. We validated scFormer on 125 simulated and 18 real scRNA-seq datasets spanning diverse tissues and biological contexts, and observed improved benchmark performance and robustness compared with existing methods. Beyond recapitulating known rare cell types, scFormer identified previously uncharacterized rare populations in airway epithelium and intestinal tissue, and showed applicability to spatial transcriptomics by identifying rare embryonic cell states that could be examined within their original tissue coordinates. Together, these results establish scFormer as a practical framework for uncovering functionally important rare cell populations, with broad implications for developmental biology, immunology, and precision medicine.

Methods

Data preprocessing

Single-cell RNA-seq and spatial transcriptomic datasets were represented as a non-negative expression matrix $\mathbf{C}\in {\mathbb{R}}_{\ge 0}^{N\times G}$, where $N$ is the number of cells or spatial profiles and $G$ is the number of detected genes. Each row corresponds to a cell or spatial profile, and each column corresponds to a gene. For raw count matrices, library-size normalization and log transformation were first applied:

$$\begin{array}{c}{\mathbf{X}}_{\mathit{ij}}=\text{log}\left(1,+,\frac{{10}^4{C}_{\mathit{ij}}}{{\sum }_{g=1}^G{C}_{\mathit{ig}}}\right),\end{array}$$ 1

where ${\mathbf{C}}_{\mathit{ij}}$ denotes the count of gene $j$ in cell or spatial profile $i$, and $\mathbf{X}$ denotes the resulting non-negative normalized expression matrix. For datasets distributed as preprocessed non-negative expression matrices, the provided expression values were used directly as $\mathbf{X}$, followed by the same gene-wise standardization procedure to obtain $\mathbf{Z}$. This preprocessing workflow was applied to both scRNA-seq and spatial transcriptomic expression matrices.

To quantify cell-specific gene enrichment, gene-wise standardization was then performed on $\mathbf{X}$. For each gene $j$, the mean expression ${\mu }_j$ and standard deviation ${\sigma }_j$ were computed across all cells or spatial profiles, and the standardized expression matrix $\mathbf{Z}$ was defined as:

$$\begin{array}{c}{\mathbf{Z}}_{\mathit{ij}}=\frac{{\mathbf{X}}_{\mathit{ij}}-{\mu }_j}{{\sigma }_j},\end{array}$$ 2

Thus, $\mathbf{X}$ and $\mathbf{Z}$ serve distinct roles in scFormer. The standardized matrix $\mathbf{Z}$ was used as the specificity score matrix for cell-gene edge construction and as the source of node input features. For a cell node $i$, the input feature was its standardized expression vector ${\mathbf{Z}}_{i,:}\in {\mathbb{R}}^G$. For a gene node $j$, the input feature was its standardized expression profile across all cells or spatial profiles, ${\mathbf{Z}}_{:,j}\in {\mathbb{R}}^N$. Cell and gene features were projected to the hidden dimension by node-type-specific input encoders before HGT message passing. The non-negative matrix $\mathbf{X}$, rather than the standardized matrix $\mathbf{Z}$, was retained as the expression target for the reconstruction objective during training.

Construction of cell-gene heterogeneous graph

Selecting high expression genes and forming the adjacency matrix

After obtaining the standardized expression matrix $\mathbf{Z}\in {\mathbb{R}}^{N\times G}$, scFormer constructed a cell-gene heterogeneous graph by connecting each cell to genes with high cell-specific Z-scores. Unlike traditional GNNs that rely on KNN graphs to model cell–cell relationships, which often assume homophily and may overlook heterogeneous relationships, scFormer uses genes as an intermediary bridge to link cells. This gene-mediated approach captures cell-gene associations directly and helps retain cell-specific transcriptional signals.

For each cell $i$, genes are ranked according to their standardized expression values ${\mathbf{Z}}_{\mathit{ij}}$. The selected gene set for cell $i$ is defined as

$$\begin{array}{c}{\mathrm{\Gamma }}_i^K={\text{TopK}}_j\left({\mathbf{Z}}_{\mathit{ij}}\right)\end{array}$$ 3

where $K$ denotes the number of genes retained for each cell. In all main analyses, K = 20 was used as a fixed default graph-construction parameter rather than being optimized separately for individual datasets or selected according to evaluation labels. This value was chosen to maintain a sparse but sufficiently informative cell-gene graph. To assess whether this default choice introduced parameter-specific bias, we performed a sensitivity analysis across K = 5, 10, 20, 30 and 50, using 25 simulated datasets spanning different proportions of differentially expressed genes, thereby covering sparse-to-denser marker-profile settings. The results showed that rare-cell detection performance increased from smaller $K$ values to K = 20 and then approached a plateau, whereas larger $K$ values produced only marginal F1-score gains while increasing runtime and graph size (Supplementary Fig. 1). Here, $K$ controls the sparsity of the cell-gene graph by limiting the number of selected genes per cell.

We then construct a binary adjacency matrix $\mathbf{A}\in {\{\text{0,1}\}}^{N\times G}$ as follows:

$$\begin{array}{c}{\mathbf{A}}_{\mathit{ij}}=\left\{\begin{array}{c}1,ifj\in {\mathrm{\Gamma }}_i^K\\ 0,otherwise\end{array},,\right)\end{array}$$ 4

Thus, ${\mathbf{A}}_{\mathit{ij}}=1$ indicates that gene $j$ is selected as one of the top $K$ Z-score-ranked genes for cell $i$, whereas ${\mathbf{A}}_{\mathit{ij}}=0$ indicates no selected cell-gene edge.

Defining node types

To enable heterogeneous graph modeling, we distinguish between two node types: cell nodes (indexed by $\left\{0\right),1,...,\left(N,-,1\right\}$) and gene nodes (indexed by $\left\{N\right),N+1,...,\left(N,+,G,-,1\right\}$). This indexing convention ensures that the numerical ranges of cell nodes and gene nodes do not overlap, facilitating the definition of directed edges and their types.

Constructing the cell-gene edge indices

We extract nonzero entries from the adjacency matrix $\mathbf{A}$ to form the selected cell-gene associations. Let

$$\begin{array}{c}\mathrm{\Omega }\in \left\{\left((i,j)|,{\mathbf{A}}_{\mathit{ij}},=,1,,,0,\le ,i,<,N,,,0,\le ,j,<,G\right\}\right),\end{array}$$ 5

where $\mathrm{\Omega }$ denotes the set of all selected cell-gene pairs. Because cell and gene nodes are placed on disjoint index sets, we define a constant offset $\mathrm{\Delta }=N$. A gene index j, where $0\le j<G$, is mapped to the graph node index $\mathrm{\Delta }+j$ to distinguish it from the cell indices $\{0,...,N-1\}$.

To represent each selected association bidirectionally, we add both a gene-to-cell edge and a cell-to-gene edge. For each $(i,j)\in \mathrm{\Omega }$, the gene-to-cell edge has source node $\mathrm{\Delta }+j$ and target node i, Formally,

$$\begin{array}{c}{\mathcal{E}}_{\text{gene}\to \text{cell}}=\left\{\left((,\mathrm{\Delta },+,j,,,i,),|,(,i,,,j,),\in ,\mathrm{\Omega }\right\}\right).\end{array}$$ 6

For the same selected pair $(i,j)\in \mathrm{\Omega }$, we also add the reverse cell-to-gene edge, with source node i and target node $\mathrm{\Delta }+j$:

$$\begin{array}{c}{\mathcal{E}}_{\text{cell}\to \text{gene}}=\left\{\left((,i,,,\mathrm{\Delta },+,j,),|,(,i,,,j,),\in ,\mathrm{\Omega }\right\}\right).\end{array}$$ 7

By merging these two directed edge sets, we obtain the complete set of edges:

$$\begin{array}{c}E={\mathcal{E}}_{\text{gene}\to \text{cell}}\cup {\mathcal{E}}_{\text{cell}\to \text{gene}}.\end{array}$$ 8

In practice, these edges are often stored in a $2\times \left|\left(\mathcal{E}\right|\right)$ array, where the first row contains source-node indices and the second row contains target-node indices.

Specifying edge and node types.

We also define an edge-type vector $\mathbf{edge}\_\mathbf{type}$ to annotate whether an edge represents gene to cell or cell to gene, such that:

$$\begin{array}{c}\mathbf{edge}\_\mathbf{type}=\left[\underset{\left|{\mathcal{E}}_{\text{gene}\to \text{cell}}\right|edges}{\underbrace{\text{0,0},\cdots ,0}},,,\underset{\left|{\mathcal{E}}_{\text{cell}\to gene}\right|edges}{\underbrace{\text{1,1},\cdots ,1}}\right].\end{array}$$ 9

Similarly, a node-type vector $\mathbf{n}\mathbf{o}\mathbf{d}\mathbf{e}\_\mathbf{t}\mathbf{y}\mathbf{p}\mathbf{e}$ can label each node as either cell or gene:

$$\begin{array}{c}\mathbf{node}\_\mathbf{type}=\left[\underset{N\text{cell nodes}}{\underbrace{\text{0,0},\cdots ,0}},,,\underset{Ggenenodes}{\underbrace{\text{1,1},\cdots ,1}}\right].\end{array}$$ 10

These annotations enable heterogeneous graph algorithms to handle different node and edge types explicitly.

Subsampling of a cell-gene heterogeneous graph

For mini-batch training, scFormer sampled cells first and then constructed the corresponding batch-specific gene set. Before mini-batch construction, Python, NumPy, and PyTorch random seeds were fixed, and cell indices were randomly shuffled before being split into mini-batches. In the main experiments, the batch size was set to 300 cells.

For each mini-batch $B$, the gene nodes were defined as the union of the top $K$ Z-score-selected genes from all cells in that batch:

$${\mathrm{\Gamma }}_B=\bigcup _{i\in B}{\mathrm{\Gamma }}_i^K.$$ 11

Thus, for a mini-batch containing $\left|B\right|$ cells, each sampled cell contributes up to $K$ selected genes, and duplicated genes selected by multiple cells are merged. The number of gene nodes in a mini-batch is therefore determined by $\left|{\mathrm{\Gamma }}_B\right|$, with an upper bound of $\left|B\right|\times K$, rather than by an independently fixed cell-to-gene sampling ratio.

The mini-batch subgraph was induced by the sampled cells $B$, the selected gene set ${\mathrm{\Gamma }}_B$, and all corresponding bidirectional cell-gene edges between them. The only graph expansion after cell sampling was the one-hop inclusion of selected gene nodes defined by ${\mathrm{\Gamma }}_i^K$; no additional multi-hop neighbor expansion or independent gene-node sampling was performed. Instead, the batch graph was defined directly by the specificity-based cell-gene associations established during graph construction. The cell-node features in this mini-batch were ${\mathrm{Z}}_B,:$, and the gene-node features were ${\mathit{Z}}_{{:,\mathrm{\Gamma }}_B}^T$, namely the standardized profiles of the selected genes across all cells or spatial profiles.

This batching strategy preserves the selected cell-gene connections within each batch rather than sampling cell and gene nodes independently. Because each sampled cell contributes its own top $K$ genes, sampled cell nodes retain their specificity-based cell-gene edges when at least one gene is retained for that cell. Gene nodes are included only if they are selected by at least one sampled cell, so isolated gene nodes are not introduced during mini-batch construction. The generated mini-batch indices were saved and reused during training, further ensuring reproducible subgraph construction across runs using the same random seed.

Construction of the heterogeneous graph convolution network

An effective strategy for capturing the distinct characteristics of cells ($\mathbf{n}\mathbf{o}\mathbf{d}\mathbf{e}\_\mathbf{t}\mathbf{y}\mathbf{p}\mathbf{e}=0$) and genes ($\mathbf{n}\mathbf{o}\mathbf{d}\mathbf{e}\_\mathbf{t}\mathbf{y}\mathbf{p}\mathbf{e}=1$) is the application of a HGT layer. In scFormer, cell nodes and gene nodes are first projected into a common hidden dimension by node-type-specific input encoders before HGT message passing. Let $h_v^{\left(l,-,1\right)}\in {\mathbb{R}}^{d_h}$ denote the hidden representation of node $v$ at layer $l-1$, where $d_h$ is the hidden dimension. The number of attention heads is denoted by $H$, and the dimension of each head is $d_k=d_h/H$.

For each node type $\tau \in \left\{\left(\text{0,1}\right\}\right)$, where $\tau =0$ denotes cell nodes and $\tau =1$ denotes gene nodes, scFormer uses node-type-specific linear projections to compute queries, keys, and values:

$$\begin{array}{cc}& {\mathbf{Q}}^{\left(\tau \right)}={\mathbf{H}}_{\tau }^{\left(l,-,1\right)}{\mathbf{W}}_Q^{\tau },{\mathbf{K}}^{\left(\tau \right)}={\mathbf{H}}_{\tau }^{\left(l,-,1\right)}{\mathbf{W}}_K^{\tau },{\mathbf{V}}^{\left(\tau \right)}\hfill \\ \hfill & ={\mathbf{H}}_{\tau }^{\left(l,-,1\right)}{\mathbf{W}}_V^{\tau },\hfill \end{array}$$ 12

where ${\mathbf{W}}_Q^{\tau },{\mathbf{W}}_K^{\tau },{\mathbf{W}}_V^{\tau }\in {\mathbb{R}}^{d_h\times d_h}$. The projected vectors are reshaped into $H$ attention heads, each with dimension $d_k$.

For an edge $s\to t$, let $s$ be the source node, $t$ be the target node, and $r\in \text{0,1}$ be the edge type, where $r=0$ denotes a gene-to-cell edge and $r=1$ denotes a cell-to-gene edge. For each attention head $h$, scFormer applies relation-specific transformations to the key and value vectors:

$$\begin{array}{c}{\tilde{\mathbf{k}}}_s^{r,h}={\mathbf{W}}_{\mathit{ATT}}^{r,h}{\mathbf{k}}_s^h,{\tilde{\mathbf{v}}}_s^{r,h}={\mathbf{W}}_{\mathit{MSG}}^{r,h}{\mathbf{v}}_s^h,\end{array}$$ 13

where ${\mathbf{W}}_{\mathit{ATT}}^{r,h},{\mathbf{W}}_{\mathit{MSG}}^{r,h}\in {\mathbb{R}}^{d_k{\times d}_k}$. A relation-specific prior ${\mu }_{r,h}$ is also learned for each edge type and attention head.

The unnormalized attention score from source node $s$ to target node $t$ under edge type $r$ and head $h$ is computed as:

$$\begin{array}{c}{e}_{\mathit{ts}}^{r,h}=\frac{{\mathbf{q}}_t^h·{\tilde{\mathbf{k}}}_s^{r,h}}{\sqrt{d_k}}{\mu }_{r,h}.\end{array}$$ 14

The attention coefficient is then obtained by applying Softmax over all incoming edges of the same target node $t$, separately for each attention head:

$$\begin{array}{c}{\mathrm{\alpha }}_{\mathit{ts}}^{r,h}=\frac{\text{exp}(e_{\mathit{ts}}^{r,h})}{{\sum }_{s\prime \in \mathcal{N}(t)}{\text{exp}(e}_{ts\prime }^{r,h})}.\end{array}$$ 15

Thus, the Softmax normalization is performed across source nodes connected to the same target node, rather than across all nodes or all edges in the graph.

The message passed from source node $s$ to target node $t$ is computed for each head as:

$$\begin{array}{c}{\mathbf{m}}_{\mathit{ts}}^{r,h}={\mathrm{\alpha }}_{\mathit{ts}}^{r,h}{\tilde{\mathbf{v}}}_s^{r,h}.\end{array}$$ 16

Messages from all heads are concatenated, and incoming messages are summed over the neighborhood of the target node:

$$\begin{array}{c}{\mathbf{m}}_t^l=\sum _{s\in \mathcal{N}(t)}{\text{Concat}}_1^H({\mathbf{m}}_{\mathit{ts}}^{r,h}).\end{array}$$ 17

The aggregated message is then passed through a node-type-specific output projection, followed by dropout, a learnable skip connection, and layer normalization:

$$\begin{array}{c}{\mathbf{h}}_t^l={\text{LayerNorm}}_{{\tau }_t}\left({\beta }_{{\tau }_t},{\mathbf{W}}_O^t,\text{GELU},\left({\mathbf{m}}_t^l\right),+,\left(1,-,{\beta }_{{\tau }_t}\right),{\mathbf{h}}_t^{l-1}\right),\end{array}$$ 18

where ${\tau }_t$ is the type of target node $t$, ${\mathbf{W}}_O^t\in {\mathbb{R}}^{d_h\times d_h}$ is the node-type-specific output projection, and ${\beta }_{{\tau }_t}$ is a learnable skip coefficient. Stacking multiple HGT layers yields the final cell embeddings ${\mathbf{H}}_c^{\text{final}}$ and gene embeddings ${\mathbf{H}}_g^{\text{final}}$, which are subsequently used for clustering and reconstruction.

Training of scFormer

During training, each mini-batch cell-gene subgraph generated by the aforementioned subsampling strategy was passed through the HGT network. The batch-specific node features were propagated through HGT layers to compute cell embeddings and gene embeddings. For a mini-batch containing the sampled cell set $B$ and the selected gene set ${\mathrm{\Gamma }}_B$, the resulting cell and gene embeddings are denoted as follows:

$$\begin{array}{c}{\mathbf{H}}_c\in {\mathbb{R}}^{\left|B\right|\times d_h},{\mathbf{H}}_g\in {\mathbb{R}}^{\left|{\mathrm{\Gamma }}_B\right|\times d_h},\end{array}$$ 19

where $d_h$ denotes the latent dimension. The standardized matrix $\mathbf{Z}$ was used for node input features, whereas the non-negative matrix $\mathbf{X}$ was retained as the reconstruction target. These embeddings were optimized using a multi-component objective consisting of a KL-based reconstruction regularizer, a pseudo-label-based clustering loss, an intra-cluster compactness term, and an optional batch-correction term.

KL divergence loss for information preservation

The reconstruction objective was computed using the non-negative mini-batch expression submatrix ${\mathbf{X}}_{B,{\mathrm{\Gamma }}_B}$, rather than the standardized Z-score matrix $\mathbf{Z}$. Given the learned cell embeddings ${\mathbf{H}}_{\mathit{c}}$ and gene embeddings ${\mathbf{H}}_{\mathit{g}}$, scFormer decoded cell-gene affinities using inner products between cell and gene embeddings:

$$\begin{array}{c}{S}_{c\to g}={\mathbf{H}}_c{\mathbf{H}}_g^{\top },S_{g\to c}={\mathbf{H}}_g{\mathbf{H}}_c^{\top }.\end{array}$$ 20

The decoded affinity matrices and the corresponding non-negative expression targets were converted into probability distributions by row-wise softmax over matched dimensions:

$$\begin{array}{cc}& P_{c\to g}={\text{softmax}}_g\left(X_{B,{\mathrm{\Gamma }}_B}\right),Q_{c\to g}\hfill \\ \hfill & ={\text{softmax}}_g\left(S_{c\to g}\right),\hfill \end{array}$$ 21

$$\begin{array}{cc}& P_{g\to c}={\text{softmax}}_c\left(X_{B,{\mathrm{\Gamma }}_B}^{\top }\right),Q_{g\to c}\hfill \\ \hfill & ={\text{softmax}}_c\left(S_{g\to c}\right).\hfill \end{array}$$ 22

The KL divergence regularizer was then defined as:

$$\begin{array}{c}{\mathcal{L}}_{\mathit{KL}}=D_{\mathit{KL}}\left(P_{c\to g},\Vert ,Q_{c\to g}\right)+D_{\mathit{KL}}\left(P_{g\to c},\Vert ,Q_{g\to c}\right),\end{array}$$ 23

where $D_{\mathit{KL}}\left(P,\Vert ,Q\right)=\frac{1}{N}{\sum }_{i=1}^N{P}_i\text{log}\frac{P_i}{Q_i}$ quantifies the divergence between two distributions $P$ (original) and $Q$ (reconstructed).

This KL term was used as an auxiliary distribution-matching regularizer, not as a likelihood model for raw sparse counts. Library-size effects were addressed before training by normalization in $\mathbf{X}$. Because this term is not formulated as a count-generative likelihood, scFormer does not explicitly model zero inflation. Zero entries are retained in the non-negative matrix $\mathbf{X}$ and contribute to the softmax-normalized target distributions within each mini-batch. After softmax normalization, the reconstruction term compares relative cell-gene association patterns within each mini-batch rather than absolute count magnitudes. Importantly, the softmax normalization was applied to the mini-batch expression submatrix defined by the sampled cells $B$ and the selected gene set ${\mathrm{\Gamma }}_B$, rather than to the full transcriptome. Therefore, the KL term regularizes relative cell-gene association patterns within the selected mini-batch subgraph after graph construction, rather than driving gene selection from all detected genes. Because the reconstruction target is $\mathbf{X}$, negative standardized Z-score values are not used in the KL objective. Thus, the loss avoids applying KL divergence directly to negative standardized values while still encouraging the learned embeddings to preserve expression-derived cell-gene relationships.

Cross-entropy loss for cell type discrimination

Initial pseudo-labels were generated once before model training using Leiden clustering on the normalized expression matrix $\mathbf{X}$. These pseudo-labels were kept fixed during training and used as weak supervision to stabilize the learned cell embeddings. Resolution parameters were chosen according to dataset size, with 0.2 for datasets containing < = 500 cells, 0.5 for datasets containing 500–5,000 cells, and 0.8 for datasets containing > 5,000 cells when no custom value was provided. Neighborhood graphs for Leiden initialization used 5, 10, or 15 nearest neighbors for small, medium, or large datasets, respectively, unless a dataset-specific setting was specified.

The clustering loss was computed using label smoothing between the pseudo-labels and the model-predicted cell probabilities:

$$\begin{array}{c}{\mathcal{L}}_{\mathit{cluster}}=-\frac{1}{\left|B\right|}{\sum }_{i\in B}{\sum }_{k=1}^c{\tilde{y}}_{\mathit{ik}}log{p}_{\mathit{ik}},\end{array}$$ 24

where $K_c$ is the number of pseudo-label classes in the training set, ${\tilde{y}}_{\mathit{ik}}$ denotes the smoothed pseudo-label distribution, and $p_{\mathit{ik}}$ denotes the predicted probability that cell $i$ belongs to class $k$.

Intra-cluster compactness term

To encourage compact cell embeddings within each pseudo-labeled group, scFormer computed the mean cosine similarity among cells assigned to the same pseudo-label within each mini-batch. For each class $k$ represented in the mini-batch, let $C_k$ denote the set of cells assigned to that class. The intra-cluster similarity score was defined as:

$$\begin{array}{c}{\mathcal{S}}_{\mathit{intra}}=\sum _{k:\left|C_k\right|>1}\frac{1}{{\left|C_k\right|}^2}\sum _{i,j\in C_k}\frac{{\mathbf{H}}_c^{\left(i\right)}·{\mathbf{H}}_c^{\left(j\right)}}{\Vert {\mathbf{H}}_c^{\left(i\right)}\Vert \Vert {\mathbf{H}}_c^{\left(j\right)}\Vert }.\end{array}$$ 25

This term is a similarity score rather than a positive penalty. It was therefore subtracted from the total objective, consistent with maximizing intra-cluster compactness during optimization.

Batch correction loss (optional)

When batch correction was enabled, scFormer added a batch-correction term computed from the cell embeddings in the current mini-batch. This term consisted of a distribution-alignment component and a local-structure component. For each batch $b$ represented in the mini-batch, the mean embedding vector ${\mu }_b$ and variance vector $v_b$ were computed across cells from that batch. The distribution-alignment term penalized variation in these batch-level embedding moments:

$$\begin{array}{c}{\mathcal{L}}_{\mathit{distribution}}={\text{mean}}_l{\text{Var}}_b\left({\mu }_{b,l}\right)+0.5{\text{mean}}_l{\text{Var}}_b\left(v_{b,l}\right).\end{array}$$ 26

The local-structure component was computed from Euclidean distances between cells and their nearest neighbors in the embedding space. The number of neighbors was set to $k_s=min(15,\left|B\right|-1)$:

$$\begin{array}{c}{\mathcal{L}}_{\mathit{structure}}=\frac{1}{\left|B\right|}\sum _{i\in B}\frac{1}{k_s}\sum _{j=1}^{k_s}{\Vert H_c^i-H_c^{n_i(j)}\Vert }_2,\end{array}$$ 27

where $n_i(j)$ denotes the index of the j-th nearest neighboring cell of cell $i$ among the other cells in the same mini-batch, based on Euclidean distance in the cell embedding space. The optional batch-correction loss was:

$$\begin{array}{c}{\mathcal{L}}_{\mathit{batch}}={\mathcal{L}}_{\mathit{distribution}}+0.5{\mathcal{L}}_{\mathit{structure}}.\end{array}$$ 28

Total loss function

The overall training objective used fixed weights for all reported experiments:

$$\begin{array}{c}{\mathcal{L}}_{\mathit{total}}={\mathcal{L}}_{\mathit{KL}}+{\mathcal{L}}_{\mathit{cluster}}-{\mathcal{S}}_{\mathit{intra}}+I_{\mathit{batch}}·0.5{\mathcal{L}}_{\mathit{batch}}.\end{array}$$ 29

where $I_{\mathit{batch}}$ equals 1 when batch correction is enabled and 0 otherwise. Thus, without batch correction, the optimized objective is ${\mathcal{L}}_{\mathit{KL}}+{\mathcal{L}}_{\mathit{cluster}}-{\mathcal{S}}_{\mathit{intra}}$. With batch correction enabled, the optional batch term is added with coefficient 0.5. No additional tuned loss weights were used in the main experiments.

Model parameters were updated using the AdamW optimizer with learning rate $5\times {10}^{-4}$ and weight decay 0.1. Dropout was set to 0.3 in the HGT network, and label smoothing was set to 0.1 for the pseudo-label classification term. Training was run for a fixed 100 epochs in the main experiments, and the final model state was used for downstream embedding extraction and clustering. No validation split, validation-based early stopping, or validation-based model selection was used.

This training protocol reflects the intended use of scFormer as a dataset-specific representation-learning method for single-cell and spatial transcriptomic analyses. The model is trained separately for each dataset to learn embeddings for downstream analysis, rather than being trained as a supervised predictor intended for direct transfer to unseen datasets. To reduce unstable optimization and degenerate memorization, training combined sparse cell-gene graph construction, dropout, AdamW weight decay, label smoothing, and the multi-component objective described above.

Inference of cell clusters

Following model training, we apply the robustly trained classifier to the entire cellular graph to assign each cell to its predicted cluster. The model maintains performance across the complete dataset by leveraging a union of subgraphs that encompasses all cells. The trained network produces HGT-derived cell embeddings, which are then mapped to per-cell assignment scores via a learnable assignment function. Final cluster labels are obtained by selecting the highest-scoring cluster (argmax), yielding predicted cluster assignments across the dataset. This approach yields a comprehensive set of predicted cluster labels across the entire dataset, facilitating downstream biological interpretation and analysis.

Simulated data

To construct a systematic benchmarking dataset for evaluating rare-cell identification by scFormer and competing methods, synthetic scRNA-seq datasets were generated using Splatter (Zappia et al. 2017) (v1.30.0) across a predefined multidimensional parameter grid. A full-factorial design was used to vary three differential expression (DE)-related parameters: de.prob (0.1, 0.3, 0.5, 0.7, 0.9), de.facLoc (0.1, 0.3, 0.5, 0.7, 0.9), and de.facScale (0.2, 0.4, 0.6, 0.8, 1.0), resulting in 125 unique parameter combinations. For each combination, one dataset containing 2,500 cells and 5,000 genes was simulated. Cells were assigned to a major population (Group1) and a rare population (Group2) at a 99:1 ratio to model extreme class imbalance typical of rare-population settings.

Simulated count matrices were processed using a standardized preprocessing workflow. Specifically, counts were log-transformed as log2(counts +1) for methods requiring normalized inputs; for methods operating on raw counts, the original simulated counts were used as input. No label-informed gene filtering or post hoc modification of the simulated gene set was performed, ensuring that all methods were evaluated on the same underlying simulated expression matrices.

To quantify the transcriptomic separability between the major and rare populations across parameter combinations, a separation metric was computed for each dataset. Principal component analysis (PCA) was applied to the expression matrix, and the centroid of each population was calculated along the first principal component (PC1). Dataset separation was defined as the absolute difference between the two centroids on PC1. This separation metric was used to characterize a continuous difficulty gradient across the simulated datasets for downstream performance stratification.

Comparison with other methods

To evaluate the performance of scFormer in the task of rare-cell identification, a systematic comparison was conducted against seven representative rare-cell identification methods. The selected competing methods, including CellSIUS, FiRE, GapClust, GiniClust3, RaceID3, scCAD, and SCMER, cover rarity-scoring, differential-expression, clustering-based, and feature-learning strategies, and were selected because they could be applied consistently across the simulated and real scRNA-seq benchmarks. Methods that require additional modalities, spatial coordinates, or do not directly produce rare-cell assignments were not included in the rare-cell F1 benchmark, but relevant batch-correction methods were evaluated separately in the multi-sample PBMC analysis. To ensure reproducibility and to reduce inadvertent bias introduced by dataset-specific tuning, we followed the parameterization recommended by the original publications and official documentation for each method, and we applied a consistent evaluation protocol across all datasets. Method parameters were not optimized against the evaluation labels; instead, key settings were fixed before performance evaluation according to the original publications, official tutorials, or package defaults. The complete software versions and parameter settings are provided in Supplementary Table 1. The datasets were processed and evaluated following the official tutorials for each respective method. The performance of each algorithm was quantitatively compared using standard classification metrics, including precision, recall, and F1 score.

Furthermore, to assess the capability of scFormer in batch effect correction, a comparative analysis was performed against prominent batch correction methods, namely Harmony, BBKNN, MNN Correct, and ComBat. Batch-correction baselines were run using standardized implementations with commonly used recommended settings. For the Harmony-Leiden pipeline used as the two-stage integration and clustering baseline in Fig. 4, Harmony was applied to PCA embeddings using batch labels, followed by nearest-neighbor graph construction on Harmony-corrected embeddings and Leiden clustering. To assess the effect of Leiden resolution, we performed a resolution sweep across 0.2, 0.4, 0.6, 0.8, 1.0, 1.2, 1.5, 2.0, 2.5, and 3.0. At each resolution, predicted rare cells were defined using the same cluster-abundance criterion as in the main analysis, namely cells assigned to clusters comprising ≤ 5% of the dataset. The complete resolution-sweep results are provided in Supplementary Table 2.

Fig. 4.

scFormer simultaneously corrects batch effects and identifies rare cell populations. a UMAP plots illustrating batch integration performance in a PBMC dataset (30,669 cells across four batches). Visualizations compare the uncorrected data with results from scFormer, Harmony, ComBat, BBKNN, and MNN. Colors denote the batch of origin for each cell (s1d1-s4d1, where s indicates sample and d indicates donor; all batches are four samples from donor 1). b Quantitative evaluation of batch correction performance. scFormer achieves the highest iLISI score (2.695), exceeding Harmony and other benchmarked methods, which signifies more effective batch integration. c Precision, recall, and F1 scores for rare-cell identification using scFormer, scCAD and the Harmony-Leiden pipeline. For Harmony-Leiden, Harmony was used for batch correction followed by Leiden clustering, with clusters ≤ 5% of cells designated as rare. Sensitivity to Leiden resolution was evaluated separately across resolutions from 0.2 to 3.0, with the complete results provided in Supplementary Table 2. d UMAP projections of 30,669 PBMC cells colored by the original annotations (leftmost) alongside the rare clusters identified by scCAD (second from left), the Harmony-Leiden pipeline (third from left), and scFormer (rightmost)

Rare population definition

For real datasets, we used the original study annotations as reference cell-type annotations. A reference cell type was defined as rare if its prevalence within the dataset was ≤ 5% (default threshold r = 5%). For each method, predicted rare cells were defined as cells belonging to clusters whose abundance was ≤ 5% of all cells in that dataset. We then computed cell-level precision, recall, and F1 score by comparing predicted rare cells against reference rare cells (rare vs non-rare). These published annotations were used as reference labels rather than definitive biological ground truth. Therefore, precision, recall, and F1 score in real-data benchmarks quantify concordance with the original study annotations at the rare-versus-non-rare level.

Evaluation index

For real scRNA-seq datasets, we evaluated annotation-level rare-cell recovery by calculating precision, recall, and F1 score against the reference rare labels defined from the original study annotations. Precision quantifies the proportion of cells predicted as rare that were also labeled as rare in the reference annotations, calculated as:

$$\begin{array}{c}Precision=\frac{\mathit{TP}}{TP+FP},\end{array}$$ 30

where TP (true positives) denotes cells predicted as rare that were also labeled as rare in the reference annotations, and FP (false positives) denotes cells predicted as rare but labeled as non-rare in the reference annotations. High precision ensures minimal false positives, enhancing the reliability of detecting biologically significant cell types.

Recall, or sensitivity, measures the proportion of reference-labeled rare cells recovered as rare by each method, defined as:

$$\begin{array}{c}Recall=\frac{\mathit{TP}}{TP+FN},\end{array}$$ 31

where FN (false negatives) denotes cells labeled as rare in the reference annotations but not recovered as rare by the method. Given the low abundance of reference rare cells in scRNA-seq datasets, high recall is important for reducing missed reference-annotated rare populations.

The F1 score was the harmonic mean of precision and recall, calculated as:

$$\begin{array}{c}\text{F1}score=2\times \frac{Precision·Recall}{Precision+Recall},\end{array}$$ 32

balances these metrics to provide a robust evaluation, particularly in class-imbalanced scenarios. In real-data benchmarks, higher precision, recall, and F1 scores indicate stronger concordance with published reference rare-cell annotations. Under this annotation-level evaluation, scFormer showed stronger overall concordance than competing methods across diverse scRNA-seq datasets.

Additionally, we employed the integration local inverse Simpson’s index (iLISI) and batch entropy to assess scFormer’s batch correction performance in integrating multi-batch scRNA-seq datasets. These metrics evaluate the model’s ability to mitigate batch effects while preserving biological heterogeneity, ensuring reliable downstream analyses such as differential expression or trajectory inference. The iLISI measures local batch mixing by computing the inverse of the Simpson’s diversity index within the KNN graph of each cell, expressed as:

$$\begin{array}{c}iLISI=\frac{1}{{\sum }_b{p}_b^2},\end{array}$$ 33

where $p_b$ is the proportion of cells from batch b in the neighborhood. Higher iLISI values indicate better batch integration, reflecting a uniform distribution of cells across batches in local structures.

Batch entropy quantifies the global uniformity of batch distribution, calculated as:

$$\begin{array}{c}Entropy=-{\sum }_b{p}_b·log(p_b),\end{array}$$ 34

where $p_b$ represents the proportion of cells from batch b in the dataset. Elevated entropy values signify reduced batch-related biases, indicating effective batch correction.

Post hoc spatial enrichment analysis

For the sci-Space slide_4E analysis, spatial coordinates were not used as input to scFormer, but were used only for post hoc evaluation. Spatial self-enrichment was quantified as the fraction of same-cluster cells among the 10 nearest spatial neighbors for each cell and compared with 10,000 size-matched random permutations. Empirical p values were adjusted using the Benjamini–Hochberg procedure. Concordance with published sci-Space annotations was evaluated using hypergeometric enrichment tests, and marker/module enrichment was quantified by comparing mean expression in each target cluster with all remaining cells.

Statistics and reproducibility

This study did not employ statistical methods to predetermine sample size, nor were randomization or blinding applied during experimental design or result evaluation. No data were excluded from the analysis beyond standardized data quality control measures. To ensure reproducibility of simulated data generation, a fixed random seed was set using splatter (Zappia et al. 2017) (v.1.30.0). Differentially expressed genes (DEGs) were identified using the FindMarkers function from the Seurat R package (Hao et al. 2024) (v.5.3.0), employing a two-sided Wilcoxon rank sum test with a false discovery rate (FDR) threshold of 0.05 and an inter-group absolute fold-change threshold of 1.5. Fold-change values were calculated based on the mean expression levels of each gene between groups. p values were adjusted using Bonferroni correction to account for multiple testing across the total number of genes in the dataset.

Results

Overview of the scFormer framework

Instead of relying on cell–cell homophily, scFormer encodes rare-cell specificity directly into a cell-gene heterogeneous graph, allowing rare transcriptional programs to propagate through the network without being diluted by dominant cell types. scFormer is a graph-based computational framework designed to detect rare cell populations by encoding biological priors directly into the underlying topology. Rather than treating all expressed genes equivalently, scFormer is motivated by the premise that broadly expressed genes provide limited discriminatory power for resolving rare states. Instead, it prioritizes genes exhibiting high cell-specific expression, treating them as informative anchors for distinguishing rare populations.

As shown in Fig. 1, the framework comprises four stages: input scRNA-seq matrices, feature engineering and heterogeneous graph construction, HGT-based representation learning with multi-objective optimization, and rare-cell identification.

Fig. 1.

Overview of the scFormer framework. The workflow is divided into four sequential stages: (1) Input Data: raw scRNA-seq count matrices from one or more batches are processed to generate normalized cell- and gene-level features. (2) Feature Engineering & Graph Construction: This stage processes data through two parallel streams. The first stream (top) performs normalization to generate cell and gene feature matrices (${\text{H}}_{\text{C}}$ and ${\text{H}}_{\text{G}}$). The second stream (bottom) constructs a cell-gene heterogeneous graph by filtering for specificity (Top Z-scores) to link cells with specific marker genes. (3) HGT Model & Optimization: The constructed heterogeneous graph and preprocessed features are fed into the HGT model. Through heterogeneous attention and message aggregation, the model learns low-dimensional embeddings, optimized by a total loss function (KL reconstruction, clustering discrimination, and optional batch correction). (4) Rare-Cell Identification: Cluster assignments are obtained by applying a learnable assignment function to the HGT-derived cell embeddings. Each cell is assigned to the cluster with the highest predicted score

Rather than simply preprocessing data to reduce noise, scFormer transforms the normalized expression matrix into a structured graph that highlights unique biological signals. While the initial phase handles raw input data (Step 1) and standard normalization (Step 2, top pathway), scFormer’s critical innovation lies in its specificity-driven topology construction. Specifically, scFormer employs a quantitative selection strategy (Step 2, bottom pathway) where cell-gene connections are established only for genes exhibiting high expression and statistically significant specificity (calculated via Z-scores) within a given cell. This approach ensures that the graph captures biologically relevant genes that are highly expressed in specific cell types but underexpressed elsewhere, embedding critical biological priors into the graph’s topological structure. Unlike traditional graph neural networks (GNNs) that rely on k-nearest neighbor (KNN) graphs to model cell–cell relationships, which often assume homophily and connect cells with similar features (Levine et al. 2015; Liu et al. 2025; Yu et al. 2023), scFormer uses genes as an intermediary bridge to link cells. This gene-mediated approach overcomes the limitations of homophily-based graphs, which may overlook heterogeneous relationships, and better captures complex biological interactions by leveraging gene expression patterns to connect cells with diverse functional roles.

The constructed cell-gene heterogeneous graph is processed by an HGT model (Step 3), which captures complex cell-gene interactions through node- and edge-specific operations tailored for heterophily. Unlike conventional GNNs that assume homophily, scFormer’s HGT is designed to handle heterophily, accommodating diverse cell-gene relationships where connected nodes may represent distinct biological entities. The model is optimized via a unified multi-objective loss function that primarily enforces accurate gene expression reconstruction and maximizes clustering consistency. While optimized for single-sample analysis, the framework offers the flexibility to incorporate an auxiliary loss term for batch effect correction, extending its applicability to multi-sample datasets without compromising its core performance. Finally, during inference (Step 4), cluster assignments are obtained by applying a learnable assignment function to the HGT-derived cell embeddings. Each cell is assigned to the cluster with the highest assignment score, enabling downstream identification of rare populations based on cluster abundance and marker enrichment.

scFormer demonstrates high robustness and accuracy in simulated benchmarks

To systematically assess sensitivity to rare populations, we generated 125 synthetic datasets using the Splatter package (Zappia et al. 2017). Each dataset comprised 2,500 cells and 5,000 genes, with two clusters present at a 99:1 proportion. This extreme imbalance was designed to emulate rare populations comprising ~1% of cells, a regime commonly encountered when profiling low-frequency immune subsets or small stem/progenitor niches. To probe robustness across varying levels of difficulty, simulation settings were varied to generate datasets spanning highly distinct to minimally separable cell populations. Dataset separation was quantified as the distance between cluster centroids along the first principal component (Supplementary Fig. 2).

scFormer was benchmarked against seven representative methods (CellSIUS, FiRE, GapClust, GiniClust3, RaceID3, scCAD, and SCMER). Performance was evaluated using precision, recall, and F1 score (Supplementary Dataset 1). Across all simulations, scFormer achieved the highest overall F1 score (0.988) and exhibited strikingly low variance, indicating strong robustness to extreme imbalance and heterogeneous signal strengths (Fig. 2a-b).

Fig. 2.

scFormer demonstrates superior performance in identifying rare-cell clusters across varying dataset difficulties. a F1 score comparison of scFormer and seven competing methods (scCAD, GapClust, RaceID3, FiRE, SCMER, GiniClust3, CellSIUS) across 125 simulated datasets with 2,500 cells, 5,000 genes, and extreme cluster imbalance (99:1 ratio). scFormer achieves the highest average F1 score (0.988), followed by scCAD (0.982) and GapClust (0.930). b Variance in F1 scores across all datasets, showing scFormer exhibits the lowest variance (0.008), indicating superior stability and robustness to parameter variations. c Performance comparison stratified by separation difficulty. Datasets were divided using the 10th percentile (6.9237) as the cutoff: high-separation (solid bars, separation index > 6.9237) and low-separation (striped bars, separation index ≤ 6.9237) conditions. d Variance comparison between high- and low-separation groups, demonstrating that scFormer maintains exceptional stability even under challenging conditions with minimal transcriptomic differences between cell clusters. * indicates that CellSIUS yielded F1 scores of zero across all datasets

To explicitly test biologically challenging scenarios, where rare populations differed only subtly from abundant populations, datasets were stratified by separation difficulty using the centroid distance metric. The 10th percentile of the separation distribution (6.9237) was used to distinguish low- from high-separation scenarios. Low-separation settings reflect cases such as closely related cell subtypes, subtle activation states, or continuous state transitions with small transcriptional effect sizes, which are known to confound rare-cell identification by reducing separability while maintaining strong abundance imbalance.

Under this stratification, scFormer maintained high and stable performance in both low- and high-separation conditions (Fig. 2c-d). While several baseline methods exhibited marked performance degradation and increased variability in low-separation datasets, scFormer remained robust, supporting reliable recovery of rare clusters even when inter-cluster differences were minimal. Notably, scCAD showed comparable accuracy to scFormer in some settings (Fig. 2c) but displayed weaker stability across conditions (Fig. 2d). Together, these results highlight scFormer’s advantage in jointly optimizing accuracy and robustness, which is critical for sensitively detecting biologically meaningful rare subpopulations in complex single-cell datasets. A sensitivity analysis of the graph-sparsity parameter $K$ further showed that the default value K = 20 reached a near-performance plateau while avoiding the additional runtime and graph-size increase observed with larger $K$ values (Supplementary Fig. 1).

scFormer achieves state-of-the-art performance on real datasets

To evaluate scFormer under real biological heterogeneity, performance was assessed on 18 publicly available scRNA-seq datasets spanning diverse tissues and biological contexts, including immune (PBMC), epithelial (airway), nervous system (brain and retina), and multiple organ and disease settings (kidney, liver, pancreas, breast, tonsil, and renal cell carcinoma), with dataset sizes ranging from 313 (Deng et al. 2014) to 49,147 (Hao et al. 2021) cells (Supplementary Table 3). Rare populations were defined from the original study annotations as cell types with prevalence ≤ 5% within each dataset and used as reference labels. Because annotation granularity differs across studies, these labels were treated as reference annotations for benchmarking rather than absolute ground truth. Unless otherwise specified, predicted rare cells were defined as cells in clusters with abundance ≤ 5%.

Across datasets, scFormer showed the best overall annotation-level rare-cell recovery performance (Fig. 3a). In the precision-recall landscape, scFormer most frequently occupied the high-precision/high-recall regime, indicating an improved balance between sensitivity and false discovery compared with competing methods. Aggregated across datasets, scFormer achieved the highest mean F1 score (0.595), corresponding to a 22.1% improvement over scCAD (0.488) and a 93.8% improvement over GiniClust3 (0.307) (Fig. 3b). Consistent with these averages, the F1 score distribution of scFormer exhibited the highest median with a comparatively tight interquartile range, supporting strong concordance with published reference annotations and stable behavior across heterogeneous tissues and study designs.

Fig. 3.

Annotation-level performance evaluation of scFormer against state-of-the-art methods across 18 scRNA-seq datasets. a Precision-recall scatter plot showing the performance distribution of six methods (scFormer, GiniClust3, FIRE, scCAD, GapClust, and SCMER) across all datasets. Each point represents one dataset, with dashed gray lines indicating F1 score contours (0.2, 0.4, 0.6, 0.8). b Box plot comparison of F1 scores across all methods, displaying median values, interquartile ranges, and performance distribution. Statistical significance and detailed dataset information are provided in Supplementary Dataset 2

Representative case studies further illustrate scFormer’s robustness at the dataset level. In a mouse retina dataset (Dou et al. 2022), scFormer achieved an F1 score of 0.955, substantially higher than scCAD (0.649) and GiniClust3 (0.282). In a more challenging setting with pronounced cellular heterogeneity, the human renal cell carcinoma dataset (Zhang et al. 2021), scFormer remained the top-performing method with an F1 score of 0.279, exceeding scCAD (0.207) and GiniClust3 (0.267). Together, these results support scFormer as a robust approach for recovering reference-annotated rare populations across tissues and study designs.

scFormer balances batch correction and rare-cell recovery during multi-sample integration

To evaluate scFormer for multi-sample integration, performance was assessed with respect to batch effect correction, while considering the need to preserve subtle signals relevant to rare-cell identification after integration. A PBMC dataset comprising 30,669 cells and 14,087 genes across four batches was used, and cell-type annotations from the original study were treated as the reference for quantitative evaluation.

scFormer was first benchmarked against established batch-correction methods, including Harmony, BBKNN, MNN Correct, and ComBat. UMAP visualizations revealed pronounced batch effects prior to correction, manifesting as distribution shifts among samples. After correction, batch mixing was substantially improved by scFormer and Harmony, whereas weaker correction was observed for the remaining baselines in this dataset (Fig. 4a). Quantitative metrics further supported improved batch effect mitigation by scFormer, with a higher iLISI score (2.695 vs. 2.191 for Harmony) and a higher batch entropy (1.085 vs. 0.842 for Harmony) (Fig. 4b and Supplementary Fig. 3a).

We then examined how integration influenced rare-cell recovery. Rare-cell detection methods generally do not account for batch effects, and batch-associated structure can therefore be mistaken for biological rarity in multi-sample analyses. Conversely, workflows that first correct batch effects and then cluster cells may attenuate subtle transcriptional differences that are essential for identifying rare populations.

Based on these considerations, scFormer was compared with two alternatives: (i) scCAD applied in its standard configuration without additional batch effect correction, and (ii) a two-stage pipeline in which Harmony was first used for batch effect correction, followed by Leiden clustering, with clusters comprising ≤ 5% of cells designated as rare. Under this predefined Harmony-Leiden setting, using rare populations defined from the original study annotations as reference labels, scFormer achieved an F1 score of 0.675 (precision 0.977; recall 0.516), higher than both the Harmony-Leiden pipeline (F1 0.510) and scCAD (F1 0.190) (Fig. 4c). To make the parameter dependence of this two-stage baseline explicit, we additionally evaluated Harmony-Leiden across Leiden resolutions from 0.2 to 3.0, with the complete results provided in Supplementary Table 2.

To further dissect the contribution of the batch-correction loss, we performed an ablation analysis in which the two components described in Methods were evaluated separately: the batch-level distribution-alignment component and the nearest-neighbor local-structure component. Rare-cell F1 remained stable in the no-batch, distribution-component-only, and local-structure-component-only settings (0.663, 0.662, and 0.663, respectively), whereas the full model achieved the highest F1 score (0.675). In contrast, batch-mixing metrics improved substantially after adding batch-correction components: iLISI increased from 1.176 in the no-batch setting to 2.120, 2.445, and 2.695 in the distribution-component-only, local-structure-component-only, and full-model settings, respectively; batch entropy increased from 0.167 to 0.819, 0.953, and 1.085, respectively. These results indicate that the batch-correction loss improved batch mixing without reducing rare-cell recovery in this PBMC benchmark, and that the local-structure component made an important contribution to integration performance (Supplementary Fig. 4).

Closer examination suggests two distinct failure modes in the competing methods. scCAD, which lacks explicit batch effect correction, was sensitive to batch-associated variation and classified plasmacytoid dendritic cells (pDCs) as a rare cluster under the predicted-cluster abundance criterion (cluster 46), even though pDCs did not meet the rarity criterion in this dataset (Supplementary Fig. 3b). This pattern is consistent with technical variation being confounded with biological rarity when integration is omitted. In contrast, although the Harmony-Leiden workflow effectively reduced batch effects, the sequential “correct-then-cluster” strategy obscured biologically meaningful structure: Harmony merged hematopoietic stem cells and transitional B cells with lymphoid progenitors into a single artifactual cluster (Fig. 4d) and assigned an abundant CD4 + T cell population (17.43%) to a predicted rare cluster under the evaluation criterion (Supplementary Fig. 3b). The resolution sweep further showed that Harmony-Leiden results varied with the Leiden resolution parameter: lower resolutions produced fewer predicted rare cells with higher precision but lower recall, whereas higher resolutions increased the number of predicted rare-cell calls and reduced precision. These results provide a transparent assessment of the parameter sensitivity of the Harmony-Leiden baseline (Supplementary Table 2).

By jointly optimizing batch correction and rare-cell detection within a single framework, scFormer resolved five biologically plausible rare populations: B1 B cells (1.85%), proerythroblasts (1.42%), transitional B cells (0.89%), lymphoid progenitors (1.46%), and plasma cells (1.51%) (Fig. 4d). Together, these results indicate that an integrated approach mitigates false positives arising from batch effects while preserving fine-grained biological heterogeneity that may be diminished by sequential pipelines.

scFormer reveals obscured rare epithelial states in the mouse airway

To further evaluate rare-cell detection performance, we applied scFormer and the next-best comparator, scCAD, to a single-cell RNA-sequencing dataset of mouse airway epithelial cells (Montoro et al. 2018). The dataset comprises 7,193 cells and 16,401 detected genes, with the original study annotating seven cell types, including two established rare populations: Foxi1 + lung ionocytes and goblet cells (Supplementary Fig. 5a).

scFormer resolved 17 clusters (Supplementary Fig. 5b), six of which had abundance ≤ 5% (clusters 11–16; Fig. 5b). Cluster identities were assigned by computing Jaccard similarity between scFormer-defined clusters and annotated cell types, using marker genes reported in the original study together with the top 50 markers per cluster. Clusters 12, 13, 15 and 16 showed the highest similarity to tuft, neuroendocrine, goblet and Foxi1 + ionocyte annotations, respectively (Supplementary Fig. 5c; Supplementary Table 4). Consistent marker expression further supported the recovery of rare populations reported in the original study.

Fig. 5.

scFormer identifies known rare populations and candidate proliferative epithelial rare states in the mouse airway epithelium. a-c UMAP projections highlighting the distribution of rare-cell clusters (clusters comprising < 5% of cells), as defined by the original annotations (a), scFormer (b), and scCAD (c). Cluster 14, which is uniquely detected by scFormer, is indicated by a red circle. d-f UMAP feature plots showing normalized expression of key marker genes: the basal cell marker Krt5 (d), the proliferation-associated gene Tubb5 (e), and the club cell marker Scgb1a1 (f). The color scale in (d-f) represents the level of gene expression, from low (gray) to high (midnight blue). g, h Density plots comparing cell-cycle scores between the scFormer-identified club-cell-associated rare cluster (cluster 14, red) and the remaining club cells (blue). g Distribution of S phase scores, showing modest right shift for cluster 14 (median = 0.00 vs. −0.01; p = 2.00 × 10⁻², two-sided Mann–Whitney U test; Cliff’s δ = 0.17). h Distribution of G2/M phase scores, showing a pronounced right shift for cluster 14 (median = −0.04 vs. −0.06; p = 2.33 × 10⁻⁹, two-sided Mann–Whitney U test; Cliff’s δ = 0.43). Shaded areas denote the region of overlap between distributions; dashed lines indicate group medians

Notably, both scFormer and scCAD identified cluster 11 as rare (Fig. 5a-c). However, the original annotations suggested that this cluster likely represents a heterogeneous admixture of basal and club cells. Jaccard analysis revealed minimal similarity to basal cells (J = 0.009) or tuft cells (J = 0.007), indicating that its biological identity resists simple classification (Supplementary Fig. 5c). Further analysis confirmed significant overexpression of basal cell markers (Krt5, Trp63) combined with specific upregulation of proliferation-associated genes (Tubb5, Hmgb2, H2afv) in cluster 11 (Fig. 5d-e and Supplementary Dataset 3). Given the involvement of these genes in cell-cycle control, DNA replication and mitosis (Hewitt et al. 2023; Rock et al. 2009; Saillour et al. 2014; Jiang et al. 2024; Tang et al. 2020), cluster 11 may represent a basal-cell-associated rare transcriptional state with proliferative features, rather than a fully resolved rare basal-cell type. Thus, scFormer not only recovered clusters concordant with manually annotated rare populations but also highlighted candidate rare states associated with distinct biological programs, including active proliferation.

Rare clusters are often characterized by both low abundance and weak transcriptional signatures that are easily masked by dominant populations. In this dataset, scFormer detected cluster 14 (0.92%) within the club-cell lineage, which was not identified by scCAD (Fig. 5b-c). Its club-cell identity was supported by the highest Jaccard similarity to annotated club cells and concordant expression of canonical markers such as Scgb1a1 (Martinu et al. 2023) (Fig. 5f, Supplementary Dataset 3). To further characterize this population, we performed cell-cycle profiling using a murine gene set (v2021) from the GitHub repository hbc/tinyatlas. Cluster 14 exhibited significantly elevated cell-cycle activity relative to other club cells (Mann–Whitney U test, two-sided; S phase: p = 2.00 × 10^–2, Cliff’s δ = 0.17; G2/M phase: p = 2.33 × 10^–9, Cliff’s δ = 0.43), consistent with proliferative features (Fig. 5g-h). Whereas increases in S-phase activity were modest, the pronounced G2/M shift suggested enrichment of G2/M-associated transcriptional programs in this cluster. These results support cluster 14 as a rare club-cell-associated transcriptional state with proliferative features, while motivating additional controls to assess whether its recovery was driven primarily by cell-cycle-associated genes.

To further examine whether these rare airway clusters were driven primarily by cell-cycle-associated genes, we reran scFormer after removing 94 canonical murine S/G2M genes detected in this dataset, while keeping the same preprocessing, clustering, and model parameters. The club-cell-associated cluster remained detectable after this control: its best-matched cluster contained 44 cells, all of which originated from the original cluster 14 (Jaccard = 0.667). These cells retained uniform expression of club-cell markers, including Scgb1a1, Scgb3a1, Scgb3a2, and Cyp2f2, and showed no detectable Mki67 or Top2a expression. The basal-cell-associated cluster also retained a high-purity matched cluster after cell-cycle gene removal, with 84 of 87 cells originating from the original cluster 11 (Jaccard = 0.522), and retained expression of basal markers including Krt5 and Trp63. These results indicate that the rare airway clusters were not solely attributable to S/G2M gene-driven clustering, although the partial recovery of cluster 11 supports a more conservative interpretation of this population as a basal-cell-associated rare transcriptional state with proliferative features (Supplementary Fig. 6).

scFormer recovers revival stem cells and resolves injury-associated rare immune subsets in mouse intestinal crypts

To assess whether scFormer can identify rare populations without manual guidance, we analyzed a mouse intestinal scRNA-seq dataset comprising 6,644 single-cell transcriptomes from intestinal crypts collected during regeneration after epithelial injury. Intestinal homeostasis depends on LGR5 + crypt-base columnar cells (CBCs), which are depleted following damage and transiently replaced by rare, clusterin (Clu)-expressing revival stem cells (revSCs) that drive regeneration (Ayyaz et al. 2019; Barker et al. 2007; Metcalfe et al. 2014; Tao et al. 2015).

scFormer partitioned the dataset into 19 clusters and recovered six rare populations (≤ 5% frequency), three of which represented < 1% of all cells (Fig. 6a-b). Comparison with the major lineages reported in the original study revealed overall concordance (Fig. 6c and Supplementary Table 5). Clusters 13 and 15 mapped to enterocytes (Jaccard similarity > 0.34), cluster 16 mapped to tuft cells (Jaccard = 0.40), and cluster 14 mapped to the dominant lymphocyte population (Jaccard = 0.27) (Fig. 6c and Supplementary Table 5). Notably, revSCs were recovered as cluster 8 (Fig. 6c; Supplementary Table 5), which was enriched in irradiated samples and exhibited strong expression of canonical revSC markers, including Clu and Anxa1 (Fig. 6d-e; Supplementary Dataset 4). These results indicate that scFormer reconstructs established intestinal architecture and recovers a rare regenerative population without relying on predefined labels.

Fig. 6.

scFormer uncovers rare-cell clusters in unannotated mouse intestinal epithelium. a UMAP projection of 6,644 mouse intestinal epithelium cells, colored by the 19 clusters identified by scFormer. b The same UMAP embedding as in (a), with the six rare clusters highlighted. c Dot plot showing the expression of canonical lineage markers used for cell-type annotation. Dot size denotes the percentage of cells within each cluster expressing a given gene, and color intensity reflects mean expression (from low to high). Cell populations are annotated as: Enterocytes (EC), Enteroendocrine cells (EE), Goblet cells (GC), Paneth cells (PC), Tuft cells (TC), Crypt Base Columnar cells (CBC), reviving Stem Cells (revSC), Proliferating cells (Pro.), and Immune cells (IM). d-i UMAP feature plots displaying the normalized expression of representative markers: Clu, Anxa1, Ebf1, Cd79a, Siglech and Pld4. The color scale in (d-i) represents the level of gene expression, from low (gray) to high (midnight blue)

Beyond previously reported populations, scFormer further resolved two low-frequency immune clusters that were not emphasized in the original analysis. Both were detected almost exclusively in non-irradiated samples and were markedly diminished after injury (Supplementary Fig. 7a-b), suggesting state-dependent immune composition at homeostasis. Although broadly classified as lymphocytes, their Jaccard similarity to the main lymphocyte population was extremely low (< 0.05), indicating distinct, previously uncharacterized subtypes (Supplementary Table 5).

The first subset, Cluster 17, represented a B-cell subpopulation expressing Ebf1, Cd79a, and MHC-II molecules (H2-Aa, H2-Ab1) (Fig. 6f-g), consistent with a role in immune surveillance and tolerance to commensal microbiota, with its loss after injury potentially reflecting radiation sensitivity or inflammatory redistribution (Fedl et al. 2024; Davis et al. 2010; Chen et al. 2018; Suzuki et al. 2023). The second subset, Cluster 18 was identified as a plasmacytoid dendritic cell (pDC) subset marked by Siglech, Pld4, and Irf8 (Fig. 6h-i). Given their capacity for type I interferon production, their disappearance during injury may help prevent excessive inflammation that could hinder epithelial repair (Ascic et al. 2024; Gavin et al. 2021; Valente et al. 2023). Collectively, these findings highlight scFormer’s ability to uncover functionally distinct rare subsets and provide new insights into the immune regulation of the intestinal niche.

scFormer enables rare-cell identification in spatial transcriptomics

We next evaluated scFormer in a spatial transcriptomics setting using the sci-Space dataset, which comprises 52,636 spatially resolved gene-expression profiles derived from 122,278 single-cell transcriptomes from E14 mouse embryonic sagittal sections (Srivatsan et al. 2021). Using gene expression profiles as input, scFormer identified 29 transcriptionally distinct populations (Supplementary Fig. 8a). For detailed evaluation, we focused on a representative tissue section, slide_4E (16,222 cells). Within this section, scFormer identified 25 clusters (Fig. 7a), and these clusters were then examined in their original spatial coordinates. The spatial distribution of these clusters was broadly consistent with the reference anatomical annotations (Fig. 7b). Specifically, scFormer recovered anatomically widespread stromal populations, such as fibroblasts (Fstl1 +, Supplementary Fig. 9a), while also distinguishing organ-associated parenchymal populations, including cardiac (Myh7 +) (Gacita et al. 2021), hepatic (Alb +) (Hung et al. 2022), and spinal cord (Plxna2 +) (Sekine et al. 2019) clusters, each showing localization consistent with the expected anatomical domains (Supplementary Fig. 9b-d). To quantitatively evaluate spatial organization beyond visual inspection, we performed three post hoc analyses using the original tissue coordinates and published sci-Space annotations: k-nearest-neighbor spatial self-enrichment, hypergeometric enrichment against published annotations, and marker/module expression enrichment. These analyses are summarized in Supplementary Fig. 11 and Supplementary Datasets 5–7.

Fig. 7.

scFormer identifies rare cell states in spatial transcriptomic data. a UMAP projection of 16,222 cells from an E14 mouse embryonic sagittal section (slide_4E), colored by the 25 clusters identified by scFormer. Slide_4E is shown because it yields the highest recovery of nuclei. b Left, spatial map of the embryonic section, with each spot colored by its dominant cluster as determined by scFormer. Right, corresponding anatomical annotations of major structures, including cortex, heart, liver, lung, and neural tube. c Spatial distribution of clusters 8, 20 and 21. These clusters show spatially organized distributions consistent with hepatic and pancreatic territories. Color intensity reflects local cell density. d UMAP feature plots showing expression patterns of representative marker genes. The top row displays the hepatocyte marker Alb and pancreatic exocrine markers Ctrb1 and Cpa1. The bottom row displays biliary epithelial markers Cftr, Krt7 and Sox9. Color intensity reflects expression level from low (gray) to high (midnight blue). e Spatial visualization of cells co-expressing Cftr, Krt7 and Sox9 (red), showing the anatomical distribution of the biliary epithelial-marker-positive subset. f–h Spatial distribution (top) and corresponding marker expression (bottom) for three neurodevelopmental populations: cluster 7 (forebrain lineage, Foxg1 +) (f), cluster 14 (radial glia, Slc1a3 +) (g) and cluster 15 (neural stem/progenitor cells, Sox2 +) (h)

These analyses further show that scFormer can identify rare cell states in spatial transcriptomic data and allow their anatomical distributions to be examined after mapping cluster labels back to the original tissue coordinates. In the hepatic region of slide_4E, scFormer distinguished the abundant hepatocyte population (Cluster 8, Alb +) from a rare biliary epithelial-like cell state (Cluster 20; 0.92%, n = 149). After projection back to the original tissue coordinates, Cluster 20 showed significant spatial self-enrichment relative to the permutation background (0.326 observed versus 0.009 permutation background; 35.74-fold enrichment; BH-adjusted empirical q = 1.14 × 10⁻⁴) and showed a distribution consistent with developing biliary structures (Fig. 7c). Consistent with its hepatic location, Cluster 20 was enriched for the published Liver anatomical annotation (57/149 cells; 6.02-fold enrichment; BH-adjusted q = 1.94 × 10⁻²⁹). A subset of cells in Cluster 20 co-expressed Cftr, Krt7, and Sox9 (14.8% of Cluster 20 cells; 4.15-fold higher mean module expression than other cells), a multigene signature characteristic of biliary epithelium (Amarachintha et al. 2022; Liu et al. 2022; Rimland et al. 2021), supporting its annotation as a biliary epithelial-like cell state. Spatial visualization of cells co-expressing these markers further showed that this marker-positive subset was concentrated within the hepatic region (Fig. 7e). Detailed spatial enrichment, annotation enrichment, and marker-module statistics for Cluster 20 are summarized in Supplementary Fig. 11 and Supplementary Datasets 5–7. These findings indicate that rare cell states identified by scFormer from spatial transcriptomic expression profiles can be examined in their original anatomical context.

scFormer also identified an additional rare cell state. Cluster 21 represented a rare subset (0.52%, n = 84) without the Cftr/Krt7/Sox9 co-expression pattern used for Cluster 20 but exhibiting enrichment of pancreatic marker genes Ctrb1, Cpa1, and Clps (Bodas et al. 2025; Tamura et al. 2018; Zhang et al. 2013), as shown in the UMAP feature plots and supplementary marker visualizations (Fig. 7d and Supplementary Fig. 10a-d). Spatial mapping showed that these cells could be examined in their original tissue coordinates, with a distribution shown in Fig. 7c. Cluster 21 also showed strong spatial self-enrichment relative to the permutation background (0.461 observed versus 0.005 permutation background; 89.52-fold enrichment; BH-adjusted empirical q = 1.14 × 10⁻⁴). Annotation-level comparison with published sci-Space labels showed that Cluster 21 was enriched in pancreatic-associated annotation categories, including Pancreatic Islet Cells in manual_annotation_2 (61/84 cells; 193.12-fold enrichment; BH-adjusted q = 2.36 × 10⁻¹⁵²) and Epithelial Cells in final_cluster_label (61/84 cells; 9.42-fold enrichment; BH-adjusted q = 9.62 × 10⁻⁴⁹). Consistent with the marker visualizations, Ctrb1/Cpa1 module expression was enriched in Cluster 21, with 56.0% of Cluster 21 cells co-expressing both markers. Detailed spatial enrichment, annotation enrichment, and marker-module statistics for Cluster 21 are summarized in Supplementary Fig. 11 and Supplementary Datasets 5–7. These results support the description of Cluster 21 as a pancreatic marker-enriched rare cell state that can be examined within the original tissue coordinates.

scFormer further delineated rare neurodevelopmental populations. Cluster 7 (5.31%, n = 862) showed high Foxg1 expression, consistent with a telencephalic lineage, and its spatial distribution overlapped forebrain cortical regions (Fig. 7f) (Mariani et al. 2015). Cluster 14 (2.82%, n = 458) expressed the glial marker Slc1a3 together with Notch-pathway components such as Hes5, mapping to the ventricular zone characteristic of radial glia (Fig. 7g) (Bifari et al. 2017; Ruan et al. 2021). Cluster 15 (2.22%, n = 360) showed high Sox2 expression, consistent with neural stem and progenitor identity (Fig. 7h) (Jung et al. 2020). Consistent with these visual patterns, clusters 7, 14, and 15 showed 13.25- to 22.57-fold spatial self-enrichment over permutation background, and their published annotation and marker-enrichment results are summarized in Supplementary Fig. 11 and Supplementary Datasets 5–7. Collectively, these results support the use of scFormer for identifying rare populations in spatial transcriptomic data and examining their anatomical distributions within embryonic tissue sections.

Discussion

In this study, we introduce scFormer, a HGT-based computational framework designed to address the dual challenges of identifying rare cells in scRNA-seq data while simultaneously mitigating batch effects in multi-sample analyses. Through a specificity-driven cell-gene heterogeneous graph and an optional batch-correction module, scFormer achieves high accuracy, robustness and practical reliability in rare-cell detection.

scFormer was rigorously evaluated across 125 simulated and 18 real scRNA-seq datasets, where it consistently outperformed existing approaches. In simulations, scFormer achieved the highest F1 score (0.988) and maintained stable performance across both high- and low-separation scenarios. By contrast, methods such as RaceID3, FiRE, and GiniClust3 exhibited substantial performance degradation under low-separation conditions, reflecting limited ability to detect rare cells with weak transcriptional signatures.

In real datasets, scFormer achieved a mean F1 score of 0.595, representing a 22.1% and 93.8% improvement over scCAD (0.488) and GiniClust3 (0.307), respectively. For example, in a mouse retina dataset, scFormer reached an F1 score of 0.955, markedly exceeding scCAD (0.649) and GiniClust3 (0.282). These gains are consistent with the model design. First, the construction of specificity-driven cell-gene heterogeneous graphs captures rare transcriptional programs that are often under-represented in cell–cell graphs that rely primarily on homophily. Second, the heterogeneous graph transformer architecture explicitly models distinct biological relationships, preserving subtle rare-cell signals within high-dimensional data. Mechanistically, this provides a direct rationale for why rare clusters can still be resolved when their same-state cellular neighbors are limited.

Recent transformer-based graph frameworks have highlighted that joint analysis of RNA- and ATAC-seq data can improve the resolution of rare cell states in cancer, as exemplified by MarsGT (Wang et al. 2024). In a matched lymphoma dataset from 10 × Genomics, MarsGT reported that B-cell subpopulations could not be effectively distinguished when using RNA-only or ATAC-only data, whereas joint modeling recovered a rare B-lymphoma precursor-like state (BLS1) within B cells. Notably, scFormer recovered an analogous rare B-cell state using the RNA-seq data alone (cluster 13; Supplementary Fig. 12), suggesting that a specificity-driven cell-gene topology can amplify weak transcriptomic signatures even without additional modality. Given the technical challenges and higher costs associated with acquiring paired RNA-seq and ATAC-seq data, scFormer demonstrates broader applicability as an RNA-first framework. It serves not merely as a baseline, but as a robust foundation for future multi-omics extensions.

The practical value of scFormer is further enhanced by its batch-correction capability. By combining global distribution alignment with local structure preservation, scFormer mitigates batch effects while retaining rare-cell signatures. This design avoids relying on batch-level distribution alignment alone: rare-cell information is first represented through the specificity-driven cell-gene graph, while the local-structure component constrains the embedding so that fine-grained neighborhood relationships are maintained during integration. In a PBMC dataset, scFormer achieved an iLISI score of 2.695, outperforming Harmony (2.191), demonstrating effective removal of technical artifacts without obscuring biologically relevant variation. The component ablation further showed that separating the distribution-alignment and local-structure components maintained rare-cell F1 while improving batch mixing, and that the full model achieved the strongest overall integration metrics. These results support the view that local structure preservation helps balance batch alignment with rare-cell recovery.

The wide applicability of scFormer was further validated across diverse biological contexts. In the mouse airway epithelial cell dataset, scFormer recovered known rare types, including Foxi1 + lung ionocytes and goblet cells, while highlighting basal-cell-associated (cluster 11) and club-cell-associated (cluster 14) rare transcriptional states with proliferative features. These rare airway clusters were further supported by the S/G2M gene-removal control, indicating that they were not solely attributable to cell-cycle-gene-driven clustering. In a large-scale human T-cell dataset, scFormer detected proliferating cells, plasmacytoid dendritic cells (pDCs), platelets, and hematopoietic stem and progenitor cells (HSPCs), and further identified a monocyte-derived macrophage subpopulation with M2-like polarization characteristics (cluster 19). In the unannotated mouse intestinal epithelium dataset, scFormer recovered revival stem cells (revSCs), and uncovered two rare lymphocyte subpopulations (B cells and pDCs) absent from the original analysis. These subpopulations, exclusive to non-irradiated samples, suggest specialized roles in immune homeostasis, illustrating the potential of scFormer for discovering functionally relevant rare cells without prior annotations. In spatial transcriptomics, scFormer identified 29 clusters, several of which showed anatomical consistency when examined in the original tissue coordinates. This spatial organization was further supported by k-nearest-neighbor spatial self-enrichment analysis, enrichment against published sci-Space annotations, and marker/module expression quantification, including support for biliary epithelial-like and pancreatic marker-enriched rare cell states. Together, these results indicate that the scFormer formulation is not restricted to a single experimental setting and can be applied to both conventional scRNA-seq and spatially resolved transcriptomic profiles.

A further indicator of robustness is that scFormer successfully executed on all 18 real datasets (100% success rate), whereas every competing method failed on at least one dataset (Supplementary Dataset 2). This reliability underscores its stability and generalizability for rare-cell identification. Nevertheless, several directions remain. First, although scFormer is designed as an RNA-first framework, extending the specificity-driven graph to incorporate additional modalities or emerging biomolecular detection technologies, such as chromatin accessibility and nanopore-based biomolecule detection, may enhance rare-state resolution and enable more mechanistic interpretation (Li et al. 2024), while transcriptome-scale perturbation strategies such as CRISPR-Cas13 screening may help functionally validate candidate rare-cell programs (Si et al. 2025). Second, although the K-sensitivity analysis supports K = 20 as a practical default in simulated rare-cell benchmarks, broader evaluation of graph sparsity and other hyperparameters across additional tissues, sequencing depths and spatial platforms, as well as future adaptive gene-selection strategies based on dataset-specific specificity thresholds, may further refine dataset-specific defaults for routine use.

In conclusion, the performance of scFormer derives from its specificity-aware graph architecture and integrated batch-correction strategy, which together address central limitations of existing approaches. By prioritizing biologically meaningful signals while controlling technical noise, scFormer provides a reliable and versatile framework for discovery in single-cell datasets.

Authors’ contributions

J.H., X.F. and L.Z. conceptualized the study. J.H. developed the methodology and software. J.H. and X.X. performed the formal analysis and validation. L.Q. and L.Z. contributed resources. J.H. wrote the original draft. X.F., F.L. and L.Z. reviewed and edited the manuscript. X.F. and L.Z. supervised the project.

Funding

This research was funded by the National Key R&D Program of China (2022YFC3400401), the National Natural Science Foundation of China (32270604, 32470599), the Guangdong Basic and Applied Basic Research Foundation (2026A1515010528), the Shenzhen Natural Science Foundation in Basic Research Fund (JCYJ20250604175316022), and research funding from the Undergraduate Innovation and Entrepreneurship Training Program of the School of Life Sciences, Sun Yat-sen University.

Data availability

Details of the datasets used in this manuscript are provided in Supplementary Table 3. All described datasets were obtained from various public websites using the accession codes provided in Supplementary Table 3, including the NCBI Gene Expression Omnibus (GEO) [https://www.ncbi.nlm.nih.gov/geo/]. The 10 × PBMC dataset is obtained from GitHub [https://github.com/ttgump/scDeepCluster/blob/master/scRNA-seq%20data/10X_PBMC.h5]. The preprocessed human tonsil data are available from the Broad Institute Single Cell Portal [https://singlecell.broadinstitute.org/single_cell/study/SCP2169/slide-tags-snrna-seq-onhuman-tonsil].

scFormer is publicly available at GitHub [https://github.com/StickTaTa/ScFormer].

Declarations

Ethics approval and consent to participate

Not applicable.

Consent for publication

The work described has not been published before and is not under consideration for publication elsewhere. All co-authors have approved the final version.

Competing interests

The authors declare no competing interests.