Differentiable graph clustering with structural grouping for single-cell RNA-seq data

Abstract

Motivation

Clustering cells into subpopulations is one of the most crucial tasks in single-cell RNA sequencing (scRNA-seq) data analysis, which provides support for biological research at cellular level. With the development of graph neural networks, deep graph clustering approaches have achieved excellent performance by modeling the topological relationships between cells. However, existing approaches rely on cell node and its neighbors to obtain the cell feature representation, which ignore the graph cluster structure hidden in scRNA-seq data. Besides, how to bridge the heterogeneous gap between cell node feature and its structural information remains a highly challenging problem.

Results

Here, we propose a novel differentiable graph clustering with structural grouping (DGCSG) for scRNA-seq data, which incorporates graph cluster information into deep graph clustering model by designing a differentiable clustering mechanism to learn clustering-friendly representation. Firstly, an interactive module is devised to dynamically transfer node representations learned by autoencoder (AE) to graph attention autoencoder (GATE) in layer-by-layer manner. Then, to characterize graph cluster information, a differentiable clustering mechanism is proposed to transform K-way normalized cuts from a discrete optimization problem into differentiable learning objective through spectral relaxation, which jointly optimizes the GATE by allocating more attention scores to nodes in the same graph cluster. Finally, a decoupled self-supervised optimization is proposed, which guides the representation learning of AE and GATE in the interactive module. Extensive evaluations on 14 scRNA-seq benchmarks verify the superiority of DGCSG compared with state-of-the-art baselines.

Availability and implementation

The code associated with this work is available on GitHub (https://github.com/Xiaoqiang-Yan/DGCSG).

1 Introduction

Driven by innovations in high-throughput sequencing technology, a massive amount of single-cell RNA-sequencing (scRNA-seq) data is generated, which provides support for measuring transcriptional gene expression at cellular resolution. Based on the high-throughput sequencing information, cell type discovery has become the basis for various complex biological tasks, such as drug discovery (Tan et al. 2024), disease diagnosis (Khan et al. 2024), and cancer research (Marot-Lassauzaie et al. 2024, Tirosh and Suva 2024). However, technical and biological factors cause scRNA-seq data to present complex characteristics such as high dimensionality, significant sparsity, and substantial noise (Angerer et al. 2017). These characteristics make it challenging to accurately discover cell types from large-scale data.

To address the aforementioned difficulties of cell type discovery, various traditional clustering approaches have been proposed, which aim to partition unlabeled scRNA-seq data into biologically meaningful cell clusters according to gene expression patterns. For example, Wang et al. (2017) learned cell similarity measure through multikernel learning. Kiselev et al. (2017) and Žurauskienė and Yau (2016) projected high-dimensional gene expression data to a low-dimensional feature space based on principal component analysis. Xu et al. (2023) provided a single-cell clustering tool to identify important genes through random forests. However, these traditional clustering approaches encounter challenges in precisely modeling the nonlinear characteristics of scRNA-seq data since they leverage objective functions based on linear embedding.

In recent years, lots of deep scRNA-seq clustering approaches have been designed to learn discriminative representation from scRNA-seq data, in which one of the most popular deep clustering frameworks is autoencoders (AEs). For example, as a pioneer work, Eraslan et al. (2019) designed a deep count autoencoder (DCA) model, which adapts a typical AE to denoise scRNA-seq data. Inspired by DCA, Tian et al. (2019) developed a scDeepCluster model to learn cell features through zero-inflated negative binomial (ZINB) loss. Li et al. (2020) utilized stacked AE to iteratively learn cell representations and cluster assignments for scRNA-seq data. Besides, some methods incorporate AE with traditional clustering models together to enhance the scRNA-seq clustering performance. For example, Chen et al. (2020) characterized scRNA-seq data by utilizing a denoising AE, and then clustered the learned cell representations with a traditional K-means algorithm. Recently, contrastive learning-based methods are proposed to learn cell representations by constructing positive and negative sample pairs, which minimize the similarity of negative pairs while maximizing that of positive pairs in the representation space. For example, Ciortan and Defrance (2021) employed a self-supervised contrastive learning framework to learn the augmented data representation and then clustered the representations learned by the model. Wan et al. (2022) combined a mask estimation task with a contrastive learning framework to facilitate AE learning clustering-friendly representations. However, these aforementioned AE-based clustering approaches and their variants ignore the typological information between different cells, thereby hardly capturing cell–cell relationships.

To leverage cellular typological information and improve clustering accuracy, some researchers resort graph neural networks (GNNs) to model the relationships among cells. GNN-based clustering approaches first construct cell–cell graph and then learn cell representations from the whole graph. For example, Wang et al. (2021) utilized GNN to aggregate cell–cell relationships and reduce the impact of excessive dropout events on downstream analysis. Liu et al. (2024a) designed a dual-channel network including AE and GNN to fuse node and structural features. Yu et al. (2022) learned latent representations by combining ZINB loss and topologically adaptive graph convolutional neural network. Although existing GNN-based approaches have made significant progress in scRNA-seq clustering, they still have the following limitations: (i) The high dimensionality, significant sparsity and substantial noise of scRNA-seq data pose challenges to constructing a reliable graph and accurately characterizing cell relationships. In other words, inaccurate structural information undermines clustering performance because the information-sharing pattern is disrupted. (ii) GNNs often convey information among adjacent cells anonymously. Given the inherent noise in scRNA-seq data, numerous edges in the constructed graph are misconnected. Therefore, it is necessary to employ an attention mechanism to accurately capture graph structure information.

The attention mechanism calculates the importance of node relationships through a learnable attention score, assigning different weights to the links between nodes in the constructed graph, thus capturing the relationships between nodes more accurately. For example, Cheng and Ma (2022) achieved promising results on scRNA-seq clustering by designing a graph attention network. However, existing models based on attention mechanism still have the following shortcomings: (i) in cell–cell graphs, there often exists the heterogeneous gap between the feature representation of cellular node and their structural information, which might interfere clustering performance on scRNA-seq data (the difference between the heterogeneous gap and cellular heterogeneity is illustrated in Fig. 1, available as supplementary data at Bioinformatics online). (ii) Existing approaches rely on node feature and its neighbors to obtain the attention scores, which ignore the structural groupings (also known as graph clusters) for characterizing the topological structure hidden in scRNA-seq data.

In this paper, we propose a novel differentiable graph clustering with structural grouping (DGCSG), which incorporates graph cluster information into attention mechanism of graph attention autoencoder (GATE) by devising a differentiable graph clustering objective. Specifically, we first construct a cell–cell graph by applying Pearson correlation analysis and network-enhanced denoising (Wang et al. 2018) to the scRNA-seq data after quality control. Subsequently, to address the heterogeneous gap between cell node and their structural information, we build an interactive module based on AE and GATE, which dynamically fuses the cell node representations learned by AE into GATE in layer-by-layer manner. Thus, the structural features captured by GATE can be promoted by the node features of AE. Afterward, to incorporate graph cluster into deep graph clustering, we design a differentiable clustering mechanism to transform K-way normalized cuts from a discrete optimization problem into a differentiable learning objective through spectral relaxation. This differentiable clustering objective can be jointly optimized with GATE to allocate more attention scores to cell nodes in the same graph cluster, providing better cell representation for downstream clustering. Finally, a decoupled self-supervised optimization is proposed, which guides the representation learning of AE and GATE in the interactive module through two high-confidence distributions. Our contributions can be summarized as follows:

• To characterize graph cluster information of cell–cell graph, a novel differentiable clustering mechanism is proposed to transform K-way normalized cuts from a discrete optimization problem into a differentiable learning objective through spectral relaxation.

• A layer-by-layer interactive module based on AE and GATE is devised to deal with the heterogeneous gap between cell node and structural information, while a decoupled self-supervised optimization is designed to enhance the representation of scRNA-seq data by the module.

• Experimental evaluations on 14 scRNA-seq benchmarks validates that the proposed method significantly outperforms existing state-of-the-art baselines.

2 Materials and methods

Suppose we have access to unlabeled scRNA-seq data $\mathbf{X}=\left\{x_{ij}\right\}\in {\mathbb{R}}^{N\times M}$, where N and M are the number of cell and its gene expression count, $x_{ij}$ indicates the expression value of the jth gene in the ith cell. To capture the topological information of cells, we construct a cell–cell graph $G=(\mathbf{X},\mathbf{A})$ with Pearson coefficient matrix, in which the degree matrix $\mathbf{D}=\text{diag}(d_1,d_2,\dots ,d_N)$ can be calculated by $D_{ii}={\sum }_j{(\mathbf{A})}_{ij}$, where $\mathbf{A}$ is the adjacency matrix. The proposed DGCSG aims to allocate scRNA-seq data into different subpopulations by characterizing cell node feature and its structural information derived by graph clusters simultaneously. The model architecture of DGCSG is shown in Fig. 1, which consists of the following components: data preprocessing, interactive module, differentiable graph clustering and decoupled self-supervised optimization. Specifically, in data preprocessing, we construct initialized cell–cell graph $G=(\mathbf{X},\mathbf{A})$ with Pearson coefficient matrix between cells, then perform network enhancement to remove noisy edges of the graph. Note 1, available as supplementary data at Bioinformatics online, shows the construction process of the cell–cell graph. In differentiable graph clustering, to incorporate graph cluster information into deep graph clustering framework, the K-way normalized cuts problem is transformed from a discrete optimization problem into a differentiable learning objective ${\mathcal{L}}_{SG}$ through spectral relaxation. Then, the interactive module is composed of AE and GATE with L layers, in which the node representation $\mathbf{Z}$ of AE is transferred into corresponding layer of GATE. By integrating differentiable graph clustering objective ${\mathcal{L}}_{SG}$, GATE learns a representation $\mathbf{H}$ of cells that captures both node feature and graph cluster simultaneously. Finally, a decoupled self-supervised optimization is proposed, which guides the representation learning of AE and GATE in the interactive module through two high-confidence distributions.

Figure 1.

The model architecture of DGCSG. DGCSG is composed of four modules: (A) data preprocessing, (B) interactive module, (C) differentiable graph clustering, (D) decoupled self-supervised optimization. Firstly, the data preprocessing module generates a gene expression matrix $\mathbf{X}$ and a cell–cell graph $\mathbf{G}$. Next, $\mathbf{X}$ and $\mathbf{G}$ are input to the interactive module for representation learning, and the learned representations are dynamically fused layer by layer. Meanwhile, the differentiable graph clustering module utilizes the cell–cell graph to learn graph cluster information and assists GATE in assigning higher attention scores to nodes within the same cluster through the differentiable loss. Finally, the decoupled self-supervised optimization module is devised to guide the learning of representations $\mathbf{Z}$ and $\mathbf{H}$.

2.1 Differentiable graph clustering with structural grouping

In graph clustering, graph cluster is an important structural information except for the neighbors of cell nodes, which can describe the structural relationships between cells more comprehensively. Thus, it is potential to introduce graph cluster information into self-attention mechanism, which can assign higher attention scores to cell nodes within the same cluster. In the literature, K-way normalized cuts (Shi 2003) method is an effective graph clustering approach, which is capable of dividing the original graph into multiple subgraphs. In this study, we resort K-way normalized cuts to partition cell nodes $\mathbf{X}$ into K disjoint graph clusters, i.e. $V={\cup }_{i=1}^K{C}_i$ and $C_i\cap C_j=\varnothing ,\forall i\ne j$, by maximizing intracluster links and minimizing intercluster links as follows:

$$\frac{1}{K}{\sum }_{k=1}^K\frac{\mathit{links}\left(C_k,C_k\right)}{\mathit{degree}\left(C_k\right)}=\frac{1}{K}{\sum }_{k=1}^K\frac{{\sum }_{i,j\in C_k}{A}_{i,j}}{{\sum }_{i\in C_k,j\in V}{A}_{i,j}},$$ (1)

where $C_i$ is the ith cell cluster, $\mathit{links}\left(C_k,C_k\right)$ is the number of links in cluster $C_k$, $\mathit{degree}\left(C_k\right)$ is the number of links connected to $C_k$. To learn the graph cluster, we use $\mathbf{F}\in {\{0,1\}}^{N\times K}$ to indicate the cluster assignment matrix, where $F_{i,j}=1$ means that the ith cell node is allocated into jth graph cluster, and $F_{i,j}=0$ means that it is not in this graph cluster. Now, the K-way normalized cuts problem [Equation (1)] is rewritten as follows:

$$\begin{array}{c}\max \frac{1}{K}{\sum }_{k=1}^K\frac{F_k^T\mathbf{A}{F}_k}{F_k^T\mathbf{D}{F}_k},\hfill \\ \mathit{subject}\hspace{0.17em}to\hspace{0.17em}\mathbf{F}\in {\{0,1\}}^{N\times K},\mathbf{F}{1}_K=1_N,\hfill \end{array}$$ (2)

where $F_k$ is the kth column of $\mathbf{F}$, $\mathbf{F}{1}_K=1_N$ indicates that each node can only belong to one cluster.

As shown in Equation (2), the cluster assignment matrix $\mathbf{F}$ of the K-way normalized cuts problem is a discrete variable, and the optimization of this variable is NP Hard (Shi and Malik 2000). Moreover, this discrete variable cannot be solved by gradient descent in deep graph clustering model. To solve this problem, we transform the discrete variable optimization problem of K-way normalized cuts into a continuously differentiable form using spectral relaxation method. From Equation (2), we can observe that the objective function of the K-way normalized cuts is a generalized Rayleigh quotient, which can be optimized by transforming it into a spectral decomposition problem of the matrix (Li 2015). Therefore, we first introduce a variable $E_k$ and set $E_k={\mathbf{D}}^{1/2}{F}_k{\left(F_k^T\mathbf{D}{F}_k\right)}^{-1/2}$ to transform Equation (2) into the form of a standard Rayleigh quotient,

$$\begin{array}{c}\max \frac{1}{K}{\sum }_{k=1}^K\frac{E_k^T{\mathbf{D}}^{-1/2}\mathbf{A}{\mathbf{D}}^{-1/2}{E}_k}{E_k^T{E}_k},\hfill \\ \mathit{subject}\hspace{0.17em}to\hspace{0.17em}{E}_k={\mathbf{D}}^{1/2}{F}_k{\left(F_k^T\mathbf{D}{F}_k\right)}^{-1/2},E_k^T{E}_k=1,\hfill \end{array}$$ (3)

where $E_k$ is constrained by $E_k^T{E}_k=1$. Next, we relax the discreteness condition by allowing $E_k$ to take arbitrary real values and treat it as a soft clustering partition. To further simplify the problem, we provide the following proposition to transform the objective function of K-way normalized cuts into the problem of solving the K largest eigenvalues.

Proposition 1.

After relaxing the discreteness condition, the K-way normalized cuts problem can be transformed into the maximization of the average sum of the K eigenvalues by applying the Lagrange multiplier method.

The proof of Proposition 1 is provided in Proof 1, available as supplementary data at Bioinformatics online. By utilizing the above proposition, the objective function of K-way normalized cuts can be written as follows:

$$\max \frac{1}{K}{\sum }_{k=1}^K{\lambda }_k,$$ (4)

where ${\lambda }_k$ is the eigenvalue of the matrix ${\mathbf{D}}^{-1/2}\mathbf{A}{\mathbf{D}}^{-1/2}$ and the corresponding eigenvector is $E_k$. However, it is computationally expensive to directly solve the eigenvalues, especially for matrices with large dimensions. The Fan (1949) theorem states that maximizing the sum of eigenvalues can be achieved simply by finding the corresponding eigenvectors. We resort the Fan theorem to transform the problem into an optimization of a subset of eigenvectors, thereby avoiding the computation of the full eigenvalue spectrum.

Theorem 1

(Fan). Let $\mathbf{B}\in {\mathbf{R}}^{n\times n}$  is a symmetric matrix, the eigenvalues of $\mathbf{B}$  is  ${\lambda }_{\text{max}}={\lambda }_1\ge {\lambda }_2\ge \cdots \ge {\lambda }_{n-1}\ge {\lambda }_n={\lambda }_{\text{min}}$  with corresponding eigenvectors $u_1,u_2,\dots ,u_n$, we can obtain,

$${\lambda }_1+\cdots +{\lambda }_k=\max tr\left({\mathbf{W}}^T\mathbf{B}\mathbf{W}\right),$$ (5)

where ${\mathbf{W}}^{*}=\left[u_1,\dots ,u_k\right]\mathbf{O}$ is the optimal solution, and $\mathbf{O}$ is orthogonal.

 Finally, we apply the Fan theorem to convert Equation (4) into the following form:

$$\begin{array}{c}\max \frac{1}{K}tr\left({\mathbf{E}}^T{\mathbf{D}}^{-1/2}\mathbf{A}{\mathbf{D}}^{-1/2}\mathbf{E}\right)\hfill \\ \mathit{subject}\hspace{0.17em}to\hspace{0.17em}\mathbf{E}={\mathbf{D}}^{1/2}\mathbf{F}{\left({\mathbf{F}}^T\mathbf{D}\mathbf{F}\right)}^{-1/2},{\mathbf{E}}^T\mathbf{E}=\mathbf{I}.\hfill \end{array}$$ (6)

The optimal solution of Equation (6) is the eigenvectors corresponding to the K largest eigenvalues of ${\mathbf{D}}^{-1/2}\mathbf{A}{\mathbf{D}}^{-1/2}$. By relaxing the K-way normalized cuts to a Top-K eigenproblem (Sgherzi et al. 2021), we can transform discrete matrix $\mathbf{F}$ into a continuous matrix $\mathbf{E}$, in which $\mathbf{E}$ can be treated as the soft partition matrix. Now, the continuous matrix can be parameterized using GATE layers.

2.2 Interactive module between cell node and graph structural information

GATE can assign different attention scores to different cell neighbors, which helps to learn cell representations. However, the existing GATE and its variants only consider information about the node and its neighbors to allocate the attention scores. In this study, we design a differentiable graph clustering mechanism to incorporate structural grouping information into the GATE attention learning process. However, the cell node feature is also an indispensable cue in scRNA-seq clustering for characterizing gene expression information. To simultaneously learn the node and structural information of cells, we devise an interactive module based on AE and GATE. Meanwhile, in order to eliminate the heterogeneous gap between cell nodes and their structural information, we dynamically fuse the learned representations in a layer-by-layer manner.

Firstly, we build an AE with L layers to capture the cell node features. The corresponding construction process can be found in Note 2, available as supplementary data at Bioinformatics online. Next, to combine cell node information and structural information driven by graph clusters, we construct a GATE that has the same number of layers as AE. Thus, the representation $h_i^{(l)}$ of cell node $x_i$ in the lth layer of GATE is formulated as follows:

$$h_i^{(l)}=\sigma \left({\sum }_{j\in {\mathcal{N}}_i}at{t}_{i,j}{\mathbf{W}}_{\mathit{gat}}{h}_j^{(l-1)}\right),$$ (7)

where ${\mathcal{N}}_i$ is the neighbors of cell node $x_i$, $\sigma (·)$ is ReLU function, ${\mathbf{W}}_{\mathit{gat}}$ is a learnable parameter. $at{t}_{i,j}$ is the attention score between cell nodes $x_i$ and $x_j$, which can be calculated as follows:

$$at{t}_{i,j}=\frac{\hspace{0.17em}\text{exp}\hspace{0.17em}\left(\mathit{LeakyReLU}\left(a^T\left({\mathbf{W}}_{\mathit{gat}}{h}_i\left|\right|{\mathbf{W}}_{\mathit{gat}}{h}_j\right)\right)\right)}{{\sum }_{k\in {\mathcal{N}}_i}\hspace{0.17em}\text{exp}\hspace{0.17em}\left(\mathit{LeakyReLU}\left(a^T\left({\mathbf{W}}_{\mathit{gat}}{h}_i\left|\right|{\mathbf{W}}_{\mathit{gat}}{h}_j\right)\right)\right)},$$ (8)

where a denotes a vector of weights. $\left|\right|$ represents the concatenation of two vectors. Thus, the node representation learned by AE is transferred layer by layer to corresponding layer in GATE for dynamic fusion in the following form:

$${\mathbf{U}}^{(r)}={\mathbf{Z}}^{(r)}+{\mathbf{H}}^{(r)},$$ (9)

where $r\in \{0,1,\dots ,L\}$ represents the number of encoder layers, and ${\mathbf{Z}}^{(r)}$ and ${\mathbf{H}}^{(r)}$ are the representations of the rth layer in AE and GATE, respectively, with ${\mathbf{Z}}^{(0)}={\mathbf{H}}^{(0)}=\mathbf{X}$. After dynamic fusion with the node representation learned by AE, the learned representation of the lth layer in GATE is expressed in the following form:

$${\mathbf{H}}^{(l)}=\sigma \left(\mathbf{att}·{\mathbf{W}}_{\mathit{gat}}^{(l)}{\mathbf{U}}^{(l-1)}\right).$$ (10)

To incorporate the graph cluster information into GATE, we use $\mathbf{S}=\mathbf{att}·{\mathbf{W}}_{\mathit{gat}}^{(l)}{\mathbf{U}}^{(l-1)}$ to parameterize the graph cluster information $\mathbf{E}$ in Equation (6). Then, the objective function of K-way normalized cuts can be incorporated into GATE as follows:

$$\begin{array}{c}\max \frac{1}{K}tr\left({\mathbf{S}}^T{\mathbf{D}}^{-1/2}\mathbf{A}{\mathbf{D}}^{-1/2}\mathbf{S}\right),\hfill \\ \mathit{subject}\hspace{0.17em}to\hspace{0.17em}{\mathbf{S}}^T\mathbf{S}=\mathbf{I}.\hfill \end{array}$$ (11)

Now, we define the differentiable graph clustering loss for the lth layer and normalize it to obtain as follows:

$${\mathcal{L}}_{SG}^{(l)}={\Vert \frac{1}{N}{\mathbf{S}}^T\mathbf{S}-\mathbf{I}\Vert }_F^2-\frac{1}{NK}tr\left({\mathbf{S}}^T{\mathbf{D}}^{-1/2}\mathbf{A}{\mathbf{D}}^{-1/2}\mathbf{S}\right).$$ (12)

By combining the differentiable graph clustering loss of each layer in GATE, we can obtain the total differentiable graph clustering loss as follows:

$${\mathcal{L}}_{SG}={\sum }_{l=1}^L{\mathcal{L}}_{SG}^{(l)}.$$ (13)

We reconstruct the cell–cell graph by designing an inner product decoder to decode the learned representations of GATE and define the reconstruction loss of GATE as follows:

$$\begin{array}{c}\widehat{\mathbf{A}}=\mathit{sigmoid}\left({\mathbf{H}}^{(L)T}{\mathbf{H}}^{(L)}\right),\hfill \\ {\mathcal{L}}_{\mathit{GATEres}}=\mathit{MSEloss}(\mathbf{A},\widehat{\mathbf{A}}).\hfill \end{array}$$ (14)

The reconstruction loss of the interactive module is calculated as the sum of the reconstruction losses of AE and GATE,

$${\mathcal{L}}_{\mathit{res}}={\mathcal{L}}_{\mathit{AEres}}+{\mathcal{L}}_{\mathit{GATEres}},$$ (15)

where ${\mathcal{L}}_{\mathit{AEres}}$ is the reconstruction loss of AE and the specific definition can be found in Note 3, available as supplementary data at Bioinformatics online.

2.3 Decoupled self-supervised clustering loss

Self-supervised learning empowers the model to learn discriminative representations by mining deep semantic information from massive unlabeled data. In the self-supervised clustering literature, dual-channel networks often are optimized under the supervision of a common highly confidence signal derived from Student’s t-distribution on both the AE and GNN channels, such as Yan et al. (2023) and Bo et al. (2020). However, due to the uniqueness of features learned from different channels, a common high-confidence distribution cannot effectively guide the learning process of two channels. In this subsection, a decoupled self-supervised clustering loss is formulated to simultaneously guide the representation learning of AE and GATE in the interactive module.

For the learned representations ${\mathbf{Z}}^{(L)}$ and ${\mathbf{H}}^{(L)}$ of AE and GATE, we first utilize K-means to initialize the cluster centers c for the learned cell node representations ${\mathbf{Z}}^{(L)}$. Then, the similarity between cell representations and cluster centers can be transformed into cluster assignments ${\mathbf{Q}}^z$ and ${\mathbf{Q}}^h$ based on Student’s t-distribution, as defined by the following formula:

$$q_{ij}=\frac{{\left({\Vert z_i-c_j\Vert }^2+1\right)}^{-1}}{{\sum }_{k=1}^K{\left({\Vert z_i-c_k\Vert }^2+1\right)}^{-1}},$$ (16)

where $z_i$ represents the learned representation of cell $x_i$, and $c_j$ represents the jth cluster center.

Then, the enhanced cluster assignments ${\mathbf{P}}^z=\left[p_{ij}^z\right]\in {\mathbb{R}}^{N\times K}$ and ${\mathbf{P}}^h=\left[p_{ij}^h\right]\in {\mathbb{R}}^{N\times K}$ are generated by quadratically operating the soft cluster assignments ${\mathbf{Q}}^z$ and ${\mathbf{Q}}^h$, providing a more reliable supervisory signal for the training process (Wang et al. 2019). Specifically, the enhanced cluster assignments ${\mathbf{P}}^z$ and ${\mathbf{P}}^h$ are calculated as follows:

$$p_{ij}^z=\frac{q_{ij}/{\sum }_{i=1}^N{q}_{ij}}{{\sum }_{k=1}^K\left(q_{ik}^2/{\sum }_{i=1}^N{q}_{ik}\right)},$$ (17)

where ${\mathbf{P}}^z$ and ${\mathbf{P}}^h$ can be seen as clustering assignments with higher confidence that ${\mathbf{Q}}^z$ and ${\mathbf{Q}}^h$ (the detailed explanation of why ${\mathbf{P}}^z$ and ${\mathbf{P}}^h$ can be treated as high-confidence assignments to guide clustering is presented in Note 4, available as supplementary data at Bioinformatics online). Thus, in each iteration, we minimize the Kullback–Leibler (KL) to make these two clustering assignments closer to each other, which ensures the clustering consistency between cluster assignment and its enhanced one according to the following equation:

$$\begin{array}{c}{\mathcal{L}}_{KL}^z=KL\left({\mathbf{P}}^z\left|\right|{\mathbf{Q}}^z\right)={\sum }_i{\sum }_j{p}_{ij}^z\hspace{0.17em}\text{log}\hspace{0.17em}\left(p_{ij}^z/q_{ij}^z\right),\hfill \\ {\mathcal{L}}_{KL}^h=KL\left({\mathbf{P}}^h\left|\right|{\mathbf{Q}}^h\right)={\sum }_i{\sum }_j{p}_{ij}^h\hspace{0.17em}\text{log}\hspace{0.17em}\left(p_{ij}^h/q_{ij}^h\right).\hfill \end{array}$$ (18)

Now, the overall loss function of the proposed DGCSG consists of reconstruction loss [Equation (15)], decoupled self-supervised clustering loss [Equation (18)] and differentiable graph clustering loss [Equation (13)], which can be formulated as follows:

$${\mathcal{L}}_{\mathit{overall}}={\mathcal{L}}_{\mathit{res}}+{\mathcal{L}}_{KL}^z+\alpha {\mathcal{L}}_{SG}+\beta {\mathcal{L}}_{KL}^h,$$ (19)

where $\alpha$, $\beta$ are the tradeoff parameters. This loss function combines data reconstruction, differentiable structural grouping information, and decoupled self-supervised optimization together, thus promoting the clustering performance.

2.4 Datasets and evaluation metrics

DGCSG is evaluated on 14 real scRNA-seq datasets, which include Klein (Klein et al. 2015), Chung (Chung et al. 2017), Sun.1, Sun.2, Sun.3 (Sun et al. 2019), Biase (Biase et al. 2014), Muraro (Muraro et al. 2016), Deng (Deng et al. 2014), Pollen (Pollen et al. 2014), Darmanis (Darmanis et al. 2015), 10X_PBMC (Zheng et al. 2017), Mouse bladder cells (Han et al. 2018), Habib (Habib et al. 2017), and Zeisel (Zeisel et al. 2015). Table 1, available as supplementary data at Bioinformatics online, provides detailed information about these datasets. The cells in these datasets vary from different organisms, organs and platforms, and their numbers range from tens to thousands.

In this study, we adopt two widely adopted clustering metrics including the Adjusted Rand Index (ARI) (Hubert and Arabie 1985) and the normalized mutual information (NMI) (Strehl and Ghosh 2002) to quantify the similarity between the predicted results and the real cell types. The higher the value of the metric, the higher the similarity. The definition and calculation process of clustering metrics are presented in Note 5, available as supplementary data at Bioinformatics online.

3 Results

3.1 Clustering performance analysis

To verify that DGCSG has excellent clustering performance, we compare the proposed method with 16 state-of-the-art clustering approaches. The comparison approaches include four graph clustering methods, i.e. CCGC (Yang et al. 2023), MBN (Yan et al. 2023), SDCN (Bo et al. 2020), SCGC (Kulatilleke et al. 2025), and 12 scRNA-seq clustering methods which include contrastive clustering methods, i.e. contrastive-sc (Ciortan and Defrance 2021), scNAME (Wan et al. 2022), nsDCC (Wang et al. 2024), deep embedded clustering methods, i.e. scDeepCluster (Tian et al. 2019), scziDesk (Chen et al. 2020), scSMD (Cui et al. 2025), deep graph clustering methods, i.e. scGNN (Wang et al. 2021), scGAC (Cheng and Ma 2022), scDFC (Hu et al. 2023), scTAG (Yu et al. 2022), scLEGA (Liu et al. 2024b), scDFN (Liu et al. 2024a). Among these approaches, scGAC and scDFC apply GATE for clustering. The scDFC, scDFN, and scLEGA use dual-channel networks integrating an AE and a GNN module to learn cell representations for clustering. The key implementation and configuration details of DGCSG and the comparison algorithms are presented in Note 6, available as supplementary data at Bioinformatics online. To ensure fairness, all approaches are repeatedly performed 10 times to report the average scores of ARI and NMI metrics.

From Fig. 2A, it can be observed that DGCSG achieves the highest NMI scores on 12 datasets and the highest ARI scores on 13 datasets. Notably, for the ARI and NMI metrics, we calculate the average scores of each baseline on all datasets, and DGCSG achieves the highest scores on both metrics.

Figure 2.

(A) NMI and ARI scores of DGCSG and 16 baselines on the 14 datasets. The performance of each baseline on each dataset is represented by a block, where the color and size of the block indicate the score and rank. The average performance of each baseline on all datasets is shown in the last row. In some places, there is no block because of out of memory. (B) Cell representations learned by DGCSG and five baselines on the Klein dataset. The color represents the predicted labels of each baseline. (C) The Sankey diagram shows the correspondence between the predicted results (on the left) and the real cell types (on the right) of DGCSG and four baselines on the Muraro dataset.

Tables 2 and 3, available as supplementary data at Bioinformatics online, present the detailed NMI and ARI scores for the comparison results, respectively. From the result, we get the follow observations: (i) DGCSG is superior to the deep embedded clustering baselines. For example, DGCSG obtains 17% and 22% improvements in terms of the ARI metric compared with scDeepCluster and scziDesk, respectively. This phenomenon shows that cell–cell structural relationship is an important cue for scRNA-seq clustering. (ii) Leveraging dual-channel network structure (MBN, SDCN, scDFN, scLEGA, and scDFC) to capture node and structural information of cells cannot consistently improve clustering performance. This is mainly because these baselines only fuse the learned cell node representations and structural representations, without addressing the heterogeneous gap between cell node feature and their structural information. In contrast, the interactive model in DGCSG dynamically integrates node and structural information in a layer-by-layer manner, achieving better clustering performance. (iii) Compared with GATE-based deep scRNA-seq clustering baselines (scDFC and scGAC), DGCSG achieves better clustering results. This is mainly because these baselines ignore the structural grouping information between cell nodes after constructing the cell–cell graph. In contrast, the differentiable graph clustering in our DGCSG assists GATE in allocating more attention scores to cell nodes in the same structural grouping, providing better cell representations for downstream clustering. In summary, DGCSG achieves superior clustering performance compared with other methods.

Visualize the cell representations of the Klein dataset learned by DGCSG and five other baselines to intuitively understand the cell representations in low-dimensional space. As shown in Fig. 2B, the learned cell representations of DGCSG have clear boundaries and can separate different cell types well. In contrast, scDFC, scLEGA, scDeepCluster, and scGNN show limitations in clearly separating cells from different clusters, while MBN mixes cells from different clusters together. Visualizations of cell representations for some other datasets are shown in Figs 2 and 3, available as supplementary data at Bioinformatics online.

A Sankey diagram showing the correspondence between predicted results and real cell types is used to intuitively assess the accuracy of DGCSG and other four baselines, where the number of matches between predicted labels and true labels is represented by the width of the connecting lines. In this part, we select the Muraro dataset for visualization, which contains nine cell types: acinar, alpha, beta, epsilon, delta, duct, endothelial, mesenchymal, and pp. As shown in Fig. 2C, DGCSG generates nine predicted clusters, but the predicted cluster four fails to accurately correspond to epsilon cells. This is because there are only three epsilon cells in the Muraro dataset, and the small number makes accurate prediction more challenging. In contrast, scGNN and scLEGA incorrectly predict the number of clusters, while scDeepCluster and scDFC partition the same type of cells into different clusters.

3.2 Robustness analysis under various data sparsity levels

To evaluate the robustness of DGCSG on the single-cell data with various levels of sparsity, we perform experiments from the following two aspects: sequencing depth and dropout event.

To evaluate the robustness of DGCSG at different sequencing depths, we analyze its clustering performance by subsampling the sequencing depths of the heart10k and neuron10k datasets. The dataset descriptions and experimental implementation details are provided in Table 4 and Note 7, available as supplementary data at Bioinformatics online, respectively. The clustering performance of DGCSG with graph clustering, deep embedded clustering, contrastive clustering, attention mechanism based clustering, and dual-channel clustering baselines at different sequencing depths is shown in Fig. 3A and Fig. 4A, available as supplementary data at Bioinformatics online, and the corresponding NMI and ARI values are displayed in Tables 5–8, available as supplementary data at Bioinformatics online. From these figures, it can be observed that DGCSG outperforms other baselines in clustering performance under all sequencing depth conditions. DGCSG maintains superior performance even as the sequencing depth gradually decreases. In contrast, some baselines are sensitive to sequencing depth. For example, on the heart10k dataset, the clustering performance of scLEGA has a significant decrease of 18% in the ARI metric when the sequencing depth is reduced to 10% of the original depth. In summary, DGCSG shows better robustness under different sequencing depths.

Figure 3.

(A) ARI values of DGCSG and other baselines at different sequencing depths on heart10k dataset and neuron10k dataset. Note that scDFC is not included in the figure due to out-of-memory error. (B) ARI values of DGCSG and other baselines at different dropout rates on the simulated dataset.

In addition, we evaluate the impact of dropout events on the clustering performance of DGCSG by setting different levels of dropout rate in the simulated dataset. The descriptions of the dropout rate and simulated dataset, as well as the corresponding experimental settings are provided in Note 8, available as supplementary data at Bioinformatics online. The clustering performance of DGCSG with other baselines at different dropout rates is demonstrated in Fig. 3B and Fig. 4B, available as supplementary data at Bioinformatics online, and the corresponding NMI and ARI values are displayed in Tables 9 and 10, available as supplementary data at Bioinformatics online. From these figures, it can be observed that when the dropout rate is 38.89%, only the clustering metrics of DGCSG and contrastive-sc remain above 0.9. In contrast, the clustering performance of other methods such as scDFC and scLEGA is significantly degraded, with metrics close to 0. Furthermore, when the dropout rate reaches 47.96%, DGCSG still achieves optimal clustering performance, demonstrating its robustness in handling highly sparse scRNA-seq data, as well as its effectiveness in capturing the underlying cellular structure.

In summary, DGCSG achieves excellent clustering performance under different sparsity levels, verifying its robustness and superiority.

3.3 Ablation analysis

In this part, we conduct ablation experiments to verify the effectiveness of each module in DGCSG. There are five ablation variants: (i) removing the reconstruction loss, i.e. w/o ${\mathcal{L}}_{\mathit{res}}$, (ii) removing the AE module in the interactive module, i.e. w/o AE, (iii) removing the GATE module in the interactive module, i.e. w/o GATE, (iv) removing the differentiable graph clustering loss, i.e. w/o ${\mathcal{L}}_{SG}$, (v) removing the decoupled self-supervised optimization module, and applying K-means clustering to the learned representations, i.e. w/o KL.

Figure 5A, available as supplementary data at Bioinformatics online, shows the ARI scores of these five variants with DGCSG on each dataset. The results indicate that for the majority of datasets, these variants result in either a decline or no significant change in clustering performance, with only a few datasets showing slight improvements. For example, removing the reconstruction loss slightly enhances the performance on the Sun.3 dataset, while removing GATE leads to marginal improvements on the Habib and 10X_PBMC datasets. Figure 5B, available as supplementary data at Bioinformatics online, shows the average ARI scores of DGCSG with five variants for 14 datasets, it can be observed that DGCSG outperforms these five variants in terms of clustering performance, which verifies the effectiveness of each module of DGCSG.

3.4 Model analysis

We explore the sensitivity of parameters $\alpha$ and $\beta$ to the clustering performance of DGCSG. The value of $\alpha$ is taken from {3e-6,1e-5, 3e-5,1e-4, 3e-4}, while $\beta$ takes values in the range {4, 16} with a step of 2, respectively. Figure 6, available as supplementary data at Bioinformatics online, shows the average results of 10 runs with different tradeoff parameters on all datasets. It can be seen that DGCSG achieves stable clustering on most of the datasets for different values of the tradeoff parameters. Based on these experimental results, we set $\alpha$=3e-5, $\beta$=10 as the tradeoff parameters for all scRNA-seq datasets.

Besides, exploring model convergence is not only an important step to verify the correctness of the theory but also crucial for ensuring the dependability of the results and evaluating the stability of model training. As shown in Fig. 7, available as supplementary data at Bioinformatics online, we can observe that the objective function decreases and tends to converge with increasing of training iterations, while the clustering results increase and tend to converge. This phenomenon demonstrates that DGCSG has a good convergence property on single-cell scRNA-seq data.

In addition, we analyze the time complexity of DGCSG, which is detailed in Note 9, available as supplementary data at Bioinformatics online. The corresponding running time for DGCSG compared with 16 baselines on each dataset is presented in Table 11, available as supplementary data at Bioinformatics online. Since scGAC and scDFC are out of memory on the Habib dataset, we compute the average running times of DGCSG and the 16 baselines on the other 13 datasets. The result in Fig. 8, available as supplementary data at Bioinformatics online, demonstrates the superiority of DGCSG in terms of runtime.

3.5 Significance test

To verify that DGCSG achieves a significant improvement in clustering performance compared with existing baselines, we conduct a one-sided paired t-test of the experimental results on the ARI metrics and report the corresponding P-values. Specifically, for the experiments where DGCSG achieves the best performance, we define the null hypothesis as $H_0:AR{I}_{\mathit{DGCSG}}\le AR{I}_{\text{baseline}}$. For experiments where DGCSG does not achieve the optimal result, we define the null hypothesis as $H_0:AR{I}_{\text{baseline}}\le AR{I}_{\mathit{DGCSG}}$ to verify whether the baseline outperforms DGCSG. It is important to point out that DGCSG and several baselines obtained clustering performance with ARI = 1 on Biase dataset, thus no significance test is performed on this dataset. Table 12, available as supplementary data at Bioinformatics online, summarizes the significance test results of each method on different datasets, with bolded values representing the P-values where DGCSG fails to significantly outperform the baseline. Specifically, scDFN significantly outperforms DGCSG on the Habib dataset (P<.05), and DGCSG does not significantly outperform scSMD on the Muraro and Habib datasets (P>.05). However, on all the remaining datasets, DGCSG significantly outperforms the existing baselines. The above results validate the superiority of DGCSG in single-cell clustering tasks.

4 Biological analysis

4.1 Cell trajectory inference

Cell trajectory inference reconstructs the continuous trajectory of cells in the transcriptional space, providing a powerful tool for studying complex biological processes such as organ regeneration, cell differentiation, and embryonic development. In the literature, scCRT (Shi et al. 2024) is a novel and effective cell trajectory analysis method, which can perform cell trajectory inference based on the learned cell representations in low-dimensional space. In this experiment, we follow scCRT to perform cell trajectory inference on a synthetic dataset named binary_tree_8 with 8429 cells and 531 genes.

To evaluate the effectiveness of DGCSG, we adopt Hamming–Ipsen–Mikhailov (HIM), F1 branches and F1 milestones as metrics to measure the similarity between inferred trajectories and real trajectories. Specifically, HIM measures the similarity between predicted topology and real values, while F1 branches and F1 milestones measure the accuracy of cell ordering along trajectory branches and trajectory milestones, respectively. In addition, the degree of linear correlation between pseudotime and true time is quantified by calculating the Pearson correlation coefficient. In this experiment, the visualization of the cell trajectory inference results is provided in Fig. 9, available as supplementary data at Bioinformatics online, which shows the branching evolution and the corresponding position of the cell in the trajectory, with the left color representing the cell type and the right color representing the pseudotime. From the figure, it can be observed that the trajectories and pseudotimes inferred from the representations learned by DGCSG closely match the true values. Besides, the cells from different clusters are also distributed along the cell trajectories. In contrast, the trajectories and pseudotime inferred from the representations learned by other baselines exhibit inaccuracies. From this experiment, we can conclude that DGCSG is beneficial for cell trajectory inference.

4.2 Marker gene identification

Marker genes provide important information for evaluating cell states and annotating cell types, which helps to reveal the biological functions of cells. In this part, we first utilize the Wilcoxon Rank Sum test (Rosner et al. 2006) to identify the top 50 differentially expressed genes (DEGs) for each predicted cluster. Then, we select the top three DEGs as more representative marker genes. Figure 4 shows the expression levels of these marker genes in each predicted cluster of the Muraro dataset. It can be observed that the marker genes and their expression are different for each cell cluster, and the marker genes are expressed at significantly higher levels in the corresponding cell clusters compared with other cell clusters. In addition, we evaluate the reliability of the top three DEGs by referencing the marker genes recorded in the CellMarker (Zhang et al. 2019) database. It is observed that the vast majority of marker genes identified based on DGCSG predictions are highly consistent with the known cell type marker genes. For example, the DEGs (KLK1, CTRL, CTRC) in cluster 0 predicted by DGCSG are marker genes for acinar cells. Although some DEGs are not matched to CellMarker, we observe that they are highly expressed in the corresponding clusters, suggesting that these genes can potentially serve as novel marker genes for specific cell types.

Figure 4.

Expression levels of the top three DEGs identified by DGCSG for each predicted cluster. (A) Expression dot plot (Each dot represents the expression level of the gene in each predicted cluster. The color of each dot indicates the average expression value, while the size reflects the percentage of cells that are expressing the gene in each predicted cluster.). (B) Expression heat map.

5 Discussion

In this paper, we propose a novel DGCSG for scRNA-seq data. Specifically, we first construct an interactive module to fuse the node and structural information of cells. Then, a differentiable clustering mechanism is devised to jointly optimize GATE to learn the structural grouping information among cells, allocating more attention scores to nodes in the same graph cluster. Finally, a decoupled self-supervised optimization module is designed to guide the clustering process with specific high-confidence distributions. Experiments on 14 real scRNA-seq datasets show the superiority of DGCSG compared with state-of-the-art baselines. In future, it is interesting to explore the applicability of DGCSG on single-cell multiomics data.

Author contributions

Xiaoqiang Yan (Conceptualization [supporting], Formal analysis [lead]), Shike Du (Data curation, Methodology, Writing—original draft, Writing—review & editing [equal]), Quan Zou (Supervision [supporting]), and Zhen Tian (investigation, Supervision [equal])

Supplementary data

Supplementary data are available at Bioinformatics online.

Conflict of interest: None declared.

Funding

This work was supported by the National Natural Science Foundation of China [Nos. 62131004, 62371423]; the Municipal Government of Quzhou [No. 2024D033]; and the Natural Science Foundation of Henan [Nos. 252300421226, 252300421504].

Data availability

The data underlying this article are available in https://github.com/Xiaoqiang-Yan/DGCSG.