Synthetic control removes spurious discoveries from double dipping in single-cell and spatial transcriptomics data analyses

Abstract

Double dipping is a well-known pitfall in single-cell and spatial transcriptomics data analysis: after a clustering algorithm finds clusters as putative cell types or spatial domains, statistical tests are applied to the same data to identify differentially expressed (DE) genes as potential cell-type or spatial-domain markers. Because the genes that contribute to clustering are inherently likely to be identified as DE genes, double dipping can result in false-positive cell-type or spatial-domain markers, especially when clusters are spurious, leading to ambiguously defined cell types or spatial domains. To address this challenge, we propose ClusterDE, a statistical method designed to identify post-clustering DE genes as reliable markers of cell types and spatial domains, while controlling the false discovery rate (FDR) regardless of clustering quality. The core of ClusterDE involves generating synthetic null data as an in silico negative control that contains only one cell type or spatial domain, allowing for the detection and removal of spurious discoveries caused by double dipping. We demonstrate that ClusterDE controls the FDR and identifies canonical cell-type and spatial-domain markers as top DE genes, distinguishing them from housekeeping genes. ClusterDE’s ability to discover reliable markers, or the absence of such markers, can be used to determine whether two ambiguous clusters should be merged. Additionally, ClusterDE is compatible with state-of-the-art analysis pipelines like Seurat and Scanpy.

1. Introduction

Single-cell RNA-seq (scRNA-seq) and spatially resolved transcriptomics (SRT) technologies have revolutionized transcriptomic studies by providing unprecedented snapshots of gene expression within individual cells and in a spatial context, respectively. A major task in scRNA-seq and spatial transcriptomics data analysis is to annotate cell types and spatial domains with marker genes. For example, in scRNA-seq data analysis, a standard annotation procedure includes two steps: (1) partitioning cells into clusters, and (2) finding differentially expressed (DE) genes between cell clusters as marker genes for annotating clusters as potential cell types [1, 2, 3, 4], as in state-of-the-art analysis pipelines such as the R package Seurat [5] and the Python module Scanpy [6]. However, it has been realized that this post-clustering differential expression (DE) procedure is conceptually flawed. For instance, Seurat contains the warning message that “P values should be interpreted cautiously, as the genes used for clustering are the same genes tested for differential expression.” This issue is commonly referred to as “double dipping,” meaning that the same gene expression data are used twice: first to find cell clusters and then to identify DE genes. The double dipping issue leads to an inflated false discovery rate (FDR) when identifying post-clustering DE genes as putative cell-type marker genes. The FDR inflation is more pronounced when the cell clusters are more ambiguous.

To explain this double-dipping issue in scRNA-seq cell-type annotation, we discuss two extreme scenarios. In one scenario, if two cell types are considered “distinct” (i.e., cell-type marker genes exhibit bimodal expression distributions, with the high-expression mode corresponding to the cell type in which they are highly expressed) and the inferred clusters are accurate, then double-dipping has no impact on post-clustering DE analysis. In this case, DE analysis can successfully identify cell-type marker genes that are highly expressed in a cluster corresponding to a distinct cell type (Fig. 1a, top). In the opposite scenario, when a clustering algorithm over-clusters a single cell type into two clusters, post-clustering DE analysis will identify genes highly expressed in each cluster due to double-dipping: the clustering is based on gene expression data, and variations in the expression of certain genes drive the clustering, resulting in these genes being highly expressed in one of the clusters (Fig. 1a, bottom). However, since only one true cell type exists, these post-clustering DE genes cannot be used to justify the two clusters as distinct cell types and should not be regarded as cell-type marker genes.

Fig.1: ClusterDE mitigates double dipping in post-clustering DE analysis for identifying cell-type and spatial-domain marker genes.

a-b, An illustration of the double-dipping issue in (a) scRNA-seq and (b) SRT post-clustering DE analysis. In case 1, where two cell types or spatial domains exist and are accurately identified through clustering, double dipping has minimal effect, and the identified post-clustering DE genes align well with the true cell-type or spatial-domain marker genes (top row in panels a and b). However, in case 2, where a single cell type or spatial domain is incorrectly split into two clusters, double dipping results in post-clustering DE genes that are false-positive cell-type or spatial-domain marker genes (bottom row in panels a and b). c, An overview of ClusterDE, consisting of four steps. Step (1): Given the target real data (“Target”), ClusterDE generates synthetic null data (“Null”) using scDesign3 [29] to represent the null hypothesis of a single “hypothetical” cell type (scRNA-seq) or spatial domain (SRT). Steps (2)–(3): A clustering algorithm is used to divide cells or spatial spots into two clusters, followed by a DE test to compare each gene between the clusters. This process is applied to both the target data and the synthetic null data, producing two DE scores per gene—one from each dataset. Step (4): ClusterDE calculates a contrast score for each gene by subtracting the synthetic null DE score from the target DE score. It then applies the FDR-control method Clipper [30] to determine a contrast score cutoff based on all genes’ contrast scores and a target FDR (default: 0.05). DE genes are identified as those with contrast scores exceeding this cutoff, as indicated by the vertical dashed line in the contrast score distribution across genes. d, ClusterDE mitigates the false discoveries caused by double dipping in case 2 shown in a and b (e.g., gene m is no longer identified as a false-positive marker gene).

Some biologists might argue that even if the two clusters are “indistinct” (i.e., all genes exhibit unimodal expression distributions across all cells), they could still reflect meaningful biological variation and thus represent potential cell sub-types. While this argument may hold in certain scenarios, we argue that if the two clusters are indistinct, it is impossible to determine whether they should be classified as two distinct cell types or merely arbitrary divisions along a continuum of cell state changes, without external knowledge. A more critical issue is that if clusters are not required to be distinct, cells could be continuously divided into finer and finer clusters, which would make it unreasonable to consider any such clusters as potential cell types. This would also hinder the transfer of cell type labels from one dataset to another. Given this issue and the need for a clear definition of cell types in cell atlases—reflected by the ongoing debate about what constitutes a cell type [7, 8]—we focus, in this work, on the practice of using post-clustering DE analysis to annotate distinct cell clusters as potential cell types.

Methods aimed at addressing the double-dipping issue in scRNA-seq post-clustering DE analysis include the Truncated Normal (TN) test [9] and the Countsplit method [10]. The TN test method relies on splitting cells and consists of five steps: (1) dividing the cells into two sets—training and test cells; (2) applying a clustering algorithm to the training cells to identify two clusters; (3) training a support vector machine (SVM) classifier on the training cells to predict cluster labels based on gene expression; (4) using the trained classifier to predict the cluster labels of the test cells; and (5) identifying differentially expressed (DE) genes between the two clusters in the test cells using the TN test. Since the TN test method requires gene expression levels to follow normal distributions, it applies the log-transformation to the scRNA-seq count data before the five steps. Instead of splitting cells, the Countsplit method splits the scRNA-seq cell-by-gene count matrix into training and test matrices of the same dimensions (cells and genes) using a procedure called data thinning [11], which splits each count into two counts, whose sum equals the original count, through binomial sampling. Countsplit identifies cell clusters by applying a clustering algorithm to the training matrix. Since the test and training matrices contain the same cells, the cell cluster assignments can be directly applied to the test matrix. Countsplit then identifies DE genes between the clusters based on the test matrix, allowing the use of any suitable statistical test for count data, beyond the TN test. Although both methods claimed to provide well-calibrated P values (i.e., uniformly distributed between 0 and 1 under the null hypotheses), our findings indicate that their P values are anti-conservative (i.e., enriched near 0) in the presence of gene-gene correlations (see Results). The reason behind this issue is that the validity check of P values in the TN test and Countsplit papers relied on simulation studies that implicitly assumed genes to be independent [10], an assumption that, however, does not hold in real scRNA-seq data. As a result of their anti-conservative P values, both methods still lead to inflated FDRs when applied to real scRNA-seq data.

Several cluster-free DE tests attempt to circumvent the double-dipping issue by bypassing the cell clustering step [12, 13, 14, 15, 16, 17]. However, it is important to note that these cluster-free methods are not designed to identify potential cell types. Consequently, the DE genes identified by these methods cannot be interpreted as marker genes for specific cell types, unlike the DE genes identified post-clustering. In other words, cluster-free DE genes and post-clustering DE genes serve different purposes and are not conceptually comparable. Another class of methods aim to assess the quality of clustering results, such as evaluating the “purity” of a cluster or determining if two clusters should be merged [18, 19, 20, 21, 22, 23, 24, 25]. However, these methods lack a direct statistical test for identifying DE genes and rely on setting a threshold for clustering quality—an arbitrary decision to mitigate concerns about double dipping. In this study, we focus on identifying reliable marker genes for cell clusters during the cell-type annotation process. Hence, we do not consider cluster-free DE tests and clustering assessment methods as competing alternatives in our investigation.

The double-dipping issue in post-clustering DE analysis is not unique to scRNA-seq data; it is also relevant for the more recent spatially resolved transcriptomics (SRT) data, which captures tissue context information of cells or cell neighborhoods, depending on the spatial resolution of the SRT technology used [26]. A common analysis task is spatial domain detection, which involves clustering proximal spatial spots—representing cells or cell neighborhoods—based on their gene expression profiles, with the goal of identifying spatial domains that correspond to functionally distinct tissue structures [27]. Although not widely discussed, the double-dipping issue can lead to unreliable marker genes for “indistinct” spatial domains, where no genes show sharp expression changes at the domain boundaries [28]. However, we argue that reliable spatial domains should be “distinct” from nearby domains, meaning that the domain’s marker genes should exhibit sharp expression changes at the boundaries. Otherwise, similar to the challenge of defining cell types in scRNA-seq data, indistinct spatial domains cannot have their labels reliably transferred across datasets. Despite this challenge, no methods have been developed to circumvent the double-dipping issue in spatial domain annotation. Motivated by the need for annotating distinct spatial domains, we define spatial domain marker genes as those exhibiting discontinuity or sharp changes in expression at domain boundaries (Fig. 1b, top). In contrast, genes that are not defined as spatial domain markers may still show spatial variation, but the variation is smooth (Fig. 1b, bottom). Similar to scRNA-seq data analysis, our goal is to enable post-clustering DE analysis to identify reliable marker genes for annotating distinct spatial domains.

Here we introduce ClusterDE, a post-clustering DE method designed to identify potential cell-type or spatial-domain marker genes while avoiding the inflated FDR caused by double dipping. ClusterDE effectively controls the FDR when identifying marker genes, even in cases where cell or spatial clusters are spurious. The ultimate goal is to use the identified marker genes to reliably annotate distinct cell types or spatial domains. It is important to note that ClusterDE is not intended to replace existing pipelines that perform cell or spatial clustering followed by DE analysis (e.g., Seurat). Instead, it functions as an add-on to any current pipeline, enabling more reliable discoveries. Additionally, ClusterDE offers a practical advantage by allowing users to interpret an abstract statistical null hypothesis through concrete synthetic null data (i.e., in silico negative controls). This feature enables users to assess whether the synthetic null data accurately reflects the negative control scenario they envision and, if necessary, adjust the generation of the synthetic null data.

ClusterDE demonstrates strong performance in both scRNA-seq and SRT post-clustering DE analysis. In scRNA-seq applications, ClusterDE was benchmarked against the Seurat pipeline (which includes double dipping), the TN test, and Countsplit, and was shown to be the only method that effectively controls the FDR. ClusterDE effectively avoids false-positive cell-type markers, including housekeeping genes, and prioritizes relevant canonical markers in datasets from five cell lines and peripheral blood mononuclear cells (PBMC). Additionally, ClusterDE outperforms Seurat in distinguishing cell types in an adult Drosophila dataset. When extended to SRT post-clustering DE analysis, ClusterDE also outperforms the double-dipping approach (BayesSpace for spatial clustering and Seurat for DE), demonstrating effective FDR control. ClusterDE successfully identifies no DE genes between spurious spatial clusters in a human brain tissue SRT dataset and a human pancreas cancer tissue SRT dataset. Moreover, the top marker genes identified by ClusterDE for spatial domains, which align well with annotated cancer regions, exhibit distinctly higher expression in these regions compared to the genes identified by the double-dipping approach. Hence, ClusterDE is an effective tool for scRNA-seq and SRT data analysis, enabling the reliable identification of cell-type or spatial-domain marker genes that can effectively distinguish cell types or spatial domains.

2. Results

2.1. ClusterDE: a synthetic control and contrastive approach to address double-dipping

ClusterDE introduces a novel synthetic control and contrastive approach to identify reliable cell-type and spatial-domain genes that are robust to double dipping (clustering followed by DE analysis). The approach centers on establishing an in silico negative-control data (referred to as the “synthetic null data”) to be analyzed in parallel with the real data (referred to as the “target data”) through a computational pipeline, which includes cell or spatial clustering followed by DE analysis. The contrastive approach identifies reliable cell-type and spatial-domain marker genes by comparing DE analysis results from the target data to those from the synthetic null data. The target data consists of cells or spatial spots divided into two potentially ambiguous clusters, which are then analyzed by DE analysis to identify cell-type or spatial-domain marker genes. To generate the synthetic null data, ClusterDE includes two null models: an scRNA-seq null model, which assumes that the cells in the target data belong to a homogeneous cell type, with gene expression exhibiting unimodal distributions across the cells; and an SRT null model, which assumes that the spatial spots in the target data belong to a homogeneous domain, with gene expression varying smoothly within the domain. Under the null models, no cell-type or spatial-domain marker genes are expected to be detected. To identify potential cell-type or spatial-domain marker genes, ClusterDE involves four main steps (Fig. 1c).

Step 1 of ClusterDE is synthetic null data generation, where the statistical simulator scDesign3 [29] is used to generate the synthetic null data that represents a hypothetical homogeneous cell type or spatial domain. The synthetic null data preserves key statistics of the target data, including gene means, variances, and, importantly, gene-gene correlations, while maintaining the same number of cells or spatial spots and the same set of genes (Figs. 2a and S2 for scRNA-seq; Figs. S3 and S4 for SRT). Details of the generation process for scRNA-seq and SRT data are provided in “Supplementary Methods” in the Supplementary Material.

Fig.2: ClusterDE achieved reliable FDR control and good power in simulation studies.

a, When the target data contained cells from a single type (simulation; see Supplementary Methods “Simulation setting with one cell type” in the Supplementary Material), the synthetic null data generated by ClusterDE resembled the target data well in terms of UMAP cell embeddings (left), per-gene expression mean and variance statistics (middle), and gene-gene rank correlations (right). b, On the target data in a, ClusterDE (with five DE tests) outperformed three other methods—including Seurat (which involves double dipping), Countsplit (which aims to address double dipping and works with any DE test), and TN test (which aims to address double dipping and has its own DE test)—in FDR control. The horizontal dashed line indicates the target FDR of 0.05. The five DE tests are the Wilcoxon rank-sum test (Wilcoxon), t-test, negative binomial generalized linear model (NB-GLM), logistic regression model predicting cluster membership with likelihood-ratio test (LR), and likelihood-ratio test for single-cell gene expression (bimod). c, The FDRs and power of ClusterDE and the three other methods under various severity levels of double dipping. The log fold change (logFC) summarizes the average gene expression difference between two cell types in simulation (see Supplementary Methods “Simulation setting with two cell types and 200 true DE genes” in the Supplementary Material). Corresponding to a small logFC, a small adjusted Rand index (ARI) represents a bad agreement between cell clusters and cell types, representing a more severe double-dipping issue. Across various severity levels of double dipping, ClusterDE controlled the FDRs under the target FDR thresholds (diagonal dashed line) and achieved comparable or higher power compared to the three other methods at the same actual FDRs.

Steps 2 and 3 of ClusterDE comprise a user-defined pipeline for clustering and subsequent DE analysis, with examples including Seurat in R or Scanpy in Python. ClusterDE offers flexibility, allowing users to choose their preferred clustering algorithms and DE tests to analyze the target data and synthetic null data in parallel. These two steps yield a “target DE score” and a “null DE score” for each gene. Specifically, we define a gene’s DE score as a summary statistic that quantifies the significance of the gene’s expression difference between two clusters in the target data or the synthetic null data; a higher DE score indicates that the gene is more likely DE. In the default setting of ClusterDE, the DE score is defined as the negative logarithm of the P value obtained from a DE test (e.g., the Wilcoxon rank-sum test) that compares a gene’s expression levels between two clusters.

Finally, Step 4 of ClusterDE implements a contrastive strategy to compare each gene’s target DE score and null DE score. A gene is identified as a reliable cell-type or spatial-domain marker only if its target DE score significantly exceeds its null DE score. To implement this identification strategy, ClusterDE computes “contrast score” for each gene by subtracting its null DE score from its target DE score. True non-marker genes are expected to have contrast scores symmetrically distributed around zero. ClusterDE uses the FDR control method Clipper [30] to determine a contrast score cutoff based on a target FDR (e.g., 0.05). Genes with contrast scores equal to or exceeding the cutoff are identified as DE genes. Regarding why ClusterDE can control the FDR when the number of genes is large, a mathematical proof is provided in “Theoretical justification of ClusterDE” in the Supplementary Material.

Through these four steps, ClusterDE effectively eliminates false-positive marker genes caused by double dipping, particularly when the target data consists of a single cell type or spatial domain (Fig. 1d).

2.2. Simulations verify ClusterDE’s reliable FDR control and statistical power

We conducted extensive simulation studies to validate ClusterDE as a post-clustering DE method with reliable FDR control under double dipping. We also compared ClusterDE with Seurat, the most widely used analysis pipeline that involves double dipping, and two methods designed to address double dipping—the TN test [9] and Countsplit [10]. For all four methods, we used the default Louvain clustering algorithm in Seurat, as it is commonly employed in scRNA-seq data analyses. In the DE analysis step for ClusterDE, Seurat, and Countsplit, we evaluated five DE tests available in Seurat: the Wilcoxon rank-sum test (Wilcoxon; the default in Seurat), the two-sample t-test (t-test), the negative binomial generalized linear model (NB-GLM), logistic regression (LR), and the likelihood-ratio test (bimod). The TN test was an exception, as it requires its own DE test. Since Seurat, Countsplit, and the TN test all produce a P value for each gene, we applied the Benjamini-Hochberg (BH) procedure to determine a P value cutoff for a given target FDR (e.g., 0.05). Genes with P values at or below the cutoff were identified as DE genes.

In the first simulation setting, we simulated target data from a single cell type, mimicking naïve cytotoxic T cells in a PBMC scRNA-seq dataset [31] (Fig. 2a, top left; see “Simulation designs” in the Supplementary Material), representing the most severe double-dipping scenario. Since the data consist of only a single cell type, any identified DE genes are false discoveries. At a target FDR of 0.05, all three existing methods—Seurat, Countsplit, and the TN test—failed to control the actual FDR below 0.05 (Fig. 2b). Seurat, which employs a double-dipping approach, performed the worst, with all five DE tests yielding actual FDRs of 1, indicating that Seurat consistently identified false DE genes across 200 simulation runs. Although Countsplit and the TN test were designed to mitigate double dipping, their actual FDRs still far exceeded 0.05 due to anti-conservative P values in the presence of gene-gene correlations (Fig. S5, middle to right), despite their validity in idealized settings with independent genes, as demonstrated in their own simulation studies [10]. In contrast, ClusterDE successfully controlled the FDR below 0.05 for three of the five DE tests: Wilcoxon, t-test, and LR (Fig. 2b), supported by contrast scores that satisfied the required symmetry around zero (Fig. S5, left; “Theoretical justification of ClusterDE” section of Supplementary Material). While ClusterDE did not control the FDR below 0.05 for the NB-GLM and bimod tests, likely due to violations of their parametric modeling assumptions, the FDR inflation for these tests was substantially less severe than that observed with Countsplit. Specifically, ClusterDE’s actual FDRs were 0.28 and 0.16 for NB-GLM and bimod, respectively, compared to Countsplit’s actual FDRs of 0.68 and 1 for the same tests (Fig. 2b).

In the second simulation setting, we generated datasets with varying degrees of double-dipping, again mimicking naïve cytotoxic T cells from the same PBMC scRNA-seq dataset [31] (Fig. 2c, top; see “Simulation designs” in the Supplementary Material). Each simulated dataset consisted of two cell types with equal numbers of cells and 200 pre-specified true DE genes with varying expression level differences between the cell types, summarized as the log fold change (logFC), where larger logFC values indicate greater distinctions between cell types. After applying Seurat’s default clustering algorithm to each dataset to identify two clusters, the agreement between the identified clusters and the true cell types was assessed using the adjusted Rand index (ARI), with lower ARI values corresponding to more severe double-dipping scenarios, as visualized by UMAP (Fig. 2c, top row). For DE analysis, we used Wilcoxon, the default DE test in Seurat, as it provided the best FDR control for both ClusterDE and Countsplit, while the TN test employed its own DE test. Results show that ClusterDE consistently controlled FDRs across a range of target thresholds under varying degrees of double dipping, whereas Seurat, Countsplit, and the TN test failed to control FDRs under the target thresholds and exhibited greater FDR inflation as the severity of double dipping increased (Fig. 2c, middle row). Notably, ClusterDE achieved comparable or superior statistical power relative to Seurat, Countsplit, and the TN test at equivalent actual FDR levels (Fig. 2c, bottom). These findings held across four other DE tests (t-test, NB-GLM, LR, and bimod) when used with ClusterDE, Seurat, and Countsplit (Fig. S6). To further evaluate performance in realistic scenarios with unbalanced cell type sizes, we simulated datasets with cell type size ratios of 1:4 and 1:9, where ClusterDE still outperformed the other three methods in FDR control across target thresholds and varying degrees of double-dipping. Specifically, ClusterDE consistently demonstrated robust FDR control and comparable or superior statistical power to the other methods when using Wilcoxon as the DE test (Figs. S6–S8).

Technically, ClusterDE and knockoff methods share the concept of controlling the FDR by generating in silico negative control data [32], but their applications differ significantly. Knockoff methods are designed to identify important features in high-dimensional predictive models within a supervised-learning framework, in contrast to the unsupervised-learning setting addressed by ClusterDE. Broadly, knockoff methods generate features uncorrelated with the outcome variable given the other features while preserving feature-feature correlations. To assess their applicability to our unsupervised learning setting, we applied the default model-X knockoffs method [33] to the target data from the aforementioned simulation, treating genes as features and the cell cluster label as the outcome variable. While the method controlled the FDR, it consistently yielded zero statistical power, making it impractical for DE gene identification. Furthermore, we compared three alternative strategies for synthetic null data generation in Step 1 of ClusterDE: the model-X knockoffs method, a permutation-based approach (where genes are independently permuted across all cells to create a homogeneous cell population), and a variant of ClusterDE using scDesign3 without copula modeling [29] (assuming gene independence). Fig. S9 shows that ClusterDE’s synthetic null data effectively preserved per-gene mean and variance statistics, as well as gene-gene correlations. In contrast, the model-X knockoffs method failed to maintain gene mean and variance statistics, while the permutation-based approach and the ClusterDE variant (labeled as “scDesign3 Ind”), as expected, did not preserve gene-gene correlations. As a result, only ClusterDE’s synthetic null data effectively mimics the target data while creating a single hypothetical cell type (Fig. S9). Overall, results on simulated datasets demonstrate that ClusterDE achieved the most robust FDR control and the best statistical power among the four strategies for synthetic null generation (Fig. S10).

To address concerns about the randomness involved in generating synthetic null data—a process that relies on random sampling from the null model fitted to the target data—we conducted an analysis to evaluate the stability of DE genes identified by ClusterDE. The results demonstrate that the DE genes identified by ClusterDE are relatively stable and robust to this randomness (Fig. S11). In summary, these simulation studies confirm that ClusterDE is an effective method that controls FDR under varying degrees of double dipping while maintaining strong statistical power.

2.3. ClusterDE distinguishes cell-type marker genes from housekeeping genes

We applied ClusterDE to multiple scRNA-seq datasets to demonstrate its ability to enhance the reliability of cell-type marker genes identified through post-clustering DE analysis.

In the first application, we aimed to demonstrate that ClusterDE can avoid false-positive cell-type marker gene discovery when the cells belong to only one type, not more than one. To this end, we analyzed five datasets of pure cell lines commonly used as gold standards of a “single cell type” in benchmarking studies [34, 35] (Fig. 3a, left). While biological heterogeneity within a cell line may include variations in cell cycle, differentiation, or subclonal populations, a cell line is generally not considered to be composed of distinct cell types [36, 37]. Accordingly, any post-clustering DE genes identified from a cell line should not be interpreted as between-cell-type DE genes. Hence, these cell-line datasets served as negative cases to demonstrate the inflated FDRs of existing methods and the ability of ClusterDE to eliminate this inflation. As a sanity check, we verified that the synthetic null data generated by ClusterDE resembled the target data (Fig. 3a, right). Applying ClusterDE, Seurat, Countsplit, and the TN test to the five datasets, we found that all methods except ClusterDE identified thousands of DE genes, often exceeding 50% of all genes, indicating severe FDR inflation at the target threshold of 5%. In contrast, ClusterDE identified zero DE genes in 22 out of 25 combinations of the five datasets and five DE tests (Wilcoxon, t-test, NB-GLM, LR, and bimod). Notably, ClusterDE consistently identified zero DE genes in all five datasets when using Wilcoxon, leading us to designate Wilcoxon as the default DE test for ClusterDE.

Fig.3: ClusterDE achieved reliable FDR control and good power in identifying cell-type marker genes from real scRNA-seq data.

a, UMAP visualizations of target data (left) and synthetic null data (right) of five cell lines. b, Numbers of DE genes (at the target FDR of 0.05) identified by ClusterDE and the three other methods (Seurat, Countsplit, and TN test). While the three other methods found thousands of DE genes—false positives when interpreted as cell-type marker genes—within a single cell line, ClusterDE made no false discoveries when used with most DE tests. The numbers in black and white indicate the number of DE genes and the proportions of DE genes among all genes, respectively. The five DE tests are the Wilcoxon rank-sum test (Wilcoxon), t-test, negative binomial generalized linear model (NB-GLM), logistic regression model predicting cluster membership with likelihood-ratio test (LR), and likelihood-ratio test for single-cell gene expression (bimod). c, UMAP visualizations of target data (left) and synthetic null data (right) for four datasets containing two monocyte subtypes: CD14⁺ monocytes and CD16⁺ monocytes. The synthetic null data captured the global topology of cells in the target data while filling the gap between the two cell subtypes. The dashed box labels the dataset used in panels f and g. d, ClusterDE identified DE genes between the two cell subtypes. The numbers in black and white indicate the number of DE genes and the proportions of DE genes among all genes, respectively. e, The ranks of two exemplary genes (a monocyte subtype marker FCGR3A in red and a well-known housekeeping gene B2M in blue) in the DE gene lists of ClusterDE and Seurat across the five DE tests and the four datasets in c. In each boxplot representing the distribution of 20 ranks, the center horizontal line represents the median, and the box limits represent the upper and lower quartiles. f, The top DE genes identified by ClusterDE exhibited distinct expression patterns in the two cell clusters identified by Seurat clustering, a phenomenon not observed for the top DE genes identified by Seurat. For ClusterDE and Seurat, the top DE genes are defined as the common DE genes found by the five DE tests in d at the target FDR of 0.05. The UMAP plots show each top DE gene’s normalized expression levels in the dataset “Rep2 10x(V2)” (marked by the dashed box in c; see“Dimensionality reduction and visualization” in the Supplementary Material). The density plots depict each top DE gene’s normalized expression distributions in the two cell clusters. g, Gene set enrichment analysis (GSEA) of the ranked DE gene lists identified by ClusterDE and Seurat with five DE tests from the dataset Rep2 10x(V2). The red lines represent the enrichment of the CD14⁺/CD16⁺ monocyte marker gene set, and the blue lines represent the enrichment of the housekeeping gene set. The normalized enrichment score (NES) reflects the direction and magnitude of enrichment, and the P value indicates the significance of enrichment.

In the second application, we analyzed eight PBMC datasets of CD14⁺/CD16⁺ monocytes [38] to evaluate ClusterDE’s ability to detect known or potential marker genes for these two monocyte subtypes. The datasets were generated from two technical replicates using four UMI-based scRNA-seq protocols (10X Genomics Versions 2 and 3, Drop-seq, and inDrop). After applying the default Seurat clustering to identify two clusters in each dataset, we found that four datasets exhibited relatively accurate clustering results (ARI > 0.5; Fig. 3c, left; Fig. S12, top), while the other four showed poor agreement between clusters and the two monocyte subtypes (ARI < 0.2; Fig. S12, bottom). We anticipated that an effective post-clustering DE method would be able to detect meaningful marker genes for monocyte subtypes in the first four datasets with high-quality clusters, but not necessarily in the latter four datasets with low-quality clusters. The reason is that low-quality clusters do not accurately reflect the true distinction between the two cell subtypes, making it unreasonable to expect reliable subtype marker genes to be identified by comparing these clusters. When applied to the latter four datasets with low-quality clusters, Seurat, Countsplit, and the TN test often identified hundreds to thousands of DE genes (Fig. S13). In contrast, ClusterDE did not identify any DE genes in most cases (Fig. S13), yielding a more reasonable result.

On the first four datasets with high-quality clusters (Fig. S12, top), which align well with the two monocyte subtypes, we expected post-clustering DE genes to reveal cell-subtype markers. As a sanity check, we verified that ClusterDE’s synthetic null data resembled the target data but filled the “gap” between CD14⁺ and CD16⁺ monocytes, representing a single “hypothetical” cell type in each dataset (Fig. 3c, right). When applied to these four datasets, ClusterDE with Wilcoxon identified between 55 and 173 DE genes (corresponding to 1–4% of all genes) at the target FDR of 5%. In contrast, Seurat and Countsplit identified between 1 and 1,288 DE genes (0–26% of all genes), while the TN test consistently identified at least 1,187 DE genes (25% of all genes) (Fig. 3d). Given the knowledge that the two monocyte subtypes are not drastically different, identifying thousands of potential marker genes seemed implausible. Thus, the number of DE genes identified by ClusterDE appeared more reasonable and aligned with biological expectations.

Examining the post-clustering DE genes identified by ClusterDE and Seurat across the five DE tests on the four datasets (resulting in 20 DE gene lists for each method), we found that ClusterDE more effectively distinguished known subtype marker genes from housekeeping genes compared to Seurat. For each dataset, both methods performed post-clustering DE analysis on the same two clusters identified by Seurat’s default clustering, with ClusterDE acting as an add-on to address the double-dipping issue in Seurat’s post-clustering DE results. The distinction in the two methods’ results was evident in the ranking of specific genes within the DE gene lists. For instance, we focused on FCGR3A (CD16), a canonical marker for distinguishing CD14⁺ monocytes from CD16⁺ monocytes, and B2M, a widely recognized housekeeping gene expressed across various cell types [39]. Notably, ClusterDE consistently ranked FCGR3A among its top DE genes (typically ranked between 1 and 10) while placing B2M consistently low in its DE gene lists (generally ranked below 1,000) (Fig. 3e, top). In contrast, Seurat ranked the two genes similarly (with ranks between 10 and 100), making it difficult to discern which gene was more likely to be a subtype marker without prior knowledge (Fig. 3e, bottom). Using the dataset “Rep2 10x(V2)” as an example, we examined the five most frequently identified post-clustering DE genes (determined based on the top 50 DE genes identified by each of the five DE tests) by ClusterDE and Seurat (Fig. 3f). Our analysis revealed that the five genes identified by ClusterDE displayed more distinct distributions of normalized expression levels between the two clusters compared to the five genes identified by Seurat, suggesting that the genes identified by ClusterDE are more likely to be subtype markers (Fig. 3f).

Further, we performed gene set enrichment analysis (GSEA) to evaluate the enrichment of two gene sets—known CD14⁺/CD16⁺ monocyte markers and housekeeping genes—in the post-clustering DE gene lists identified by ClusterDE and Seurat. GSEA revealed a strong enrichment of monocyte subtype markers among the top-ranked DE genes identified by ClusterDE, with a clear distinction from housekeeping genes (Fig. 3g, top). In contrast, monocyte subtype markers exhibited a weaker enrichment in Seurat’s top-ranked DE genes and showed a similar enrichment pattern to that of housekeeping genes, making it difficult to distinguish the two gene sets in Seurat’s ranked DE gene list (Fig. 3g, bottom). GSEA results from the other three datasets further confirmed that ClusterDE consistently outperformed Seurat in distinguishing monocyte subtype markers from housekeeping genes (Fig. S14).

Considering the common practice of using only the top k DE genes (e.g., k = 100) for further investigation, we summarized the numbers of monocyte subtype markers and housekeeping genes among the top k = 1 to 100 DE genes identified by ClusterDE and Seurat across the five DE tests on the four datasets. Fig. S15 shows that ClusterDE consistently identified more monocyte markers and fewer housekeeping genes among the top DE genes compared to Seurat. To further explain why ClusterDE better distinguished monocyte markers from housekeeping genes, we used the minus-average (MA) plots [40] to illustrate the role of the synthetic null data as a contrast to exclude housekeeping genes from the top DE genes. In the MA plots (Fig. S16), we observed that four exemplary housekeeping genes (ACTB, ACTG1, B2M, and GAPDH, marked in blue in Fig. S16) exhibited both large target DE scores and large null DE scores, resulting in close-to-zero contrast scores. Consequently, these genes were excluded from the top DE genes by ClusterDE. In contrast, Seurat identified these housekeeping genes as top DE genes solely based on their large target DE scores. On the other hand, we examined four exemplary monocyte markers (CD14, FCGR3A, MS4A7, and LYZ, marked in red in Fig. S16) and found that they had large target DE scores but small null DE scores, allowing ClusterDE to correctly identify them as top DE genes.

In the third application, we used a scRNA-seq atlas of the adult Drosophila neuronal visual system [41] to demonstrate ClusterDE’s usage for guiding the merging of spurious clusters caused by over-clustering. The original study clustered the data using the Louvain algorithm at a resolution of 10, resulting in 208 clusters, and identified spurious neuron clusters using a random forest-based method in Seurat. We applied ClusterDE to three pairs of spurious neuron clusters flagged as over-clustered and subsequently merged in the original study (Fig.4a). For all three pairs, ClusterDE identified zero DE genes, supporting the original merging decisions, while Seurat (using Wilcoxon as the DE test) identified hundreds of DE genes at the target FDR of 5% (Fig.4b). The gating plots, commonly used in single-cell literature to assess the separation of clusters based on marker gene expression, also indicate that each of the three pairs of neuron clusters should be merged (Fig. 4c), consistent with the conclusion that these pairs of clusters were spurious. Additionally, we examined three specific genes—alrm, nrv2, and Gs2—that Seurat identified as DE in at least two of the three pairs of neuron clusters. These genes are glial cell markers [42, 43], unlikely to be expressed in neurons, and more plausibly attributed to contamination from ambient glial RNA. Hence, ClusterDE successfully identified cases of over-clustering and avoided false-positive marker gene identifications between spurious clusters.

Fig.4: Application of ClusterDE to the adult Drosophila neuronal atlas.

a, t-SNE visualization of the data with highlighted cell types: T1 (Clusters 1 and 2), Tm3 (Clusters 126 and 127), Tm1 (Clusters 140 and 141), and Tm2. b, Numbers of DE genes (at the target FDR of 0.05) identified by ClusterDE and Seurat (with Wilcoxon as the DE test). c, Gating plots comparing Clusters 1 vs. 2 (left), Clusters 126 vs. 127 (middle), and Clusters 140 vs. 141 (right). d, Boxplot showing accuracy achieved by RF and SVM models using the 105 DE genes found by ClusterDE, the top 105 genes (ordered by increasing P value) found by Seurat only, and 105 random genes. A two-sided t-test comparing the accuracy obtained using the DE genes from ClusterDE vs the DE genes from Seurat only shows a significant difference (P value < 2.2 × 10⁻¹⁶,n = 100 samples each) for both RF and SVM models. e, Heatmap of the 105 DE genes found by ClusterDE. f, Heatmap of the top 105 DE genes found by Seurat only. In both e and f, genes are ordered by hierarchical clustering as implemented by R function <monospace>heatmap.2</monospace>.

We further analyzed the Tm1 and Tm2 cell types in the same dataset to assess the reliability of cell-type marker genes identified by ClusterDE and Seurat. ClusterDE identified 105 DE genes, while Seurat identified 768, including the 105 genes found by ClusterDE and an additional 663 genes. To evaluate the ability of these genes to distinguish the two cell types, we trained binary classifiers using random forest and support vector machine with three gene sets as features: the 105 DE genes identified by ClusterDE, the top 105 DE genes uniquely identified by Seurat (ranked by p-value), and a random selection of 105 genes as a baseline. The binary classifiers were tested on held-out cells (20% of the dataset), revealing significantly higher classification accuracy with genes identified by ClusterDE compared to those from Seurat (P value < 2.2 × 10⁻¹⁶, two-sided t-test, n = 100 train-test splits). Moreover, the accuracy improvement over the random baseline was substantially greater for ClusterDE’s DE genes. Gene expression heatmaps further supported this finding, showing that DE genes identified by ClusterDE exhibited more distinct expression differences between Tm1 and Tm2 cell types compared to those identified solely by Seurat. GSEA revealed that the DE genes identified by ClusterDE were significantly enriched in pathways specific to the visual system, including G protein-coupled receptor (GPCR) activity and eye development pathways, consistent with the known roles of Tm1 and Tm2 cell types. Furthermore, the enrichment of “positive regulation of response to stimulus” in Tm2 aligns with the knowledge that Tm1 exhibits a delayed response compared to Tm2 [44]. In contrast, DE genes uniquely identified by Seurat were enriched in more general pathways applicable to many cell types (Fig. S17). These analyses demonstrate that ClusterDE identifies more biologically relevant and reliable cell-type marker genes.

2.4. ClusterDE reveals reliable spatial-domain marker genes

In spatial domain detection and annotation, spatial clustering (e.g., using BayesSpace [45]) is initially applied to SRT data to infer putative spatial domains (referred to as “spatial clusters”). Then, DE tests are conducted to identify potential domain marker genes, leading to a double-dipping issue similar to that in scRNA-seq data analysis. We define “spatial-domain marker genes” as those exhibiting discontinuous expression changes between adjacent domains while being more highly expressed in one domain compared to the others (Fig.S18). In contrast, other genes may still display spatial variation but change smoothly across adjacent domain boundaries (Fig.S18).

Both scRNA-seq and spatial clustering rely on the gene expression profiles of cells or spots; however, spatial clustering algorithms also incorporate the spatial coordinates of spots. This incorporation creates spatial dependency among spots within clusters, which ClusterDE accounts for in the generation of synthetic null data (see “Supplementary Methods” in the Supplementary Material). Using both simulated and real data, we demonstrated how ClusterDE effectively resolves the double-dipping problem in post-spatial clustering DE analysis, facilitating the identification of reliable and interpretable spatial-domain marker genes.

We first simulated an SRT dataset with a single spatial domain (see “Supplementary Methods” in the Supplementary Material) and applied ClusterDE alongside a common SRT post-clustering DE analysis pipeline, which included the popular spatial clustering algorithm BayesSpace [45] to infer spatial domains, followed by two commonly used DE tests—the Wilcoxon rank-sum test and t-test in Seurat—to identify spatial-domain marker genes (referred to as BayesSpace+Seurat). Since the dataset contained only one domain, no true domain marker genes should be detected by this post-clustering DE analysis. However, only ClusterDE met this expectation, whereas BayesSpace+Seurat identified hundreds of false-positive spatial-domain marker genes, failing to control the target FDR of 0.05 (Fig. S19a). Further examination of the P values obtained from BayesSpace+Seurat revealed an anti-conservative bias, whereas the corresponding contrast scores from ClusterDE exhibited symmetry around zero, satisfying the symmetry requirement (Fig. S19b), which explained the observed differences in FDR control. In summary, in this single spatial domain scenario, only ClusterDE successfully controlled the FDR and avoided reporting spurious spatial-domain marker genes.

We next simulated two datasets with multiple domains: (1) a two-domain dataset containing 200 pre-specified true domain marker genes and (2) a three-domain dataset containing 200 and 168 pre-specified true domain marker genes for each pair of adjacent domains. These true marker genes defined distinct domains with clear boundaries in the target datasets; these domains were approximately captured by spatial clusters (Fig. S19c for two domains; Fig. 5a for three domains). Since ClusterDE compares two spatial clusters at a time, we analyzed each pair of adjacent clusters separately in the three-domain case—specifically, Layer 5 vs. Layer 6 and Layer 6 vs. WM. In both the two- and three-domain scenarios, BayesSpace+Seurat failed to control the target FDR of 0.05 (Fig. S19d for two domains; Fig. S20 for three domains). In contrast, ClusterDE successfully controlled the FDR while maintaining high power to detect most true domain marker genes in both datasets (Fig. S19d for two domains; Fig. S20 for three domains). To further assess the reliability of the identified spatial-domain marker genes, we visualized the top three genes identified by ClusterDE and BayesSpace+Seurat, respectively. The top genes detected by ClusterDE exhibited sharp expression changes at the spatial domain boundaries, whereas those identified by BayesSpace+Seurat showed no such distinct changes (Fig. 5b). In conclusion, these simulation experiments demonstrated that ClusterDE effectively controlled FDR while reliably identifying spatial-domain marker genes.

Fig.5: ClusterDE mitigates double dipping and identifies reliable spatial-domain marker genes in SRT post-clustering DE analysis.

ClusterDE demonstrates robust statistical power in identifying spatial-domain marker genes from both a simulated SRT dataset containing three domains and the PDAC-B cancer slice from the human primary pancreatic cancer (PDAC) SRT dataset. a, Visualization of annotated spatial domains alongside the spatial expression patterns of three representative principal components (PCs) among the top five PCs in the target data (simulated) and synthetic null data. b, Spatial expression patterns of top domain marker genes. The first and second rows display the expression of top domain marker genes of Layer 5 and Layer 6, respectively, detected by ClusterDE and BayesSpace+Seurat (BayesSpace for clustering and Seurat for DE). c, Visualization of annotated domains alongside the spatial expression patterns of three representative PCs among the top five PCs in the target data (PDAC-B) and synthetic null data. d, Spatial expression patterns of top domain marker genes. The first and second rows display the expression of top domain marker genes of the Duct Epithelium and Interstitium, respectively, detected by ClusterDE and BayesSpace+Seurat.

We applied ClusterDE to real spatially resolved transcriptomics (SRT) datasets to verify its performance in identifying spatial-domain marker genes, analyzing slices from the human dorsolateral prefrontal cortex (DLPFC) dataset [46] and the human primary pancreatic cancer (PDAC) dataset [47].

In the DLPFC dataset, we first applied ClusterDE and BayesSpace+Seurat to Layers 1, 4, and 5. Based on manual annotations, each of these layers represents a homogeneous domain; therefore, any detected DE genes within a layer would be false-positive domain marker genes. Using BayesSpace, we identified two clusters within each layer and applied both methods to compare these clusters. ClusterDE identified nearly zero DE genes, while BayesSpace+Seurat detected hundreds to thousands of DE genes (Fig. S21). This result highlighted the double-dipping issue and demonstrated ClusterDE’s effectiveness in addressing it.

Next, we used BayesSpace (in its default setting, without constraining the number of clusters per layer) to perform spatial clustering across the entire DLPFC slice. Several layers (annotated spatial domains) were divided into smaller clusters. Focusing on Layers 3 and WM, we applied ClusterDE and BayesSpace+Seurat to pairs of clusters within each domain. In both cases, ClusterDE did not identify DE genes, supporting the merging of these clusters and demonstrating its ability to correct over-clustering (Fig. S22). In contrast, BayesSpace+Seurat identified DE marker genes between most pairs of clusters, potentially leading to the misidentification of spurious spatial domains due to over-clustering. These findings demonstrate that ClusterDE effectively avoids identifying spurious domain marker genes and prevents the annotation of false spatial domains, whereas BayesSpace+Seurat, due to double-dipping issues, cannot mitigate these problems.

In the PDAC dataset (Fig. 5c), we examined pairs of adjacent spatial domains (duct epithelium vs. interstitium, and interstitium vs. cancer regions) to evaluate the quality of spatial-domain marker genes identified by ClusterDE and BayesSpace+Seurat. The top marker genes detected by ClusterDE exhibited clearer domain-specific patterns compared to those identified by BayesSpace+Seurat (Fig. 5d). For instance, the gene CRP, identified by ClusterDE, was exclusively expressed in the duct epithelium region. In contrast, the top marker genes identified by BayesSpace+Seurat did not align with specific domains, instead showing smooth expression changes across the entire slice. We also examined the DE ranks of four known marker genes and two housekeeping genes. ClusterDE ranked the known marker genes VMA1, POSTN, SFRP2, and F3 higher and assigned lower ranks to the housekeeping genes TMSB10 and S100A6 compared to BayesSpace+Seurat (Fig. S23). These results underscore ClusterDE’s effectiveness in detecting spatial-domain marker genes that represent spatial domains by exhibiting sharp expression changes at domain boundaries.

3. Discussion

In conclusion, ClusterDE effectively addresses the double-dipping issue in post-clustering DE analysis of scRNA-seq and SRT data. One practical consideration is the scalability of ClusterDE, which primarily depends on the speed of synthetic null data generation. Our results show that this process can be completed within a reasonable timeframe (Fig. S24). Notably, the generation process is fully parallelized, enabling significant reductions in computational time with the use of additional CPUs (Fig. S24).

Another consideration is the power of ClusterDE, which is influenced by the conservative nature of the synthetic null data. The null model assumes a single cell type or spatial domain, which is necessary for controlling over-clustering when the target data contains only one homogeneous cell type or spatial domain. However, when the target data consists of two distinct cell types or spatial domains, this conservative null can lead to power loss. Despite this limitation, the goal of ClusterDE is to annotate distinct cell types or spatial domains, and the top DE genes identified by ClusterDE are typically sufficient for this purpose. Improving the power of ClusterDE while maintaining control of false discoveries is an important area for future research.

ClusterDE adapts well to a wide range of clustering algorithms and DE tests. Through extensive simulation studies and real data analyses, we demonstrated that ClusterDE effectively avoids false discoveries caused by double-dipping and identifies biologically meaningful cell-type or spatial-domain marker genes. For post-clustering DE analysis involving more than two clusters, we recommend using ClusterDE in a stepwise manner, potentially following a hierarchical organization of clusters based on their similarities (Fig. S25). In this approach, users compare pairs of ambiguous clusters at each step, using identified DE genes to determine whether the clusters are biologically meaningful and should remain distinct. For SRT data with multiple spatial clusters, this stepwise method would involve comparisons between spatially adjacent clusters.

Finally, while ClusterDE addresses the double-dipping problem specifically in post-clustering DE analysis, the concept of synthetic null data (in silico negative controls) has broader potential applications. It could be extended to other analyses affected by double-dipping, such as post-pseudotime DE analysis [48] and data integration analyses. Since double-dipping is an almost inevitable challenge in single-cell and spatial omics data analysis, we propose a general strategy to reduce false discoveries by implementing synthetic null data and applying a contrastive approach. This strategy can help achieve more reliable findings across various analyses.

Data Availability

All datasets used in the study are publicly available. The pre-processed datasets are available at https://figshare.com/articles/dataset/ClusterDE_datasets/23596764.