Reference-informed evaluation of batch correction for single-cell omics data with overcorrection awareness

Abstract

Batch effect correction (BEC) is fundamental to integrate multiple single-cell RNA sequencing datasets, and its success is critical to empower in-depth interrogation for biological insights. However, no simple metric is available to evaluate BEC performance with sensitivity to data overcorrection, which erases true biological variations and leads to false biological discoveries. Here, we propose RBET, a reference-informed statistical framework for evaluating the success of BEC. Using extensive simulations and six real data examples including scRNA-seq and scATAC-seq datasets with different numbers of batches, batch effect sizes and numbers of cell types, we demonstrate that RBET evaluates the performance of BEC methods more fairly with biologically meaningful insights from data, while other methods may lead to false results. Moreover, RBET is computationally efficient, sensitive to overcorrection and robust to large batch effect sizes. Thus, RBET provides a robust guideline on selecting case-specific BEC method, and the concept of RBET is extendable to other modalities.

Reference-informed statistical method provides robust guidance on case-specific selection of batch effect correction methods for single-cell omics data with awareness to over-correction, and facilitates biologically meaningful insights from data

Introduction

Single-cell sequencing provides an unprecedented opportunity to define molecular cell type and cell state in a data-driven fashion, and has achieved great accomplishment in exploring the mechanism of different diseases at the cell level^1–3. Researchers may take the advantage to investigate in-depth biological insights by integrating massive datasets, either publically available or generated in-house, for multiple downstream analyses, such as cell type and state annotation, differential gene expression, trajectory inference, cell-cell communication, etc. But the underlying difficulty is the large variations or batch effect in these datasets, which are collected from different labs, experiments, capturing times, handling personnel, and even technology platforms^4,5. These differences confound the true biological variations during data integration, which need to be removed by batch effect correction (BEC) tools to ensure biologically meaningful results^6–8. Indeed, various BEC methods have been proposed in recent years, and a recent benchmark study compared 14 BEC methods in terms of batch mixing, accuracy of cell type annotation, computational efficiency for large datasets, etc., and recommended Harmony, LIGER and Seurat as top performers⁹.

However, these benchmark studies lack comparisons on key metrics that could impact routinely performed downstream analyses such as trajectory inference and cell-cell communication. First, the unwanted technical variations between samples from different batches should be effectively removed after BEC, both locally and globally. However, existing BEC evaluation tools such as kBET¹⁰ or LISI⁷, may have a lower power in cases where only parts of cells have batch effect. Second, these tools lack algorithms to estimate overcorrection, that is, the degradation of true biological information while eliminating technical bias¹⁰. Overcorrection makes downstream analyses highly problematic and often leads to wrong conclusion. Third, BEC methods should be evaluated for capability to cope with large batch effect sizes. Previous benchmark studies ultilized datasets with limited batch numbers (2–5) and arguably not sufficient batch effect sizes, while these are possibly several order of magnitude higher in real practice, given the overwhelming availability of scRNA-seq datasets¹¹. Thus, a comprehensive and reliable test metric taking full consideration of the aforementioned scenarios is an unmet need for BEC applications.

In this paper, we propose a novel statistical method, Reference-informed Batch Effect Testing (RBET) for evaluating the performance of BEC tools. This method takes advantage of reference gene (RG) expression/variation pattern and maximum adjusted chi-squared (MAC) statistics for two-sample distribution comparison (Fig. 1A). In both simulated and real data, RBET demonstrates a superior performance in detecting batch effects, with overcorrection awareness, large batch effect robustness and high computational efficiency, as compared to kBET and LISI. More importantly, RBET excels in prioritizing BEC tools that feed downstream analyses consistent with prior biological knowledge (Fig. 1C).

Fig. 1. Study overview.

A Overview of RBET. Single-cell sequencing data from different batches are integrated using various batch effect correction (BEC) tools. RBET first selects reference genes (RGs) either based on literature or calculated from data, and then performs batch effect testing (BET) on the selected data using MAC statistics, and finally chooses the BEC method with the smallest RBET value. B The choices of RBET are validated through downstream analyses including cell annotation, trajectory inference and cell-cell communication. C Summary of the performance of RBET and its competitors LISI and kBET in five scRNA-seq datasets and one scATAC-seq dataset (mouse brain). RBET consistently makes optimal choices under all the circumstances, while LISI and kBET only performs well in three examples.

Results

Overview of RBET framework

Motivated by the consistent expression patterns of housekeeping genes across various cell types under diverse conditions^11–14 (see also Fig. S1; “Methods”), we assume that the integrated data should have no batch effect on genes with similar characteristics both locally and globally, termed reference genes (RGs) (see “Methods”). Based on this assumption, RBET framework for evaluating the performance of different BEC methods is comprised of two steps (Fig. 1A). Step 1 selects RGs specific to each dataset, with two strategies available. We collected existing validated tissue-specific housekeeping genes as RGs from published literature.For cases where validated tissue-specific housekeeping genes were not available, we directly selected RGs from the datasets, assuming that RGs should be stably expressed both within and across phenotypically different clusters (see “Methods”). By default, we used the first strategy in this step. Step 2 detects batch effect on these RGs in the integrated dataset. Noting that testing the batch effect between two samples equals to comparing their underlying distributions in a high-dimensional setting, we proposed to map the dataset into a two-dimensional space using UMAP¹², and used our previously designed MAC statistics¹³ for batch effect detecting (see “Methods”).

We next compared the performance of RBET with its competitors, LISI and kBET, by comprehensive simulations, and then validated the performance of RBET through real data downstream analyses, including cell annotation, trajectory inference and cell-cell communication (Fig. 1B). Note that a smaller RBET and kBET value indicates a better BEC performance, while a larger LISI value signals a better one. Since most scRNA-seq downstream analyses require full gene expression matrices, we focused on six BEC tools capable of returning full dimensional data, including Seurat¹⁴, scanorama¹⁵, scMerge¹⁶, limma¹⁷, Combat¹⁸ and mnnCorrect¹⁹ (Fig. 1A). Tools like Harmony⁷ and fastMNN²⁰, which only output low-dimensional embedding, were not further explored in this study.

RBET substantially outperforms its competitors and is sensitive to overcorrection

We first compared the batch effect detection performance of RBET, kBET and LISI under two simulation strategies: (1) Gaussian examples with different means or covariance structures modeling different patterns of batch effect as a toy model (Fig. S2A–S2C); and (2) examples with simulated gene expression level mimicking real data under different cell type numbers and batch effect sizes (Fig. S3A–S3B). In our simulations, the power of each method was evaluated from 100 independent repetitions under significance level of 0.05 (“Methods”).

In Gaussian examples, RBET showed comparable performance with kBET and LISI with different means or covariance structures (Fig. S2A–S2C). When turning to examples with simulated gene expression data, RBET outperformed LISI in terms of detection power, while kBET lost control over type I error across both single and multiple cell types (Fig. 2A, B). Moreover, RBET topped the computational efficiency test, demonstrating its potential to scale up for high batch number in big dataset (Fig. 2C). Since such scaled scenario may introduce larger batch effect sizes, we tested RBET, kBET and LISI for their variability using the coefficient of variation (CV) by artificially increasing the effect size (“Methods“). Strikingly, RBET remained its variation across full size range, while the variations of LISI and kBET collapsed into zero when batch effect size was large, indicating their reduced discrimination capacity in datasets with strong batch effect (Fig. 2D). In real data, the batch effect may ocurr in only parts of cell types²¹. Thus, we also evaluated the performance of these metrics under conditions with partial batch effects, and RBET achieved higher detection power while maintaining control over type I error (Fig. S4).

Fig. 2. RBET substantially outperforms competitors and is sensitive to overcorrection.

A–D RBET outperforms its competitors kBET and LISI in the simulated gene expression example. The power of RBET, kBET and LISI under different batch effect sizes in dataset with (A) one cell type and (B) two cell types. The gray dotted line presents the theoretical power (0.05) when the null hypothesis is true, i.e., there is no batch effect. C The computational time used for 20 repetitions under two settings: (a) one cell type; (b) two cell types. D The variability of different methods measured by the coefficient of variation (CV) under different batch effect sizes. When its CV equals 0, the method has no discriminative power in BEC evaluation. E, F RBET is sensitive to overcorrection while LISI and kBET are not. We simulate a complex dataset containing 2 batches of cells, each with 9 cell types, which are B cell, CD14+ monocyte, CD16+ monocyte, CD4 + T cell, cytotoxic T cell, dentritic cell (DC), megakaryocyte (MK), natural killer cell (NK), and plasmacytoid dentritic cell (pDC). BEC is performed by Seurat V5 with different numbers of anchors (k). E UMAP plots show examples of simulated data before (k = 0) and after (k = 1, 3, 200) BEC. F The values of RBET, kBET and LISI with the change of k.

Importantly, we investigated the performance of RBET, LISI and kBET when facing overcorrection of batch effects. We noted that some BEC algorithms, i.e., Seurat V5, use nearest neighbor as foundation for BEC, and increasing the number of neighbors used for correction, i.e., increasing the number of anchors (k), may lead to the loss of variation in gene expression and true cell type information, potentially causing serious overcorrection. Indeed, when k increased from 1 to 200 in Seurat, CD14+ monocytes were erroneously divided into two clusters, and pDCs were incorrectly merged with a subset of cytotoxic T cells (Fig. 2E). Additionally, RGs exhibited a loss of expression variation, which should have remained stable before and after BEC (Fig. S3C). Accordingly, we found that RBET value decreased gradually in the first phase until k reached 3 (minimal value), while a further increase of k led to gradual increase of RBET value at the second phase, which coincided with an increased level of overcorrection (Fig. 2F). Such biphasic change was not seen for kBET and LISI. This highlighted the importance of incorporating RGs in RBET.

To sum up, RBET was favored in multi-scenario simulated datasets in terms of detecting power, type I error control, computational efficiency, robustness to batch effect, and sensitivity to overcorrection.

RBET outperforms competitors in terms of cell annotations

To benchmark RBET in real data analysis, we started from the pancreas dataset which included 3 technical batches (Celseq, Celseq2 and SMARTseq2) and 13 cell types with varied numbers of cells in each cell type (7–2065)²² (Fig. 3A). To evaluate RBET performance, we collected experimentally validated housekeeping genes specific to pancreas as candidate RGs²³, and checked whether they were differentially expressed across batches (Table S1, see “Methods”). Cells from different batches had strong batch effect, as evidenced by not well mixed clusters in UMAP (Fig. 3B), as well as the large values of RBET and kBET and a small value of LISI (Fig. 3C). After data integration by six BEC tools, both RBET and kBET selected Seurat as the best method, while LISI favored scanorama (Fig. 3C). The results of methods not selected by any evaluation tool were given in Fig. S5A.

Fig. 3. RBET selects the optimal BEC method in pancreas data validated by high cell annotation accuracy.

A Histogram of cell type composition in each batch. B UMAP visualization of original dataset colored by batches (up) and cell types (down). C Evaluation scores of RBET, LISI and kBET for different BEC methods, with their optimal choices colored in red, blue and green, respectively. D UMAP visualization of selected integrated datasets colored by batches (up) and cell types (down). E The clustering quality (measured by Silhouette Coefficient) and the annotation precision (measured by accuracy between inferred cell annotations and real cell tags) of different BEC methods. Note that accuracy ranges in [0,1], while SC ranges in [–1,1], and a larger value indicates a better result.

As noted, scanorama clusters were not well mixed by batches (Fig. 3D). Thus, we further quantified the clustering quality using Silhouette Coefficient (SC; “Methods”). SC ranges form -1 to 1, where values closer to 1 indicate better-defined and well-separated clusters. Seurat received a much higher SC score than scanorama (Fig. 3E), indicating better clustering performance from Seurat. Next, we utilized cell annotation to further validate the selections. The cell types were annotated by ScType²⁴ with the help of marker genes²⁵. We compared cell annotation results from different BEC methods with real cell tags using accuracy (ACC), Adjusted Rand Index (ARI) and Normalized Mutual Information (NMI) (“Methods”). Both Seurat and scanorama obtained high scores exceeding 0.9 for ACC, ARI and NMI, with Seurat outperforming scanorama in all metrics (Fig. 3E, S5B). Overall, Seurat demonstrated superior annotation precision and clustering quality, confirming the selection of RBET and kBET.

RBET outperforms competitors in terms of trajectory analysis

Trajectory analysis infers dynamic process of cell differentiation and state transition, and provides great insights into the biological phenomena^26,27. In this section, we validated the performance of RBET for multi-lineage CD8 + T cell development^28,29, single-lineage monocyte²⁷ and embryo dataset³⁰ with literature-based RGs given in Table S2-S4.

The CD8 + T cell dataset was extracted from ovarian, lung and colorectal cancers, including 40 samples and 16,133 cells (Fig. 4A, B, S6A)¹¹. We found that RBET and LISI both preferred Seurat, but kBET gave a constant value of 1 to all the BEC tools (Fig. 4C), suggesting a failure in evaluating dataset with high batch effect size, consistent with our simulation results (Fig. 2D). Results for other methods were given in Fig. S6. To nail down the ideal BEC tool for downstream trajectory analysis, we took advantage of a top rated tree-based tool for trajectory inference, Slingshot²⁶. This dataset consisted of two common CD8 + T cell lineages, CD8+ Terma (cytotoxic) and CD8+ Tex (exhuastic), which are both differentiated from CD8+ Tn (naïve) cells as previously validated¹¹. Seurat, limma, Combat and scanorama correctly identified these two lineages using Slingshot, while lineages derived from scMerge and mnnCorrect were not consistent with CD8 + T cell biology (Fig. 4D, S6B). Since this T cell dataset was derived from different cancer types, their cell distribution along the pseudotime trajectory may vary accordingly, but they should still more or less reflect the cell fraction characteristics of T cell subtype in certain cancer (Fig. 4E, S6C). For example, colorectal cancer had relatively less Temra cells, and ovarian cancer had less Tex cells (Fig. 4A), which was correctly reflected in Seurat trajectory but not in limma and mnnCorrect (Fig. 4E, S6C, note the last peak at the end of pseudotime for each lineage). We then searched for the expression patterns of classical marker genes along the lineage-dependent pseudotime of CD8 + T cell differentiation: (1) a gradual decrease of Tn marker CCR7 and a gradual upregulation of activation marker NKG7 along the trajectories; (2) the terminal stage marker genes FGFBP2 for Terma and PDCD1, HAVCR2 for Tex, gradually increase along Temra and Tex lineages, respectively; (3) the Tem marker GZMK expresses similarly high in the middle of both trajectories, since Tem is a common progenitor state before differentiation into Temra and Tex stages¹¹. According to these expected gene expression patterns, trajectories derived from Seurat generated acceptable results (Fig. 4F), and RBET’s top choice Seurat outperformed other tools for this complex dataset in terms of marker gene consistency and cell distribution consistency (Fig. 4G). Interestingly, CD8+ Tex lineage is the known target for anti-PD-1/PD-L1 treatment³¹, and PDCD1 that encodes PD-1 protein expressed relatively higher in Tex lineage for lung cancer as compared to other cancers in Seurat processed trajectory (Fig. 4H), which correctly reflected current superior clinical outcome of anti-PD-1/PD-L1 treatment for lung cancer.

Fig. 4. RBET chooses the optimal BEC method for CD8 + T cell dataset validated by trajectory analysis.

A Histogram of cell type composition per cancer type. B UMAP visualization of original dataset colored by batches (up) and cell types (down). C Evaluation scores of RBET, LISI and kBET for different BEC methods, with their optimal choices colored in red, blue and green, respectively. kBET lose its power by giving the same value to all methods. D UMAP visualization of selected integrated datasets colored by batches (up) and cell types (down), together with inferred trajectories (down). E Density plots for colorectal cancer (CRC), lung cancer and ovarian cancer along the two trajectories. F The expression profiles of 6 marker genes along the pseudotime, split by lineages, solid lines for Temra lineage and dashed lines for Tex lineage. G The consistency of the expression profile across batches for 6 marker genes in (F) (up), and the consistency of the expression profile across lineages for 3 marker genes (CCR7, NKG7 and GZMK) (down). A smaller value indicates better consistency. H Expression profiles of PDCD1 for CRC, lung cancer and ovarian cancer along the Tex lineage.

In addition, RBET also chose the optimal BEC methods for single-lineage monocyte and embryo datasets (Fig. S7–S10). In summary, using different examples with varying numbers of batches, sizes of batch effect and numbers of cell types, we concluded that (1) kBET, LISI and RBET selected different BEC methods as the optimal one; (2) the BEC method selected by RBET gave optimal results on trajectory analysis. Thus, if we used LISI or kBET to guide the selection, we may get misleading results on trajectory, hindering our understanding in biology system.

RBET outperforms competitors in terms of cell-cell communication analysis

To investigate the influence of BEC on cell-cell communication analysis, we focused on the testicular data from healthy males^32,33, which contained two batches that covered all the developmental stages from spermatogonial stem cells (SSCs) to mature spermatozoa. With almost identical composition of cell types, the batch influence was mainly in late stage (Fig. 5A, B). This batch effect was still retained after limma and Combat correction as their sperm cells were still split by batches, while other four methods gave acceptable batch correction on UMAP (Fig. 5D, S11A). To facilitate RBET, 12 housekeeping genes were curated from literature and four of them were selected as RGs (Table S5). RBET, LISI and kBET selected scanorama, mnnCorrect and Combat, respectively (Fig. 5C). To make further comparison, we dived into downstream analyses including trajectory inference and cell-cell communication analysis. A single developmental trajectory from SSCs to sperm cells was seen from scanorama and mnnCorrect data, while the trajectory produced by Combat split into two ends, contradicted with biology (Fig. 5D, S11A).

Fig. 5. RBET achieves the best performance in testicular data in terms of cell-cell communication.

A Histogram of cell composition in each batch. B UMAP visualization of original dataset colored by batches (up) and cell types (down). C Evaluation scores of RBET, LISI and kBET for different BEC methods, with their optimal choices colored in red, blue and green, respectively. D UMAP visualization of selected integrated datasets colored by batches (up) and cell types (down), together with inferred trajectories (down). E Strength of communications from Sertoli cells to other germ cells. The arrow outlined in red indicates the strongest communication strength. F Pathway activity of signaling pathways reported to be enriched by intercellular signaling from Sertoli cells to SSCs. G Sankey plot of the intercellular signaling from Sertoli cells to SSCs.

Next, we focused on cell-cell communication from Sertoli cells to other cell types, since the former serves as germ cell ecological niches that support germ cell development³⁴. These three BEC methods all gave the highest communication score to the Sertoli-SSC pair, which is consistent with previous report³² (Fig. 5E). Further, we explored ligand-receptor (L-R) interactions and inferred their related KEGG pathway activity between cell types. For the four established Sertoli-SSC pathways that are crucial for spermatogenesis³⁵, only scanorama recovered all of them (Fig. 5F). Furthermore, data corrected by scanorama identified transcription factor (TF) SMAD1 and HES1 as key L-R downstream signaling effectors for Sertoli-SSC communication and spermatogenesis (Fig. 5G), which is consistent with previous findings that SMAD1/2 and HES1 are intracellular effectors of Notch signaling pathway for SSC proliferation and differentiation^32,36. However, the same analysis based on results from other BEC tools only identified one or none of these important TFs (Fig. 5G, S11D). In summary, RBET correctly selected scanorama as the best BEC tool in the testicular dataset, which was validated by trajectory inference and cell-cell communication analysis.

Applying RBET to single-cell ATAC-seq data

RBET was not only effective in single-cell RNA-seq data, but also extendable to other single-cell modalities. Here, we evaluated a scATAC-seq integration task, using two batches of gene activity data for mouse brain derived from scATAC-seq peak calling³⁷. Notably, the gene activity matrix presented the regulatory potential score of each gene isoform at the single-cell resolution³⁸, and thus scMerge which required standard gene name as input was not applicable here. Meanwhile, housekeeping gene isoform is not commonly used, and therefore data-based RBET was implemented, illustrating the broader application of data-based RG selection and its potential in other single-cell modalities. As shown in Fig. 6A, raw data contained two technical batches with 7 cell types, including varying numbers of cells in each cell type. Witnessed the large batch effect in the raw data (Fig. 6B), we performed BEC with all the methods other than scMerge, and evaluated their results (Fig. 6C). RBET and LISI both selected mnnCorrect, while kBET chose Combat as the optimal method (Fig. 6C). Obviously, Combat underperformed in the UMAP visualization, as it left batch structure within the same cell type, while mnnCorrect fully integrated batches with well-mixed cells (Fig. 6D). The poorer clustering quality of Combat was also evidenced by its smaller SC value (Fig. 6E). Furthermore, cell annotation metrics implied that Combat had comparable accuracy performance (ARI, ACC and NMI) with mnnCorrect (Fig. 6E, S12B). To sum up, mnnCorrect outperformed Combat, implying an inferior choice of kBET.

Fig. 6. RBET performs the best in scATAC-seq data (mouse brain).

A Histogram of cell type composition in each batch. B UMAP visualization of original dataset colored by batches (up) and cell types (down). C Evaluation scores of RBET, LISI and kBET for different BEC methods, with their optimal choices colored in red, blue and green, respectively. D UMAP visualization of selected integrated datasets colored by batches (up) and cell types (down). E The clustering quality (measured by Silhouette Coefficient) and the annotation precision (measured by accuracy between inferred cell annotations and real cell tags) of different BEC methods. Note that accuracy ranges in [0,1], while SC ranges in [–1,1], and a larger value indicates a better result.

Discussion

RBET provides a robust guidance on selecting case-specific BEC methods. It consists of two steps: (1) selecting RGs; and (2) batch effect testing on RGs in the integrated data. Different from kBET and LISI, the principle underlying RBET is that BEC should leave no batch effect only on RGs but not on all the genes. In both simulated (Fig. 2) and multi-scenario real datasets (Fig. 1C), RBET outperformed LISI and kBET in terms of batch effect detecting power, type I error control, overcorrection awareness, computational efficiency and large batch effect robustness.

The idea of RBET is inspired by the concept of housekeeping genes or RGs, which are stably and abundantly expressed with moderate expression variation across cells. There are two different strategies for constructing RGs, i.e., literature-based and data-based RGs. The first strategy is preferred in applications, and we manually curated experimentally validated housekeeping genes from literature (Table S6). When literature-based RGs are not available, we recommend data-based RGs, which may also provide reliable results (Table S7; Fig. 6). To be noted, the result of data-based RBET depends on its parameters, and thus further efforts are in need to make it more reliable.

Previous benchmark studies largely focused on metrics for cell annotation, with only a few downstream analyses including trajectory conservation. We went one step further to explore single- and multi-lineage trajectories (Fig. 4, S7–S10) according to known marker expression patterns along the differentiation pseudotime, and examined their expression consistency, as well as cell distribution consistency across batches. Moreover, we performed cell-cell communication analysis for the testicular dataset and identified known interaction paired as well as key L-R downstream signal pathways and TFs (Fig. 5), where RBET demonstrated its superiority .

Some BEC benchmark studies have comprehensively evaluated the data integration outcomes using up to 14 metrics. Different conclusions were drawn as some recommended all-weather top performers while other suggested a scenario-based guideline of choices. Our analyses favored the latter (Fig. 1C), since a simple linear regression model Combat was suitable for simple biological or technical replicates with limited cell type complexity (Fig. S7) and nonlinear Seurat and scanorama performed better in high batch effect and complex cell type composition scenarios (Figs. 4, 5). Nevertheless, the use of different metrics can lead to opposite conclusion, for example, a former top rated Harmony was discouraged to be used for complex dataset in a later study^9,10,37. This discrepancy may stem from the use of different test datasets, as well as different multi-metric systems, which is another dilemma for common practitioners due to the lack of consensus. Different metrics only assess, mostly indirectly, limited aspects of BEC performance, and it appears tedious to perform a 14-metric benchmarking in routine practice, not to mention the expertise required to interpret and rank metrics for datasets of different research fields. Given the multi-feature and easy to use advantages of RBET, we propose to apply RBET for routine BEC optimization practice, either by comparing different BEC tools or by finetuning parameters for a favorite BEC tool. Actually, this may open an era for case-specific optimal decision making for BEC tools instead of following a general guideline.

Finally, given the concept of RGs and dimensionality reduction that can be similarily applied to other single-cell modalities, i.e., scATAC-seq (Fig. 6), CITE-seq and MERFISH, it is conceivable to expand the utility of RBET to broader BEC tasks. RBET is provided as an R package with curated lists of housekeeping genes for multiple tissue types, RG calculation tool and MAC test function. In addition, the package offers user friendly interface to common BEC tools like Seurat, which is available at https://github.com/zlyx26/RBET.

Methods

Motivation

The underlying principle is motivated by the biological features of housekeeping genes, which refer to genes that are essential for basic cellular functions and the maintenance of cellular viability³⁹, and show uniform expression across various cell types under diverse conditions. Given their consistency, they are often used as internal controls and standards for the expression normalization and quantification^30,40,41. Unlike most other genes in scRNA-seq data, housekeeping genes are stably and abundantly expressed, thus almost unaffected by dropout events. This is important because the inflated zeros in gene expression comprise both biological and technical variations, which are hard to discern. When evaluating batch correction for a human peripheral blood mononuclear cell (PBMC) dataset by Seurat, we find that the expression of housekeeping genes (i.e., RPS17 and RPS18 in Fig. S1A, S1B) is changed not only in expression level, but also in expression variation within the cell types before and after BEC, suggesting a potential usage for separating technical and biological sources of variation. Thus, we propose that a perfect batch correction should align the expression level of housekeeping genes between batches (technical batch correction), but not at the expense of losing expression variation within the batch, or in other word, loss of biological information that causes overcorrection.

RBET framework

RBET is proposed for evaluating the success of BEC when integrating multiple single-cell sequencing datasets. It consists of two steps: (1) selecting reference genes (RGs); and (2) tesing batch effects on RGs in integrated data. There are two different strategies for selecting RGs, i.e., literature-based RGs and data-based RGs.

Literature-based reference genes selection

Although housekeeping genes show uniform expression across a panel of tissues, the expression stability of these genes differ among cell types and tissues⁴². Thus, to obtain reliable results, literature-based RGs are selected from experimentally validated housekeeping genes specific to the tissue or cell type of interest. It goes in four steps as follows. (Step 1) Annotate cell types in each batch by manual annotation or automatic annotation tools like ScType²⁴. (Step 2) Select experimentally validated housekeeping genes specific to the current tissue or cell type as candidates from existing literature. To facilitate the analysis, we provide a database of experimentally validated housekeeping genes for 23 tissues or cell types collected from literature (Table S6), also available in our R package. (Step 3) Rank the candidate genes according to the number of cell types where they are differentially expressed across batches. First, the scRNA-seq data is log-transformed by log(1+count). Next, we use Mood’s median test to test whether a candidate gene is differentially expressed in each cell type (mood.medtest function in R package RVAideMemoire, v0.9-83), and get the p-value $p_{ij}$ for the i-th candidate in the j-th cell type. Then, we count the number $n_i$ of cell types where the i-th gene is differentially expressed across batches, i.e., $n_i={\sum }_{j}I\left(p_{ij}<0.05\right)$. Finally, we rank the candidate genes according to $n_i$. (Step 4) Those candidates with the largest $n_i$ are taken as RGs, and each batch should express at least one RG. Note that the number of RGs should be at least 2.

Data-based reference genes selection

In case of no reported tissue-specific housekeeping genes or difficult cell annotation, data-based RG selection is also provided, which chooses RGs directly from data. Similar with literature-based method, it selects genes that express invariably in each batch and differentially across batches. It contains the following five steps. (Step 1) Cluster cells in each batch (FindClusters function in R package Seurat). (Step 2) Select genes that express invariably in clusters. For each batch, we subset the expression of the i-th gene in the j-th cluster as $e_{ij}$, and compute an index as the standard deviation (sd) of $e_{ij}$ divided by the mean of $e_{ij}$, that is, $I_{ij}=\mathrm{sd}\left(e_{ij}\right)/\mathrm{mean}\left(e_{ij}\right).$ Then, we rank the genes according to the index from small to large in each cluster, and count the number of clusters $c_i$ where the i-th gene is in the top r%. Next, we rank the genes according to $c_i$ from large to small in each batch, and count the number of batches $b_i$ where the i-th gene is in the top r%. Finally, we rank the genes according to $b_i$ from large to small, and the genes in the top r% are taken as candidates in clusters. (Step 3) Select genes that express invariably across clusters. For each batch, the expression of the i-th gene is taken as $e_i$ and the mean expression of the i-th gene in each cluster is taken as $m_i$. We compute an index $I_i=\mathrm{sd}\left(m_i\right)/\mathrm{mean}\left(e_i\right)$. Then we rank the genes according to the index from small to large in each batch, and count the number of batches $b_i$ where the i-th gene is in the top r%. Finally, we rank the genes according to $b_i$ from large to small, and the genes in the top r% are taken as candidates across clusters. (Step 4) We first take an intersection of candidates in clusters and across clusters. Then, we count the times $n_i$ when the i-th candidate gene expresses differentially across batches by comparing pairwise clusters in pairwise batches (mood.medtest function in R package RVAideMemoire, v0.9-83), and rank the candidates according to $n_i$ from large to small. (Step 5) The top l candidates are taken as RGs. Here, we take $r=2$ and $l=50$.

Batch effect testing based on two-sample distribution comparison

Let $\mathbf{X}\mathit{\in }{\mathbb{R}}^{\mathit{n}\mathit{\times }\mathit{p}}$ and $\mathbf{Y}\mathit{\in }{\mathbb{R}}^{\mathit{m}\mathit{\times }\mathit{p}}$ be the gene expression matrices of two different batches, where n and m are cell numbers and p is the gene number. Input matrices with different genes can be inner joined together by shared genes, and thus the gene numbers become the same. We notice that samples from two different batches can be well regarded as samples from two different distributions. In such a view, the problem of batch effect detection naturally fits into the framework of two-sample distribution test, with the null hypothesis that X and Y have the same distribution:

$$H_0:F=G\mathrm{versus}{H}_1:F\ne G,$$

where F and G are distribution functions of X and Y, respectively.

For the case $p=1$, this problem is well studied in statistical literature⁴³. However, only few methods have been proposed for the high-dimensional cases. Recently, we introduced a class of MAC statistics for this problem¹³. Theoretical analysis has demonstrated that MAC has an upper bound of order ($\log \left(nm\right)$) when there is no batch effect, which is much smaller than the lower bound of order ($n+m$) when batch effect is present¹³. Notably, MAC outperforms other methods in cases $p<10$, but it is almost inapplicable when p exceeds 10. Hence, due to the substantial size of genes in our datasets, i.e., p could be as large as thousands, MAC cannot be directly employed.

Motivated by the fact that dimensionality reduction tool UMAP well preserves the neighborhood characteristics of each point in the original space and is computationally efficient^12,44, we propose to map the original dataset into a two-dimensional vector space using UMAP, and then MAC statistics can be applied. Specifically, we put X and Y together to form a large $\left(n+m\right)\times p$ matrix, and reduce it into a two-dimensional observation $\mathbf{Z}\in {\mathbb{R}}^{\left(n+m\right)\times 2}$. Z can be partitioned into $\tilde{\mathbf{X}}\mathit{\in }{\mathbb{R}}^{n\times 2}$ and $\tilde{\mathbf{Y}}\mathit{\in }{\mathbb{R}}^{m\times 2}$, corresponding to X and Y, respectively. In this way, we obtain a two-dimensional vector for each batch in the same vector space which well-preserves the original neighborhood characteristics. Next, we implement MAC with $p=2$ to detect the batch effect among $\tilde{\mathbf{X}}$ and $\tilde{\mathbf{Y}}$, namely that among X and Y.

Let $\mathit{x}=\left\{{\mathit{x}}_{\mathit{i}}:i=\mathrm{1,\; 2},\dots ,n\right\}$ be the collection of n rows of $\tilde{\mathbf{X}}$, and $\mathit{y}=\left\{{\mathit{y}}_{\mathit{i}}:i=\mathrm{1,\; 2},\dots ,m\right\}$ be the collection of m rows of $\tilde{\mathbf{Y}}$. Thus, x and y are mapped samples from two different batches. Let $\left({\mathit{x}}_{\mathit{i}},{\mathit{y}}_{\mathit{j}}\right)$ be any pair of mapped samples, where ${\mathit{x}}_{\mathit{i}}=\left(z_{i,1},z_{i,2}\right)$ and ${\mathit{y}}_{\mathit{j}}=\left(z_{j,1},z_{j,2}\right)$, and $d\left(\mathit{a},\mathit{b}\right)=\sqrt{{\sum }_i{\left(a_i-b_i\right)}^2}$ be the Euclidean distance between two vectors. Define

$$A_{{\mathit{x}}_{\mathit{i}},{\mathit{y}}_{\mathit{j}}}=\left\{x=\left(z_1,z_2\right)\in {\mathbb{R}}^2:d\left(z_1,z_{i,1}\right)\le d\left(z_{i,1},z_{j,1}\right)\right\},$$

$$B_{{\mathit{x}}_{\mathit{i}},{\mathit{y}}_{\mathit{j}}}=\left\{x=\left(z_1,z_2\right)\in {\mathbb{R}}^2:d\left(z_2,z_{i,2}\right)\le d\left(z_{i,2},z_{j,2}\right)\right\},$$

Then, we divide the whole sample space ${\mathbb{R}}^2$ into four disjoint parts,

$$A_{11}=A_{{\mathit{x}}_{\mathit{i}},{\mathit{y}}_{\mathit{j}}}\cap B_{{\mathit{x}}_{\mathit{i}},{\mathit{y}}_{\mathit{j}}},A_{12}=A_{{\mathit{x}}_{\mathit{i}},{\mathit{y}}_{\mathit{j}}}^c\cap B_{{\mathit{x}}_{\mathit{i}},{\mathit{y}}_{\mathit{j}}},$$

$$A_{21}=A_{{\mathit{x}}_{\mathit{i}},{\mathit{y}}_{\mathit{j}}}\cap B_{{\mathit{x}}_{\mathit{i}},{\mathit{y}}_{\mathit{j}}}^c,A_{12}=A_{{\mathit{x}}_{\mathit{i}},{\mathit{y}}_{\mathit{j}}}^c\cap B_{{\mathit{x}}_{\mathit{i}},{\mathit{y}}_{\mathit{j}}}^c.$$

$\mathrm{Let}{P}_{ij}={\sum }_{k=1}^{n}I\left({\mathit{x}}_{\mathit{k}}\in A_{ij}\right),Q_{ij}={\sum }_{k=1}^{m}I\left({\mathit{y}}_{\mathit{k}}\in A_{ij}\right),R_{ij}=P_{ij}+Q_{ij},i,j=\mathrm{1,\; 2}$, then the local statistic $T_{ij}$ that compares the distribution of ${\mathit{x}}_{\mathit{i}}$ and ${\mathit{y}}_{\mathit{j}}$ is defined as

$$T_{ij}=T\left({\mathit{x}}_{\mathit{i}},{\mathit{y}}_{\mathit{j}}\right)={\sum }_{i=1}^2{\sum }_{j=1}^2\frac{{\left(P_{ij}-\left(\frac{n}{n+m}\right)R_{ij}\right)}^2}{\left(\frac{n}{n+m}\right)R_{ij}}+\frac{{\left(Q_{ij}-\left(\frac{m}{n+m}\right)R_{ij}\right)}^2}{\left(\frac{m}{n+m}\right)R_{ij}}.$$

It should be noted that if any $R_{ij}$ equals to zero, $T_{ij}$ is set to zero. Finally, MAC statistic takes the maximum of all the local statistics $T_{ij}$, i.e.,

$$\mathrm{MAC}={\max }_{1\le i\le n,1\le j\le m}{T}_{ij}.$$

Sub-sampling strategy to improve computational efficiency

Since scRNA data often comprise hundreds or even thousands of cells, the calculation of MAC statistics on the whole sample is computationally expensive. Considering that similar cells have similar gene expression profiles and cluster together in the reduced vector space, we propose a sub-sampling strategy to improve the computational efficiency. To be more specific, we rank rows of Z according to its first column, and select samples whose row numbers equal to $\mathrm{\Delta }\times j$ with $j=\mathrm{1,\; 2},\dots ,k=\left(n+m\right)/\mathrm{\Delta }$, where $\mathrm{\Delta }$ is the sampling gap. Then we calculate RBET based on the k selected samples. It should be noted that this sampling strategy simply reduces the number of local statistics used for defining MAC, without changing its theoretical properties. Generally, a larger sub-sample k of RBET resulted in a higher power but also required more computational cost. We set k = 50 because k ≥ 50 already showed a stable performance in both Gaussian examples (Fig. S2D) and simulated gene expression data (Fig. S3D).

Compared methods for batch effect testing

RBET is compared with two other methods. (1) kBET¹⁰. It assumes that if there is no batch effect, the distribution of batch labels within any given neighborhood of equal size should mirror that of the entire dataset. Based on this idea, kBET applies Pearson chi-squared test to randomly selected neighborhoods of a fixed size, and then returns an overall rejection rate which reflects the degree of mixing in the dataset. A low rejection rate indicates that the two datasets are well mixed and there is no or small batch effect. R package kBET (v0.99.6) is used here and all parameters are set by default without using sub-sampling.

2) LISI⁷ based on Inverse Simpson Index (ISI). LISI first assigns weights to adjacent cells by constructing Gaussian kernel-based distributions of neighborhoods, and then calculates ISI as the effective number of batches in each cell’s neighborhood. The mean of LISI from all the neighborhoods is defined as the overall number of batches. R package lisi (v1.0) is used for calculation with default parameters.

Test procedure for RBET and LISI

A permutation test was performed to evaluate the null hypothesis, defined as the absence of batch effect. First, we simulated a dataset based on the given batch effect size, and used RBET (LISI) to detect the batch effect, denoted as T₀. Next, we applied the permutation procedure to calculate the p-value for the null hypothesis. To do so, (a1) we randomly reshuffled the data and calculated a RBET (LISI) value from the new dataset; (a2) we repeated step (a1) for N times to obtain N values {T₁, T₂, …, T_N}, representing the observations under the null distribution; (a3) we computed the p-value, defined as $\frac{{\sum }_{i=1}^{N}I\left(T_i>T_0\right)+1}{N+1}$ ($\frac{{\sum }_{i=1}^{N}I\left(T_i<T_0\right)+1}{N+1}$). A p-value smaller than 0.05 indicated the presence of batch effect. In this paper, N was set to 100.

Power analysis

The power of RBET, LISI and kBET on detecting batch effects was estimated through 100 independent repetitions. Specifically, it is defined as the proportion of rejections when the alternative hypothesis is true, i.e., there is batch effect in the data.

Coefficient of variation (CV)

For any random variable, CV is defined as the ratio of its standard deviation to the mean. It serves as a statistical measure of relative variability, making it useful for comparing the degree of dispersion across methods with different units or scales.

Simulation of Gaussian examples

In these examples, we generated synthetic data following Gaussian distributions. We assumed that X followed a two-dimensional Gaussian distribution, $F=N\left(\left(\mathrm{0,\; 0}\right),I\right)$, and considered three different cases for the distribution of Y, G. (G1) Y followed $N\left(\left(\mu ,\mu \right),I\right)$ with µ varying from 0 to 1; (G2) Y followed $N\left(\left(\mathrm{0,\; 0}\right),\sigma I\right)$ with $\sigma$ varying from 1 to 4; (G3) Y followed $N\left(\left(\mathrm{0,\; 0}\right),\Sigma \right)$, where $\Sigma =\left({\sigma }_{ij}\right)$, ${\sigma }_{11}={\sigma }_{22}=1$, and ${\sigma }_{12}={\sigma }_{21}=\rho$ with $\rho$ varying from 0 to 1. For each case, we sampled n (= 50, 100, 200) independently and identically distributed observations from F and G, respectively.

Simulation of gene expression data

In these examples, we simulated single-cell gene expression counts by learning from the example dataset in R package scDesign3 (v1.4.0)⁴⁵, which contained 1000 genes and 6276 cells in total. Here, we simulated two settings by taking the subsets of example data as models: (1) one cell type, containing 1915 CD4 + T cells; (2) two cell types, containing 1915 CD4 + T cells and 1656 cytotoxic T cells. Cell types were selected based on the descending order of cell count in the example dataset.

Computational efficiency

We compared the total running time of RBET, LISI, kBET and kBET with sub-sampling (see supplementary material; Fig. S3E) over 20 repetitions under two settings described in Simulation of gene expression data, with effect size fixed to 0.1, evaluated from 50 independent repetitions.

Simulation on overcorrection of batch effect

We simulated data for the overcorrection problem by taking the example dataset in scDesign3 with 1000 genes and 6276 cells and the effect size fixed to 1. The data were corrected by the CCA function in Seurat (v5.1.0) using different numbers of anchors (k.anchor). The gene expression were extracted by using GetAssayData function in Seurat.

Data processing

The raw data and the integrated data were mainly analyzed with Seurat (R package, v5.1.0)¹⁴. After data normalization and scaling, the top 2000 variable genes were calculated by FindVariableFeatures function and used to construct principal components (PCs). PCs covering the highest variance were selected according to the elbow and Jackstraw plots. Clusters were calculated by FindClusters function and visualized using the Uniform Manifold Approximation and Projection for Dimension Reduction (UMAP).

Tools for integrating scRNA-seq datasets

Most downstream analysis, such as differential gene expression analysis and gene set enrichment analysis, require expression matrices as input. Here, we aimed to evaluate different BEC methods based on the validity and accuracy of subsequent downstream analysis. Therefore, six commonly used BEC tools that could return gene expression matrices with full dimensionality were applied for data integration: Seurat (R package, v5.1.0), limma (R package, v3.50.3), Combat (R package, v3.42.0), scanorama (R package, vl.7.2), scMerge 2 (R package, vl.22.0), and mnnCorrect (R package, v1.12.3). These BEC tools were implemented with default parameters.

Cell annotation

Cell annotation is a fundamental step for single-cell downstream analysis. For pancreas dataset, we utilized the marker-based automatic annotation tool ScType²⁴ together with the marker genes specific to each tissue or cell type. ScType matched the gene expression patterns of data to the known cell types in order to label individual cells or cell clusters²⁴. For mouse brain scATAC-seq dataset, as standard gene names were not provided, ScType was no longer applicable. Fortunately, we still had true cell type labels. Thus, we calculated DEGs of each cell type with FindMarkers function in Seurat, and then labeled each cell cluster manually.

Metrics used for measuring quality of cell clustering

We compared the annotation results with real cell tags under three metrics: accuracy (ACC), adjusted rank index (ARI; R package aricode, v1.0.2), and normalized mutual information (NMI; R package aricode, v1.0.2). Additionally, we evaluated the clustering quality of each cell type using silhouette coefficient (SC; R package cluster, v2.1.6). The SC for cell i is defined as $S\left(i\right)=\frac{b\left(i\right)-a\left(i\right)}{\max \mathrm{}\left(a\left(i\right),b\left(i\right)\right)}$, where $a\left(i\right)$ is the average distance between cell $i$ and all the other cells in the same cell type, and $b\left(i\right)$ is the minimum average distance between cell $i$ and all the cells in any other cell type, and the distance is calculated based on the UMAP reduced data. In this way, SC quantifies how similar a cell is to its own cluster compared to other clusters, and thus assesses whether cells of the same cell type, but from different batches, are well clustered. SC values range from -1 to 1, where higher values indicate better-defined clusters.

Trajectory inference

Trajectory analysis aims to construct an ordered sequence of cells to describe the dynamic changes in single-cell gene expression levels. Here, Slingshot (R package slingshot, v2.2.1) was applied to perform pseudotime analysis on the integrated datasets, which was recommended by previous studies²⁶. It took the low dimensional embedding of gene expression and a vector of cluster labels as the input. Here, the labels were predicted using k-means clustering method (kmeans function in R package stats, v4.1.1). Then, we utilized getLineages function in slingshot to fit a minimum spanning tree (MST) to identify lineage. In accordance with established biological knowledge, we set the root for CD8 + T cell dataset as CD8_Tn_CCR7 and the root for testis dataset as SSCs. However, monocyte dataset lacked explicit cell type annotation, and thus the trajectory was inferred automatically without a prespecified root. The final output was stored in a SlingshotDataSet object containing inferred lineage information.

Measuring the consistency of gene expression profiles along the pseudotime

The expression profile $e$ of gene G could be fitted as a smoothing function of the inferred pseudotime $t$, denoted as $e\left(t\right)$ (smooth.spline function in R package stats, v4.1.1). Assuming that $e_1\left(t\right)$ and $e_2\left(t\right)$ were the fitted curves of expression profiles of gene G in two different batches or lineages, we defined the consistency between these two curves as the averaged distance between them, i.e., $C=\frac{1}{b-a}{\int }_a^b\mid e_1\left(t\right)-e_2\left(t\right){\mid }^{2}dt$. It was actually the mean of $\mid {e_1\left(t\right)-e}_2\left(t\right){\mid }^2$ by taking $t$ as a uniformly distributed random variable on $\left[a,b\right]$. This provided us a simple measurement by first sampling n instances, $\left\{t_1,\dots ,t_n\right\}$, from $U\left[a,b\right]$, and then estimating the averaged distance as $\widehat{C}=\frac{1}{n}{\sum }_{i=1}^n\mid {e_1\left(t_i\right)-e}_2\left(t_i\right){\mid }^2$. In this paper, we took $n=1000$, which was large enough to get an accurate estimation. To mitigate the effect of gene expression size, we normalized $\widehat{C}$ by the square of the mean expression (${\mu }_G$), i.e., $\tilde{C}=\widehat{C}/{\mu }_G^2$. In case of multiple batches, we calculated their pairwise consistency and took the average.

Cell communication analysis

The biological behavior of cells is regulated by the intercellular and intracellular signaling network, which is known as cellular communication. Here, we performed cell-cell communication analysis on the integrated datasets using CellCall (R package cellcall, v1.0.7)³². First, we inferred the overall communication scores between different cell types using TransCommuProfile function. Next, we identified crucial pathways involved in communication by performing a pathway activity analysis with getHyperPathway function. Finally, we visualized the intercellular signaling from one cell type to another in sankey plot (LR2TF and sankey_graph functions).

Reporting summary

Further information on research design is available in the Nature Portfolio Reporting Summary linked to this article.

Author contributions

H.J. and J.Q. supervised the project; H.J. and J.Q. conceived of and designed the study; X.H. and H.L. collected data and performed data analysis; H.J., J.Q., X.H., H.L., and M.C. wrote the paper.

Peer review

Peer review information

Communications Biology thanks the anonymous reviewers for their contribution to the peer review of this work. Primary Handling Editors: Ani Manichaikul and Mengtan Xing.

Data availability

All the datasets used for analyses in this work are publicly available online, deposited in the NCBI’s Gene Expression Omnibus database and ArrayExpress. In particular, we used a pancreas dataset²² containing three batches of single-cell expression data obtained by different sequencing technologies, CelSeq (GSE81076)⁴⁶, CelSeq2 (GSE85241)²⁵ and SMART-Seq2 (E-MTAB-5061)⁴⁷, a human monocyte dataset⁴⁸ under accession code GSE146974, an embryo development dataset containing germ cells in two batches, one from Li et al. (GSE86146)²⁵ and the other from Guo et al. (GSE63818)²⁶, a CD8 + T cell dataset extracted from Qian et al.¹¹ under accession number E-MTAB-8107, E-MTAB-6149 and E-MTAB-6653, taking each sample as a batch, a testis dataset²¹ of testicular transcriptome sequencing data from healthy males under accession code GSE112013, and a mouse brain scATAC-seq dataset³⁷ containing two batches of gene activity data (small_atac_gene_activity).

Code availability

The code for RBET is written in R and Rcpp, and can be obtained at https://github.com/zlyx26/RBET. The code for analyses and plots is available on Zenodo: https://zenodo.org/records/14898612.

Competing interests

The authors declare no competing interests.

Supplementary information

The online version contains supplementary material available at 10.1038/s42003-025-07947-7.