Skip to main content
NCBI home page
Briefings in Bioinformatics logoLink to Briefings in Bioinformatics
. 2024 Aug 9;25(5):bbae392. doi: 10.1093/bib/bbae392

A comparison of scRNA-seq annotation methods based on experimentally labeled immune cell subtype dataset

Qiqing Fu 1, Chenyu Dong 2, Yunhe Liu 3, Xiaoqiong Xia 4, Gang Liu 5, Fan Zhong 6,✉, Lei Liu 7,✉
PMCID: PMC11312369  PMID: 39120646

Abstract

Cell-type annotation is a critical step in single-cell data analysis. With the development of numerous cell annotation methods, it is necessary to evaluate these methods to help researchers use them effectively. Reference datasets are essential for evaluation, but currently, the cell labels of reference datasets mainly come from computational methods, which may have computational biases and may not reflect the actual cell-type outcomes. This study first constructed an experimentally labeled immune cell-subtype single-cell dataset of the same batch and systematically evaluated 18 cell annotation methods. We assessed those methods under five scenarios, including intra-dataset validation, immune cell-subtype validation, unsupervised clustering, inter-dataset annotation, and unknown cell-type prediction. Accuracy and ARI were evaluation metrics. The results showed that SVM, scBERT, and scDeepSort were the best-performing supervised methods. Seurat was the best-performing unsupervised clustering method, but it couldn't fully fit the actual cell-type distribution. Our results indicated that experimentally labeled immune cell-subtype datasets revealed the deficiencies of unsupervised clustering methods and provided new dataset support for supervised methods.

Keywords: scRNA-seq, cell-type annotation, benchmark

Introduction

Single-cell RNA sequencing is a significant technology helping to resolve biological questions at the single-cell level, greatly facilitating research in areas such as cancer and the immune system [1, 2]. Cell-type annotation is the foundation of scRNA-seq data analysis.

Cell-type annotation methods include unsupervised clustering and supervised annotation. Unsupervised clustering identifies similar cell groups based on gene expression by selecting features, reducing dimensions, and clustering. The cell groups are then annotated based on the differentially expressed genes of different clusters or the expression of marker genes. Supervised annotation involves training classification models on pre-labeled reference datasets and then annotating new datasets. With the development of numerous automatic single-cell annotation methods, methodological evaluation is beneficial for researchers to select appropriate methods based on their needs. There have been studies that evaluated single-cell annotation methods [3–6].

Those studies are based on single-cell datasets with known cell labels. However, the cell labels in commonly utilized single-cell datasets are predominantly derived through computational means, including methods like StemID [7], densityMclust [8], nearest neighbor graph clustering [9], hierarchical clustering [10], and Spearman correlation analysis [11]. These labels of cells are not from experimental measurements, and thus, it is possible that there are computational biases for the corresponding cell labels (Supplementary Table 4). Even in single-cell datasets with experimental cell labels, such as CellBench [12], which is composed of five distinctly different lung cancer cell lines, the complexity of the single-cell dataset is low. The majority of automatic cell annotation methods perform well on this CellBench [6]. However, in reality, single-cell datasets are much more complex, making it difficult to evaluate the real-world application of cell annotation methods. We believe that experimentally labeled single-cell immune cell-subtype datasets can avoid the computational biases of reference datasets caused by unsupervised clustering cell annotation methods. This approach can also more accurately reflect the effectiveness of methods in practical application. Furthermore, many deep learning-based methods have been developed, such as scBERT [13] and TOSICA [14], which still need further evaluation.

In response to the challenges above, compared with the CellBench dataset, this study introduced a more complex, experimentally labeled immune subtype single-cell dataset of the same batch for the first time. The dataset reflected the complexity of real-world conditions, aiming to evaluate current single-cell annotation methods. The study assessed 18 single-cell annotation methods, comprising 10 supervised and 8 unsupervised clustering approaches. Performance evaluations were conducted across five experimental scenarios: intra-dataset five-fold cross-validation, unsupervised clustering, immune cell-subtype validation, inter-dataset annotation, and prediction of unknown cell types. These evaluations were performed on four immune cell datasets. Accuracy was used as the evaluation metric for supervised annotations, while the Adjusted Rand Index (ARI) was employed for unsupervised clustering. Our study provided the specific guidelines for single-cell annotation methods based on experimentally labeled immune cell-subtype dataset.

Methods

Methods evaluation

This study screened and evaluated 10 supervised annotation methods and 8 unsupervised clustering methods. We selected the methods that performed well in the previous evaluation studies [3–6] and the latest deep learning-based methods. Those methods included SVM [6], SingleR [15], CHETAH [16], scmapcluster [17], scmapcell [17], as well as deep learning-based methods, such as ItClust [18], TOSICA [14], Cell BLAST [19], scBERT [13], scDeepSort [20], and unsupervised clustering methods Seurat (version 4.0.5) [21], monocle3 (version 1.0.0) [22], monocle2 (version 2.22.0) [23], SHARP [24], SC3 [25], scMAE [26], scFseCluster [27], scDeepCluster [28]. We summarized the normalization, feature selection, dimension transformation, algorithm, and rejection methods in detail (Supplementary Table 7).

Datasets

We used our dataset—the Liu dataset and the other three datasets—the ZhengSort dataset, the Zheng68K dataset, and the PBMCbench dataset (Supplementary Table 3)—derived from peripheral blood immune cells. The composition of cell types is provided in Supplementary Table 5.

The Liu dataset was generated by our laboratory, where 150 mL of blood was drawn from a healthy individual who signed the informed consent of sample collection and data analysis. The study was approved by the Ethics Committee of the Institutes of Biomedical Sciences, Fudan University (No. 2018IBSJS014). Then, 10 immune cell types were obtained through positive and negative selection using antibody-coated magnetic beads from the isolated peripheral blood mononuclear cells (PBMCs). The dataset included CD4+ naive T, CD4+ memory T, CD8+ naive T, CD8+ memory T, NK, γδ T, naive B, memory B, granulocyte, and monocyte. These immune cells were then incubated with human pan-cell antibodies carrying specific nucleotide sequences (Supplementary Table 1) to label them. The labeled cells were pooled into one sample for sequencing, generating a single-cell sequencing dataset composed of mixed immune cell subtypes of the same batch. The information of immunomagnetic enrichment bead kits, immune cell surface proteins, and the theoretical purity of immune cells can be found in Supplementary Table 2.

The ZhengSort dataset was sampled from the different batches of reference datasets with the experimental labels, which were the single-cell sequencing datasets of 10 PBMC-sorted immune cells (CD14+ monocytes, CD19+ B cells, CD34+ cells, CD4+ helper T cells, CD4+/CD25+ regulatory T cells, CD4+/CD45RA+/ CD25− naive T cells, CD4+/CD45RO+ memory T cells, CD56+ natural killer cells, CD8+ cytotoxic T cells, CD8+/ CD45RA+ naive cytotoxic T cells) via magnetic bead enrichment. However, the reference dataset contained batch effects, and there was redundancy and overlap between the CD8+ Cytotoxic T-cell type and the CD8+/CD45RA+ Naive Cytotoxic cell type [11]. The ZhengSort dataset was available at https://zenodo.org/records/3357167.

The Zheng68K dataset was obtained from the same batch of single cell sequencing of peripheral mononuclear blood cells, whose labels were computational labels calculated through Spearman’s correlation with the same reference datasets as the ZhengSort dataset. Rather than using the cell labels provided by the authors, labels were re-annotated using the provided code (https://github.com/10XGenomics/single-cell-3prime-paper/tree/master/pbmc68k_analysis). The Zheng68K dataset originated from https://github.com/10XGenomics/single-cell-3prime-paper.

The PBMCbench dataset, whose labels were computational and obtained through the Louvain community detection algorithm and AUC calculation, included PBMC single-cell sequencing data obtained from low-throughput plate-based sequencing methods Smart-seq2, CEL-Seq2, and high-throughput sequencing methods 10x Chromium (v2), 10x Chromium (v3), Drop-seq, Seq-Well, and inDrops [29]. The PBMCbench dataset can be found at https://portals.broadinstitute.org/single_cell/study/SCP424/single-cell-comparisonpbmc-data.

Dataset processing

In the analysis of the Liu dataset, given the less-than-perfect purity of magnetic bead purification, we adopted the purifying method utilized in the reference [11]. We initially normalized and standardized the raw data. Next, PCA dimension reduction was conducted on various cell types based on their experimental labels, with the selection of 30 principal components for subsequent K-means clustering into distinct clusters. After the initial cluster identification, refinement was achieved by discarding clusters based on the expression of specific marker genes, as illustrated in Supplementary Fig. 1. These included NK cells (CD3E-CD4-NCAM1+), granulocytes (FCGR3B+), monocytes (CD14+), memory CD4+ T cells (CD3D + CD4 + CCR7−), naive CD4+ T cells (CD3D + CD4 + CCR7+), memory CD8+ T cells (CD3D + CD8B + CCR7−), naive CD8+ T cells (CD3D + CD8B + CCR7+), γδ T cells (CD4-CD8B-TRDC+), memory B cells (CD19 + CD79B + TNFRSF13B+), and naive B cells (CD19 + CD79B + TNFRSF13B−). The analysis code named Liu_datasets_purified.R for the Liu data analysis was available at https://zenodo.org/records/10947879. This process culminated in the generation of a refined reference dataset called the Liu dataset (Supplementary Fig. 3).

In the ZhengSort dataset and the Zheng68K dataset analysis, genes not expressed across all cells were filtered out. We calculated the median and the median absolute deviation (MAD) of the total gene expression of cells, then removed cells whose total gene expression fell below Median − 3 × MAD.

In the PBMCbench dataset analysis, we selected the entry with the highest gene expression value for duplicate gene symbols. Given that this dataset had already been pre-annotated following the exclusion of low-quality cells and the inherently small number of cells in low-throughput sequencing datasets, no additional cells were removed.

Experimental design

This study aimed to evaluate the efficacy of annotation methods within actual datasets, thus excluding the use of simulated datasets. The evaluation of annotation methodologies encompassed intra-dataset analysis, immune subtypes annotation, unsupervised clustering, inter-dataset annotation, and the prediction of known and unknown cell types.

Intra-dataset validation

For the supervised methods, we evaluated the performance by applying the five-fold cross-validation across the processed data——Liu, Zhensort, and Zheng68K dataset. The dataset was divided into five-folds in a stratified manner so that each cell population in each fold was equal. The four-fold and the remaining one-fold data were used as the training and test datasets. During validation, because of the minimal number of CD4+ helper T cells in Zheng68K, scBERT training failed. So, we removed the CD4+ helper T cells of Zheng68K only for scBERT and compared this result with other methods.

Immune subtype validation

We extracted immune subtypes from each dataset, where we deleted the non-T-cell subtypes of the Liu dataset and non-CD4+ T cells of the Zheng68K dataset. The absolute numbers of T cells of the Liu dataset and CD4+ T cells of the Zheng68K dataset were not altered, but the relative proportion of each T-cell subtype was elevated. The remaining dataset was divided into five-folds in a stratified manner so that each cell population in each fold was equal. The four-fold and the remaining one-fold data were used as the training and test datasets, respectively.

Inter-dataset prediction

We adopted the PBMCbench dataset for the inter-dataset prediction, including seven sequencing protocols. In the first scenario, we trained on the pbmc1 samples with the other six protocols and tested on the other protocol. In the second scenario, we trained on the pbmc1 sample and tested on the pbmc2 sample using the same protocol.

Unsupervised clustering

We constructed the computational reference dataset Zheng68K_sampled dataset using sample() for random downsampling from the Zheng68K dataset, with cell-type proportions identical to Zheng68K, excluding a very small number of CD4+ helper T cells. Then, we evaluated the performance of unsupervised clustering in the Liu dataset and Zheng68K_sampled dataset. For methods based on R, we used FindClusters() for Seurat, cluster_cells() for monocle3, clusterCells() for monocle2, sc3() for SC3, and SHARP() for SHARP. For methods based on Python, we obeyed the tutorial of a deep learning model to get the results. Because the cell-type number was known, we generated a 10-cluster result for the Liu and Zheng68K_sampled dataset. However, this kind of result did not perform well in the subtype distinction and metrics evaluation. So, we generated 11- and 12-cluster results and finally chose the 11-cluster result for subtype distinction comparison.

The prediction of known and unknown cell types

We stratified the Liu dataset into five-folds. Then, four-folds were treated as the training dataset and the other one-fold as the testing dataset. We removed the T cells, CD4+ T cells, and naive CD4+ T cells from the training dataset in different experiments and kept the testing dataset covering all cells. The three experiments were conducted on all the supervised methods with rejection options.

Evaluation metrics

The main metric for supervised cell annotation performance is accuracy. That for unsupervised clustering methods is ARI.

Results

scRNA-seq dataset of experimentally labeled immune cell subtypes of the same batch

Firstly, we constructed an evaluation dataset of immune cell subtypes with experimental labels of the same batch through mixed single-cell sequencing, including CD4+ naive T, CD4+ memory T, CD8+ naive T, CD8+ memory T, NK, γδ T, naive B, memory B, granulocyte, and monocyte (Methods). We determined the distribution of cell types through classic cell markers. The distribution of experimentally labeled cell types was consistent with the marker gene expression. The same experimental labels almost clustered together, indicating the reliability of the data (Fig. 1C). Consistent with literature [30–32], our dataset also showed NK cells expressing TRDC, monocytes expressing CD4, and neutrophils expressing CD14, further attesting to its reliability. We found that the clustering results for monocytes, granulocytes, B cells, NK, and T cells were distinctly separated. In contrast, within B-cell subgroups (naive B, memory B) and T-cell subgroups (naive CD4+ T, naive CD8+ T, memory CD4+ T, memory CD8+ T, γδ T), boundaries exhibited intermixing among cells, and Spearman correlation analysis showed high similarity within B cell subgroups and T-cell subgroups (Spearman correlation >0.8), making it difficult to distinctly differentiate between cell subgroups (Fig. 1B).

Figure 1.

Figure 1

Open in a new tab

The composition of the Liu dataset (A) UMAP distribution of experimental labels for cell types in the Liu dataset. (B) Spearman correlation of gene expression for different immune cell subtypes in the Liu dataset. (C) Distribution of marker gene expression by cell type.

Intra-dataset validation

To assess the impact of experimental datasets on the performance evaluation of annotation methods, we compared the Liu dataset with other commonly used peripheral blood immune cell datasets, Zheng68K and ZhengSort dataset (Fig. 2A, B).

Figure 2.

Figure 2

Open in a new tab

Intra-dataset validation. (A) Cell-type distribution in the Zheng68K dataset. (B) Cell-type distribution in the ZhengSort dataset. (C) Accuracy of cell-type annotation methods across three datasets. (D) The enlarged view of the blackbordered part (the Liu dataset) in C. (E–F) Annotation accuracy for each cell type across the three datasets.

The Zheng68K dataset comprised 11 immune cell types, while the ZhengSort dataset contained identical cell types but with a better data balance. Analysis revealed that 10 supervised cell annotation methods achieved higher accuracy on the Liu dataset than the Zheng68K dataset, with accuracy rates surpassing 0.6. Apart from Cell BLAST, these methods also showed superior accuracy on the Liu dataset over the ZhengSort dataset (Fig. 2C, Supplementary Fig. 5). Also, the performance on the Liu dataset performed better than the Zheng68K dataset. Compared with the Zheng68K dataset, the ZhengSort and Liu datasets shared a commonality regarding the approximately equal distribution of sample numbers across categories within the datasets (Supplementary Fig. 4). Research indicated that the efficiency of supervised annotation methods improves with better data balance in the training set [5]. Among the three datasets, SVM, scBERT, and scDeepSort emerged as the most effective single-cell annotation methods. They consistently ranked among the top three in annotation accuracy across all datasets, achieving accuracies as high as 0.95 (Fig. 2D, Supplementary Table 6, Supplementary Figs 6 and 8).

Despite employing supervised methods, the accuracy for annotating T-cell subtypes, including CD4+ T cells, remained notably low (Fig. 2E–F, Supplementary Table 6). Given the challenge of altering the absolute number of cells within datasets, we strategically removed non-T-cell subtypes from the training set. This approach aimed to increase the relative proportion of each T-cell subtype, thereby adjusting the data balance within the reference dataset to assess its impact on T-cell subtype annotation accuracy. Specifically, within the Liu dataset, T-cell subtypes were isolated for annotation, revealing that methods such as SVM, scBERT, and scDeepSort did not exhibit significant changes in accuracy for T-cell subtypes. Conversely, Cell BLAST and ItClust showed improved accuracy. Similarly, applying this approach to the Zheng68K dataset led to slightly increased accuracy for SVM, scBERT, Cell BLAST, and ItClust (Supplementary Fig. 10). These findings suggested that for cell subtypes with poor data balance in training datasets, isolating and specifically annotating those cell types only improved accuracy slightly.

Unsupervised clustering

Accurate single-cell annotation can significantly reduce scRNA-seq analysis workload. However, current methods still involve annotating cell types through marker genes or differential genes after unsupervised clustering. There is still a lack of comparison for unsupervised clustering methods using datasets of immune cell subgroups with experimental labels. Therefore, to evaluate the performance of unsupervised single-cell clustering methods on experimentally labeled datasets, we assessed eight state-of-the-art methods [5] using the Liu dataset. The results showed that the best-performing unsupervised method was Seurat, followed by monocle3 and monocle2, while SC3 and SHARP performed poorly (Fig. 3I, Supplementary Fig. 11). The UMAP visualization results revealed that Seurat can distinguish the immune cell subtypes, including naive B, memory B, naive CD4+ T, and memory CD4+ T. Monocle3 and scDeepCluster can distinguish naive B and memory B, while the other unsupervised clustering methods failed (Fig. 3). However, Seurat did not classify all γδ T subtypes into the same category and distinguish between the memory CD4+ T and memory CD8+ T-cell subgroups (Fig. 1A). Therefore, none of these five unsupervised clustering methods can perfectly distinguish these three cell subgroups (Fig. 3A–H). We also evaluated the methods’ performance in the computational Zheng68K_sampled dataset, and the boundary of subtypes was obviously blurred. We found that all the methods showed worse results (ARI < 0.5) compared with the Liu dataset (Methods and Supplementary Fig. 12).

Figure 3.

Figure 3

Open in a new tab

Performance of unsupervised clustering methods. (A–H) Performance of unsupervised clustering methods on the Liu dataset. (I) ARI values for different numbers of clusters, where ARI_10_clusters represent the ARI value for unsupervised clustering into n clusters (n = 10, 11,12), ARI_n_clusters represent the ARI value for unsupervised clustering into n clusters.

Inter-dataset annotation

A common application scenario is inter-dataset annotation, and currently, there is still a lack of experimentally labeled multi-batch single-cell datasets. To assess the performance of supervised methods in inter-dataset annotation, we used the PBMCbench immune cell dataset (Supplementary Fig. 13). Inter-dataset annotation included two scenarios: one was the same sample with different sequencing schemes, and the other was different samples with the same sequencing scheme. In the first scenario, we found higher accuracy for specific pairs of protocols. For instance, methods performed well when trained on 10X2A and then tested on 10Xv2B and 10Xv3, and vice versa. Remarkably, most methods had higher accuracy when trained on iD and tested on SM2 (accuracy > 0.85). We also found that the choice of training dataset was critical, and the methods trained on SM2 and iD generally resulted in lower accuracy. Given the predominance of 10X sequencing in single-cell methods, practical considerations suggested SVM, scBERT, scDeepSort, and SingleR as the most effective for inter-dataset annotation (Fig. 4A). In the second scenario, we found that the overall high-accuracy methods were SVM, followed by scBERT, then ItClust, and scDeepSort. Most methods performed well when trained and tested on 10Xv2 protocol, followed by DR, SW, iD, SM2, and CL2, which indicated that the choice of sequencing protocols was essential (Fig. 4B, Supplementary Fig. 14). Combining the results of two cross-dataset annotation scenarios, we suggested that the methods with high annotation accuracy were SVM, scBERT, and scDeepSort.

Figure 4.

Figure 4

Open in a new tab

Inter-dataset validation. (A) Accuracy of methods for the same sample under different sequencing protocols, with columns representing training sets, rows representing test sets, and values indicating the accuracy for corresponding training and test sets. Blanks indicate failure of training. SM2 stands for Smart-seq2, X10v2A for 10x Chromium(v2)_A, 10xChromium(v2)_B for X10v2_B, CL2 for CEL-Seq2, DR for Drop-Seq, iD for inDrops, and SW for Seq-Well. (B) Accuracy of methods for the same tissue under the same protocol across different samples, SW_pbmc1_pbmc2 indicates using SW_pbmc1 as the training set and SW_pbmc2 as the test set.

Predicting known and unknown cell types

The training set may not always include all cell types existing in the test set. Thus, accurately predicting known and unknown cell types enabled researchers to discover new cell subgroups. To assess the effectiveness of methods in such a task, we evaluated these methods using the Liu dataset. Methods capable of predicting unknown cell types included SVM_rejection, CHETAH, scmapcell, scmapcluster, Cell BLAST, scBERT, and scDeepSort. Given that the Liu dataset contains T-cell subgroups, we conducted tests in three scenarios: (1) the training set excluded T cells, (2) the training set excluded CD4+ T cells; and (3) the training set excluded naive CD4+ T cells. The test set included all cell types. Ideally, methods would accurately annotate positive samples while correctly excluding negative samples. In practice, with the gradual removal of T cells, CD4+ T cells, and naive CD4+ T cells, the accuracy of annotating known cell types gradually decreased and showed a significant decrease after removing naive CD4+ T cells because learning the fine features for distinguishing removed and unremoved cells became difficult. SVM, scDeepSort, scBERT still had the highest annotation accuracy (Fig. 5A). Also, the accuracy of predicting unknown cells gradually decreased; with the removal of T cells, CHETAH and SVM_rejection had the highest accuracy in annotating unknown cell types, but the accuracy was <0.5 (Fig. 5B). In summary, as the difficulty of cell annotation increased, the accuracy for both known and unknown cell types gradually decreased. Above all, the best-performing methods were SVM_rejection and CHETAH. These results indicated that accurately annotating both known and unknown cell types with single-cell annotation methods remained challenging.

Figure 5.

Figure 5

Open in a new tab

Accuracy of predicting known and unknown cell types. (A) Accuracy of predicting known cell types. (B) Accuracy of predicting unknown cell types, remove_T for removing T cells, remove_CD4+ T for removing CD4+ T cells, remove_Naive CD4+ T for removing Naive CD4+ T in the training dataset and keeping intact in the testing dataset.

Discussions and conclusions

This study selected 18 single-cell annotation methods, including 10 supervised and 8 unsupervised methods. The results suggested using SVM, scBERT, and scDeepSort for supervised single-cell annotation and Seurat for unsupervised clustering. This study provided specific guidelines for single-cell annotation methods.

Intra-dataset validation and inter-dataset annotation revealed that SVM, scBERT, and scDeepSort exhibited the highest accuracy. Abdelaal T’s studies demonstrated SVM’s strong performance, aligning with our results [6]. SVM is noted for robust generalization capabilities, effective learning from limited training samples and accurate prediction of testing samples. This study also underscores the impressive performance of deep learning methods like scBERT and scDeepSort [20]. These models remain challenging to interpret fully. scDeepSort leveraged weighted graph neural networks, and GNNExplainer revealed that scDeepSort could identify significant subgraph structures and node features corresponding to highly expressed genes and marker genes in specific cell types, which facilitated the nuanced characterization of cell types through the cell-gene graph iteration. scBERT employed multi-head attention mechanisms and self-supervised learning, initially processing vast amounts of unlabeled single-cell data to discern gene expression patterns and interactions, enhancing model performance and interpretability. The efficacy of these deep learning approaches likely stems from their ability to elucidate gene relationships, thereby capturing cell-type characteristics accurately.

In supervised annotation, the obtained conclusions were consistent between datasets with and without experimental labels. This conclusion indicated that the discrepancy between computational and experimental reference datasets did not affect the performance evaluation of supervised methods, which aligned with previous speculations that the random errors of computational labels should not have systematic impact on the evaluation [33]. SVM, scBERT, and scDeepSort performed well on annotating T-cell immune subtypes. We found that the cell annotation method TOSICA [14], which was trained by the knowledge-based mask matrix from Gene Set Enrichment Analysis (http://www.gsea-msigdb.org/gsea/downloads.jsp), was difficult to distinguish between subtypes of cells with similar functions, which might indicate that biological functional gene sets for classification did not performed well.

In unsupervised clustering, we observed that γδ T cells, memory CD4+ T cells, and memory CD8+ T cells could not be clustered accurately, the distribution of which was not conformed to a dense and circular pattern around the centroid, as seen in other cell types. Instead, their distribution leaned towards a rectangular shape. This discrepancy suggests that existing single-cell unsupervised clustering methods, which yield circular clusters, may struggle with non-circular distributions. Addressing this limitation represents a potential area for unsupervised clustering methods.

However, in the annotation of unknown cell types, Abdelaal [6] discovered that in the Zheng68K dataset, the removal of similar T-cell subtypes (such as CD4+ and memory T cells) for accuracy of annotating unknown cell types did not show a decline. Conversely, the accuracy declined in the Liu dataset, as expected. Moreover, we found that the Zheng68K dataset might have errors in differentiating CD4+ T-cell subtypes, especially naive CD4+ T cells, because the labels were dependent on the Spearman correlation with a reference dataset, which contained errors of redundancy in cell types, such as CD8+ cytotoxic T and CD8+/CD45RA+ naive cytotoxic T. Also, This method risked incorrect labeling because more than 11 immune cell types defined by Zheng68K exist in the peripheral blood, suggesting potential inaccuracies in classifying both known and unknown cells. Additionally, Abdelaal [12]‘s research design, which involved excluding specific immune cells from the training set and then only testing on those excluded cells, cannot accurately assess the method’s effectiveness in annotating cells that remained in the training set. Our study, in contrast, includes the removed and non-removed cell types. The findings reveal a contradiction in the precision of predicting known and unknown cell types, presenting a challenge in single-cell annotation.

However, this study still has limitations. Initially, the magnetically pre-enriched cells with less than 100% purity for constructing experimentally labeled datasets complicate data processing. Flow cytometry, offering higher purity cell sorting, represents a preferable alternative. However, it is worth noting that high-pressure cell sorting of flow cytometry may cause some cell damage, which may lead to more cell rupture. Additionally, it comes with a high cost, demands skilled personnel, and involves complex sample fluorescent labeling. [34, 35]. In contrast, immunomagnetic bead purification has lower purity but preserves cell viability. Meanwhile, it is less expensive, more user-friendly, and simplifies protocols [36, 37]. Therefore, the selection of purification methods depends on practical conditions. Additionally, cell rupture introduces RNA from lysed cells into the solution, complicating initial data processing [38–40]. Analyzing T-cell subtypes is challenging as CD45RA+ and CD45RO+ are both derived from the PTPRC gene, only based on which it is still difficult to distinguish naive from memory T cells. Instead, CCR7 is used as the marker gene, differentiating naive T (CCR7+) from memory T (CCR7−), albeit reducing memory T-cell amounts due to the inclusion of central memory T cells (CCR7 + CD45RA−) [41–43]. In analyzing B cell subtypes, flow cytometry differentiates naive B (CD27−) from memory B (CD27+) cells by CD27 expression [44]. However, this study reveals that memory B cells exhibit low CD27 expression (Supplementary Fig. 2), blurring the distinction from naive B cells, possibly due to transcriptional under-expression or sequencing capture inefficiency [45]. Consequently, TNFRSF13B highly expressed in memory B cells, was used for distinction [46]. For inter-dataset cell annotation, the lack of experimentally labeled immune cell subtypes of multiple batches led to the work on inter-dataset single-cell annotation being incomplete, only using the PBMCbench dataset.

In summary, this study indicated that in supervised methods, the impact of datasets labeled either experimentally or computationally on the outcomes was minimal. However, for unsupervised clustering, datasets with experimental labels more accurately reflected the clustering results. We recommend using SVM, scBERT, and scDeepSort as choices for supervised annotation methods, and Seurat for unsupervised clustering methods.

Key Points

  • We first constructed an experimentally labeled immune cell-subtype single-cell dataset of the same batch, which was better than the previous reference dataset.

  • Our study indicated that the discrepancy between computational and experimental reference datasets did not affect the performance evaluation of supervised methods.

  • We systematically evaluated 18 cell annotation methods and showed that SVM, scBERT, and scDeepSort were the best-performing supervised methods. Seurat was the best-performing unsupervised clustering method, but it couldn't fully fit the actual cell-type distribution.

Supplementary Material

Supplementary_Figure_1_bbae392
supplementary_figure_1_bbae392.jpeg (1.5MB, jpeg)
Supplementary_Figure_2_bbae392
supplementary_figure_2_bbae392.jpeg (58.2KB, jpeg)
Supplementary_Figure_3_bbae392
supplementary_figure_3_bbae392.jpeg (175.8KB, jpeg)
Supplementary_Figure_4_bbae392
supplementary_figure_4_bbae392.jpeg (682.7KB, jpeg)
Supplementary_Figure_5_bbae392
supplementary_figure_5_bbae392.jpeg (134.1KB, jpeg)
Supplementary_Figure_6_bbae392
supplementary_figure_6_bbae392.jpeg (605.9KB, jpeg)
Supplementary_Figure_7_bbae392
supplementary_figure_7_bbae392.jpeg (720.1KB, jpeg)
Supplementary_Figure_8_bbae392
supplementary_figure_8_bbae392.jpeg (1.1MB, jpeg)
Supplementary_Figure_9_bbae392
supplementary_figure_9_bbae392.jpeg (1.2MB, jpeg)
Supplementary_Figure_10_bbae392
supplementary_figure_10_bbae392.jpeg (361.8KB, jpeg)
Supplementary_Figure_11_bbae392
supplementary_figure_11_bbae392.jpeg (477.7KB, jpeg)
Supplementary_Figure_12_bbae392
supplementary_figure_12_bbae392.jpeg (1.3MB, jpeg)
Supplementary_Figure_13_bbae392
supplementary_figure_13_bbae392.jpeg (1.6MB, jpeg)
Supplementary_Figure_14_bbae392
supplementary_figure_14_bbae392.jpeg (749.5KB, jpeg)
Supplementary_Table_bbae392
supplementary_table_bbae392.docx (50.1KB, docx)
Supplementary_Table_6_bbae392
supplementary_table_6_bbae392.xlsx (12.9KB, xlsx)
Supplementary_Table_7_Summary_of_methods_bbae392
supplementary_table_7_summary_of_methods_bbae392.xlsx (14.7KB, xlsx)

Acknowledgements

This work has been supported by the Medical Science Data Center in Shanghai Medical College of Fudan University.

Contributor Information

Qiqing Fu, Institutes of Biomedical Sciences, Fudan University, 200032 Shanghai, P.R. China.

Chenyu Dong, Institutes of Biomedical Sciences, Fudan University, 200032 Shanghai, P.R. China.

Yunhe Liu, Institutes of Biomedical Sciences, Fudan University, 200032 Shanghai, P.R. China.

Xiaoqiong Xia, Institutes of Biomedical Sciences, Fudan University, 200032 Shanghai, P.R. China.

Gang Liu, Institutes of Biomedical Sciences, Fudan University, 200032 Shanghai, P.R. China.

Fan Zhong, Intelligent Medicine Institute, Fudan University, 200032 Shanghai, P.R. China.

Lei Liu, Intelligent Medicine Institute, Fudan University, 200032 Shanghai, P.R. China.

Author contributions

Qiqing Fu (Conceptualization), Yunhe Liu, Qiqing Fu, Chenyu Dong, Xiaoqiong Xia (Experiment); Qiqing Fu (Methodology), Qiqing Fu (Software), Qiqing Fu (Validation), Qiqing Fu, Chenyu Dong, Fan Zhong (Manuscript preparation), Qiqing Fu (Visualization), Lei Liu (Funding acquisition). All authors have read and agreed to the published version of the manuscript.

Conflict of interest

None declared.

Funding

This work was supported by the Peak Disciplines (Type IV) of Institutions of Higher Learning in Shanghai.

Software and data availability

The Liu dataset contains 10 types of purified immune cells, and the original dataset can be accessed via PRJNA843175, available as of the publication of this article. The processed data can be downloaded from Zenodo (https://zenodo.org/records/10947879) and is available as of the publication of this article. The code for this study is available at https://github.com/Qiqing-Fu/single-cell-annotation-methods.

References

  • 1. Vegliante  R, Pastushenko  I, Blanpain  C. Deciphering functional tumor states at single-cell resolution. EMBO J  2022;41:e109221. 10.15252/embj.2021109221. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 2. Luo  OJ, Lei  W, Zhu  G. et al.  Multidimensional single-cell analysis of human peripheral blood reveals characteristic features of the immune system landscape in aging and frailty. Nat Aging  2022;2:348–64. 10.1038/s43587-022-00198-9. [DOI] [PubMed] [Google Scholar]
  • 3. Pasquini  G, Rojo Arias  JE, Schäfer  P. et al.  Automated methods for cell type annotation on scRNA-seq data. Comput Struct Biotechnol J  2021;19:961–9. 10.1016/j.csbj.2021.01.015. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 4. Huang  Q, Liu  Y, Du  Y. et al.  Evaluation of cell type annotation R packages on single-cell RNA-seq data. Genomics Proteomics Bioinformatics  2021;19:267–81. 10.1016/j.gpb.2020.07.004. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 5. Sun  X, Lin  X, Li  Z. et al.  A comprehensive comparison of supervised and unsupervised methods for cell type identification in single-cell RNA-seq. Brief Bioinform  2022;23:23. 10.1093/bib/bbab567. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 6. Abdelaal  T, Michielsen  L, Cats  D. et al.  A comparison of automatic cell identification methods for single-cell RNA sequencing data. Genome Biol  2019;20:194. 10.1186/s13059-019-1795-z. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 7. Muraro  MJ, Dharmadhikari  G, Grün  D. et al.  A single-cell transcriptome atlas of the human pancreas. Cell Syst  2016;3:385–394.e3. 10.1016/j.cels.2016.09.002. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 8. Xin  Y, Kim  J, Okamoto  H. et al.  RNA sequencing of single human islet cells reveals type 2 diabetes genes. Cell Metab  2016;24:608–15. 10.1016/j.cmet.2016.08.018. [DOI] [PubMed] [Google Scholar]
  • 9. Single-cell transcriptomics of 20 mouse organs creates a Tabula Muris. Nature  2018;562:367–72. 10.1038/s41586-018-0590-4. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 10. Baron  M, Veres  A, Wolock  SL. et al.  A single-cell transcriptomic map of the human and mouse pancreas reveals inter- and intra-cell population structure. Cell Syst  2016;3:346–360.e4. 10.1016/j.cels.2016.08.011. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 11. Zheng  GX, Terry  JM, Belgrader  P. et al.  Massively parallel digital transcriptional profiling of single cells. Nat Commun  2017;8:14049. 10.1038/ncomms14049. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 12. Tian  L, Dong  X, Freytag  S. et al.  Benchmarking single cell RNA-sequencing analysis pipelines using mixture control experiments. Nat Methods  2019;16:479–87. 10.1038/s41592-019-0425-8. [DOI] [PubMed] [Google Scholar]
  • 13. Yang  F, Wang  W, Wang  F. et al.  scBERT as a large-scale pretrained deep language model for cell type annotation of single-cell RNA-seq data. Nat Mach Intell  2022;4:852–66. 10.1038/s42256-022-00534-z. [DOI] [Google Scholar]
  • 14. Chen  J, Xu  H, Tao  W. et al.  Transformer for one stop interpretable cell type annotation. Nat Commun  2023;14:223. 10.1038/s41467-023-35923-4. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 15. Aran  D, Looney  AP, Liu  L. et al.  Reference-based analysis of lung single-cell sequencing reveals a transitional profibrotic macrophage. Nat Immunol  2019;20:163–72. 10.1038/s41590-018-0276-y. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 16. de  Kanter  JK, Lijnzaad  P, Candelli  T. et al.  CHETAH: a selective, hierarchical cell type identification method for single-cell RNA sequencing. Nucleic Acids Res  2019;47:e95. 10.1093/nar/gkz543. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 17. Kiselev  VY, Yiu  A, Hemberg  M. scmap: projection of single-cell RNA-seq data across data sets. Nat Methods  2018;15:359–62. 10.1038/nmeth.4644. [DOI] [PubMed] [Google Scholar]
  • 18. Hu  J, Li  X, Hu  G. et al.  Iterative transfer learning with neural network for clustering and cell type classification in single-cell RNA-seq analysis. Nat Mach Intell  2020;2:607–18. 10.1038/s42256-020-00233-7. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 19. Cao  ZJ, Wei  L, Lu  S. et al.  Searching large-scale scRNA-seq databases via unbiased cell embedding with Cell BLAST. Nat Commun  2020;11:3458. 10.1038/s41467-020-17281-7. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 20. Shao  X, Yang  H, Zhuang  X. et al.  scDeepSort: a pre-trained cell-type annotation method for single-cell transcriptomics using deep learning with a weighted graph neural network. Nucleic Acids Res  2021;49:e122. 10.1093/nar/gkab775. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 21. Hao  Y, Hao  S, Andersen-Nissen  E. et al.  Integrated analysis of multimodal single-cell data. Cell  2021;184:3573–3587.e29. 10.1016/j.cell.2021.04.048. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 22. Cao  J, Spielmann  M, Qiu  X. et al.  The single-cell transcriptional landscape of mammalian organogenesis. Nature  2019;566:496–502. 10.1038/s41586-019-0969-x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 23. Qiu  X, Mao  Q, Tang  Y. et al.  Reversed graph embedding resolves complex single-cell trajectories. Nat Methods  2017;14:979–82. 10.1038/nmeth.4402. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 24. Wan  S, Kim  J, Won  KJ. SHARP: hyperfast and accurate processing of single-cell RNA-seq data via ensemble random projection. Genome Res  2020;30:205–13. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 25. Kiselev  VY, Kirschner  K, Schaub  MT. et al.  SC3: consensus clustering of single-cell RNA-seq data. Nat Methods  2017;14:483–6. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 26. Fang  Z, Zheng  R, Li  M. scMAE: a masked autoencoder for single-cell RNA-seq clustering. Bioinformatics  2024;40:btae020. 10.1093/bioinformatics/btae020. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 27. Wang  Z, Xie  X, Liu  S. et al.  scFseCluster: a feature selection-enhanced clustering for single-cell RNA-seq data. Life Sci Alliance  2023;6:e202302103. 10.26508/lsa.202302103. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 28. Tian  T, Wan  J, Song  Q. et al.  Clustering single-cell RNA-seq data with a model-based deep learning approach. Nat Mach Intell  2019;1:191–8. 10.1038/s42256-019-0037-0. [DOI] [Google Scholar]
  • 29. Ding  J, Adiconis  X, Simmons  SK. et al.  Systematic comparison of single-cell and single-nucleus RNA-sequencing methods. Nat Biotechnol  2020;38:737–46. 10.1038/s41587-020-0465-8. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 30. Kazer  SW, Aicher  TP, Muema  DM. et al.  Integrated single-cell analysis of multicellular immune dynamics during hyperacute HIV-1 infection. Nat Med  2020;26:511–8. 10.1038/s41591-020-0799-2. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 31. Schnizlein-Bick  CT, Sherman  MR, Boggs  DL. et al.  Incidence of HIV infection in monocyte subpopulations characterized by CD4 and HLA-DR surface density. AIDS  1992;6:151–6. 10.1097/00002030-199202000-00002. [DOI] [PubMed] [Google Scholar]
  • 32. Haziot  A, Tsuberi  BZ, Goyert  SM. Neutrophil CD14: biochemical properties and role in the secretion of tumor necrosis factor-alpha in response to lipopolysaccharide. J Immunol  1993;150:5556–65. 10.4049/jimmunol.150.12.5556. [DOI] [PubMed] [Google Scholar]
  • 33. Ma  W, Su  K, Wu  H. Evaluation of some aspects in supervised cell type identification for single-cell RNA-seq: classifier, feature selection, and reference construction. Genome Biol  2021;22:264. 10.1186/s13059-021-02480-2. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 34. Telford  WG. Flow cytometry and cell sorting. Front Med (Lausanne)  2023;10:1287884. 10.3389/fmed.2023.1287884. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 35. Manohar  SM, Shah  P, Nair  A. Flow cytometry: principles, applications and recent advances. Bioanalysis  2021;13:181–98. 10.4155/bio-2020-0267. [DOI] [PubMed] [Google Scholar]
  • 36. Li  Q, Zhang  X, Peng  Y. et al.  Comparison of the sorting efficiency and influence on cell function between the sterile flow cytometry and immunomagnetic bead purification methods. Prep Biochem Biotechnol  2013;43:197–206. 10.1080/10826068.2012.719846. [DOI] [PubMed] [Google Scholar]
  • 37. Frenea-Robin  M, Marchalot  J. Basic principles and recent advances in magnetic cell separation. Magnetochemistry  2022;8:11. 10.3390/magnetochemistry8010011. [DOI] [Google Scholar]
  • 38. Yang  S, Corbett  SE, Koga  Y. et al.  Decontamination of ambient RNA in single-cell RNA-seq with DecontX. Genome Biol  2020;21:57. 10.1186/s13059-020-1950-6. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 39. Lun  ATL, Riesenfeld  S, Andrews  T. et al.  EmptyDrops: distinguishing cells from empty droplets in droplet-based single-cell RNA sequencing data. Genome Biol  2019;20:63. 10.1186/s13059-019-1662-y. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 40. Young  MD, Behjati  S. SoupX removes ambient RNA contamination from droplet-based single-cell RNA sequencing data. Gigascience  2020;9:giaa151. 10.1093/gigascience/giaa151. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 41. Cho  BK, Rao  VP, Ge  Q. et al.  Homeostasis-stimulated proliferation drives naive T cells to differentiate directly into memory T cells. J Exp Med  2000;192:549–56. 10.1084/jem.192.4.549. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 42. Sallusto  F, Lenig  D, Förster  R. et al.  Two subsets of memory T lymphocytes with distinct homing potentials and effector functions. Nature  1999;401:708–12. 10.1038/44385. [DOI] [PubMed] [Google Scholar]
  • 43. Medara  N, Lenzo  JC, Walsh  KA. et al.  Peripheral memory T-cell profile is modified in patients undergoing periodontal management. J Clin Periodontol  2021;48:249–62. 10.1111/jcpe.13399. [DOI] [PubMed] [Google Scholar]
  • 44. Chong  Y, Ikematsu  H, Yamaji  K. et al.  CD27(+) (memory) B cell decrease and apoptosis-resistant CD27(−) (naive) B cell increase in aged humans: implications for age-related peripheral B cell developmental disturbances. Int Immunol  2005;17:383–90. 10.1093/intimm/dxh218. [DOI] [PubMed] [Google Scholar]
  • 45. Qiu  P. Embracing the dropouts in single-cell RNA-seq analysis. Nat Commun  2020;11:1169. 10.1038/s41467-020-14976-9. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 46. Llinàs  L, Lázaro  A, de  Salort  J. et al.  Expression profiles of novel cell surface molecules on B-cell subsets and plasma cells as analyzed by flow cytometry. Immunol Lett  2011;134:113–21. 10.1016/j.imlet.2010.10.009. [DOI] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Supplementary_Figure_1_bbae392
supplementary_figure_1_bbae392.jpeg (1.5MB, jpeg)
Supplementary_Figure_2_bbae392
supplementary_figure_2_bbae392.jpeg (58.2KB, jpeg)
Supplementary_Figure_3_bbae392
supplementary_figure_3_bbae392.jpeg (175.8KB, jpeg)
Supplementary_Figure_4_bbae392
supplementary_figure_4_bbae392.jpeg (682.7KB, jpeg)
Supplementary_Figure_5_bbae392
supplementary_figure_5_bbae392.jpeg (134.1KB, jpeg)
Supplementary_Figure_6_bbae392
supplementary_figure_6_bbae392.jpeg (605.9KB, jpeg)
Supplementary_Figure_7_bbae392
supplementary_figure_7_bbae392.jpeg (720.1KB, jpeg)
Supplementary_Figure_8_bbae392
supplementary_figure_8_bbae392.jpeg (1.1MB, jpeg)
Supplementary_Figure_9_bbae392
supplementary_figure_9_bbae392.jpeg (1.2MB, jpeg)
Supplementary_Figure_10_bbae392
supplementary_figure_10_bbae392.jpeg (361.8KB, jpeg)
Supplementary_Figure_11_bbae392
supplementary_figure_11_bbae392.jpeg (477.7KB, jpeg)
Supplementary_Figure_12_bbae392
supplementary_figure_12_bbae392.jpeg (1.3MB, jpeg)
Supplementary_Figure_13_bbae392
supplementary_figure_13_bbae392.jpeg (1.6MB, jpeg)
Supplementary_Figure_14_bbae392
supplementary_figure_14_bbae392.jpeg (749.5KB, jpeg)
Supplementary_Table_bbae392
supplementary_table_bbae392.docx (50.1KB, docx)
Supplementary_Table_6_bbae392
supplementary_table_6_bbae392.xlsx (12.9KB, xlsx)
Supplementary_Table_7_Summary_of_methods_bbae392
supplementary_table_7_summary_of_methods_bbae392.xlsx (14.7KB, xlsx)

Data Availability Statement

The Liu dataset contains 10 types of purified immune cells, and the original dataset can be accessed via PRJNA843175, available as of the publication of this article. The processed data can be downloaded from Zenodo (https://zenodo.org/records/10947879) and is available as of the publication of this article. The code for this study is available at https://github.com/Qiqing-Fu/single-cell-annotation-methods.

Articles from Briefings in Bioinformatics are provided here courtesy of Oxford University Press

RESOURCES