Mapping Cell Identity from scRNA-seq: A primer on computational methods

Abstract

Single cell (sc) technologies mark a conceptual and methodological breakthrough in our way to study cells, the base units of life. Thanks to these technological developments, large-scale initiatives are currently ongoing aimed at mapping of all the cell types in the human body, with the ambitious aim to gain a cell-level resolution of physiological development and disease. Since its broad applicability and ease of interpretation scRNA-seq is probably the most common sc-based application. This assay uses high throughput RNA sequencing to capture gene expression profiles at the sc-level. Subsequently, under the assumption that differences in transcriptional programs correspond to distinct cellular identities, ad-hoc computational methods are used to infer cell types from gene expression patterns. A wide array of computational methods were developed for this task. However, depending on the underlying algorithmic approach and associated computational requirements, each method might have a specific range of application, with implications that are not always clear to the end user. Here we will provide a concise overview on state-of-the-art computational methods for cell identity annotation in scRNA-seq, tailored for new users and non-computational scientists. To this end, we classify existing tools in five main categories, and discuss their key strengths, limitations and range of application.

1. Introduction

The term “cell” was first introduced by physicist Robert Hooke in 1665, in his work Micrographia [1], to describe the box-shaped structures he observed in cork using a microscope he self designed and perfected. This observation marks one of the earliest and most important concepts in the history of molecular biology.

In a way, the recent development of single cell (sc) technologies delineate another key technological breakthrough towards a better understanding of cells and their function. In the last decade sc-technologies, empowered the study of biological systems at the cellular level with a high throughput resolution and provided a refined characterization of cell types [2]. As the application of sc-based methods for the study of complex biological systems has become commonplace, large-scale initiatives such as the Human Cell Atlas and Tabula Sapiens, have been undertaken, with the ambitious goal to chart the human body at a single cell resolution [3], [4].

Virtually any high throughput “omics” assay (genome, epigenome, transcriptome, proteome and others) can be combined with sc-technologies, both in isolation or simultaneously (see [5] for a comprehensive review) to obtain an integrated and comprehensive overview of the molecular landscape of a cell. This review will focus on scRNA-seq, which is currently considered the most mature and most commonly used sc-assay [6].

Since its introduction in 2009 [7] scRNA-seq has undergone constant improvements, through the development of novel technologies and library preparation strategies (see [8] for a comprehensive review). Irrespective of the methodological set-up, every scRNA-seq experiment is rooted in the same conceptual framework: the underlying assumption being that different cell types express different genes. Hence the primary objective of scRNA-seq is to identify/label the distinct cell types that populate a biological sample through the indirect observation of gene expression patterns.

In scRNA-seq library preparation molecular barcodes are used to flag NGS reads from a specific cell and transcript [9]. Dedicated tools such as cellRanger [10] and UMItools [11] are used in the analytical workflow to deconvolute barcodes and identify the genes expressed by each cell. Patterns of gene expression are collected in large counts tables, with cells on the columns and genes on the rows, that summarize the number of transcripts (reads) detected in every cell for every gene.

These tables are usually processed through highly flexible and powerful computational workflows such as Seurat [12] and Scanpy [13], which enable an efficient summarization of salient features and the identification of prospective cell types through the conceptual steps illustrated in the following section.

A common issue of scRNA-seq is that -due to inherent technological limitations- the number of genes/transcripts detected is limited, and usually only 1–5 % of the transcripts of a cell are observed [14]. It follows that counts tables in scRNA-seq are typically sparse, and the majority of the expression values (read counts) correspond to zero.

To mitigate the issue, cells with a limited number of detected expressed genes -usually below 200 [15], [16]- are excluded from downstream analyses. Cell-cycle associated genes are often excluded as well, since in the context of incomplete or partial gene expression profiles their inclusion might skew cell type annotation towards different stages of the cell cycle [15], [16].

Following data preprocessing and filtering, genes showing the largest changes in expression across the dataset are identified and used in subsequent analyses (“highly variable genes”). The underlying assumption being that the changes in expression observed for these genes should reflect the transcriptional programs of different cell types. Dimensionality reduction techniques, such as principal component analysis (PCA), are applied to the highly variable genes to summarise their pattern of expression in a discrete number of Principal Components (see [17] for a comprehensive overview of dimensionality reduction techniques used in scRNA-seq) and obtain a simplified representation of gene expression patterns. Unsupervised clustering is subsequently applied to gather cells with similar patterns of gene expression into clusters. We refer the reader to [18] for a comprehensive overview of clustering methods used in this domain of application.

Finally, each cluster is annotated with a cellular identity, by expert manual curation and/or by comparing genes associated with (or differentially expressed) in a cluster with lists of cell-type specific genes as available from the literature and/or through dedicated resources [2], [19], [20], [21].

The annotation of cell type identities represents a crucial step in scRNA-seq, and provides the ground for the biological interpretation of the results. While the workflow illustrated above represents de facto the standard in the field and was successfully applied for the discovery of novel cell types for several years [21] it also presents some important limitations.

For example clustering algorithms can form clusters only in the presence of a sufficient number of observations (the threshold is typically set by the user) and might overlook scarcely represented cell types. Additionally, since no clear/universally accepted rule or threshold is defined for the delineation of “highly variable genes”, it is not always clear how many genes should be included in the analysis, and to what extent the patterns of expression of these genes might reflect different cell types, cell states and/or even random fluctuations in gene expression [20]. Another important limitation is that manual annotation of cell types is not completely reproducible and by definition can be biassed by the research question at hand and the domain knowledge of the investigator [19], [20].

In an attempt to mitigate these limitations, many computational methods were developed to automate the annotation of cell types in scRNA-seq, without the need for manual curation, by building on collections of publicly available data with high quality annotations. These methods can annotate clusters of similar cells defined according to the protocol outlined above, but also each distinct cell in the dataset, one by one, without any need of clustering and selection of “highly variable genes”. Although all these computational methods were developed for the same task they employ different computational strategies, use different algorithms and have different computational requirements, which might not always be obvious to the user.

This short review will provide a comprehensive yet concise overview of state of the art computational methods for the annotation of cellular identities in scRNA-seq, with the aim to illustrate clearly and in simple terms the key conceptual differences and their potential strengths and limitations to prospective new users and or non experts in the field.

1.1. A broad categorization of tools for cell type annotation in scRNA-seq

The key assumption of scRNA-seq is that different cell types express different genes. Building on this idea, methods for the automatic annotation of cell type identity capitalize on previous/publicly available knowledge to derive an accurate representation of characteristic patterns of gene expression and standardise cell type annotation. Hence, irrespective of the computational approach/algorithm used, knowledge bases with high quality annotations of cell types represent a crucial prerequisite.

Notwithstanding these common assumptions, contemporary methods for cell identity annotation in scRNA-seq are laid on different computational and conceptual frameworks.

At one end of the spectrum we find methods that rely on a fully deterministic approach and build on the concept of “marker gene”. Marker genes are genes that are expressed primarily in a specific cell type [22]. In this framework cell types are “simply” defined by different (linear) combinations of marker genes.

At the other end of the spectrum, more elaborate models based on Machine Learning (ML) algorithms are trained on expert annotated datasets to consistently reproduce manual annotation on new data. Unlike marker genes based tools, ML-models are not constrained to a pre-defined set of genes. Instead, they extract patterns from gene expression data using a broader set of descriptive features. This flexibility allows ML-models to capture nonlinear patterns in gene expression which could be easily overlooked by more simple deterministic approaches. However, since the number and type of features used by different ML methods can vary, the essence of the decision process can not be easily summarized in a simple human-comprehensible equation or rule. As a result ML models are associated with a general loss in the transparency and interpretability [23].

ML is a wide and quickly developing field. Different ML algorithms build on different assumptions and core concepts and might even have different computational and hardware requirements. For example, recent breakthroughs in Deep Learning (see here [24]) represent a turning point in the field of ML, but the underlying technology and methodological implications are not always clear to non-expert in the field (detailed below).

Moreover ML-models can be trained using different strategies such as: supervised learning -where labelled data are presented to the model from the start and only a predetermined number of labels can be learned and semi-supervised learning -where only a subset of the example data is associated with a known label and information from labelled data is borrowed to classify the other data. Furthermore, in an attempt to obtain more powerful and accurate ML-models, different approaches and algorithms can be (and often are) integrated in a single computational method/classifier, adding further layers of complexity.

Considering the wealth of available algorithms and training strategies, the ideal range of application of ML methods might not always be obvious.

Guided by these considerations, here we compiled an up to date (as of August 2024) catalogue of computational methods for the automatic annotation of cell identities for scRNA-seq data by integrating the scRNA-seq-tools database [25], with comprehensive manual review of scientific literature. Methods were subsequently classified in 5 broad categories (Fig. 1) depending on the main computational approach: 1) marker-based (MB); 2) classical ML (C-ML); 3) semi-supervised learning (SSL); 4) deep learning (DL) and 5) Hybrid methods (HM).

Fig. 1.

Schematic representation of the five distinct main conceptual frameworks used for the implementation of computational methods for the classification of cell type identities from scRNA-seq data: A) marker-based (MB). Counts matrix of unlabelled cells (green table) are annotated through ad hoc scoring systems employing cell type specific lists of genes (indicated by different colours); B) classical machine learning (C-ML). Machine learning algorithms are trained with labelled (orange) counts matrix data (blue); C) semi-supervised learning (SSL). Labelled data (blue= count matrix and orange= annotation labels) and unlabeled data (green) processed in the same analytical workflow to transfer labels; D) deep learning (DL); a recent breakthrough in ML applies neural networks to learn labelled data (counts=blue, labels=orange) E) Hybrid methods any of A to D is combined in a single workflow.

In the following sections, to facilitate the comparison and aid prospective users in the selection of the most appropriate method for their specific task, we will offer an overview of the main strengths, limitations and ideal scope of application of each of these 5 categories. Further, the most important properties of every tool, along with its categorization will be summarized in 5 tables (Table 1 to Table 5).

Table 1.

marker-based (MB) tools.

Tool	Impl	AL	#cited	Repository	Custom markers
ACTIONet [26]	Python/R	cell	53	http://github.com/decomverse/ACTIONet	Yes
adobo [27]	Python	cluster	37	http://oscar-franzen.github.io/adobo/	Yes
Besca [28]	Python	cluster	19	http://bedapub.github.io/besca/	Yes
CIA [29]	Python	cell	0	http://github.com/ingmbioinfo/cia	Yes
CellAssign [30]	R	cell	320	http://github.com/irrationone/cellassign	Yes
CellMeSH [31]	Python	cluster	11	http://github.com/shunfumao/cellmesh	No
deCS [32]	R	cluster	19	http://github.com/bsml320/deCS	No
DigitalCellSorter [33]	Python	cluster	30	http://github.com/sdomanskyi/DigitalCellSorter	Yes
MarkerCount [34]	Python	both	14	http://github.com/combio-dku/MarkerCount/tree/master	Yes
MACA [35]	Python	both	19	http://github.com/ImXman/MACA	No
Sargent [36]	R	cell	11	NA	No
scCATCH [37]	R	cluster	215	http://github.com/ZJUFanLab/scCATCH	No
SCINA [38]	R	cell	184	http://github.com/jcao89757/SCINA	Yes
scMayoMap [39]	R	cluster	10	http://github.com/chloelulu/scMayoMap	Yes
scMiko [40]	R	cluster	12	http://github.com/NMikolajewicz/scMiko	Yes
scMAGIC [41]	R	cell	15	http://github.com/TianLab-Bioinfo/scMAGIC	No
scMRMA [42]	R	cluster	28	http://github.com/JiaLiVUMC/scMRMA	Yes
scQCEA [43]	R	cell	3	http://isarnassiri.github.io/scQCEA/	No
SCSA [44]	Python	cluster	122	http://github.com/bioinfo-ibms-pumc/SCSA	Yes
scSorter [45]	R	cell	91	http://github.com/cran/scSorter	Yes
scType [46]	R	cluster	413	http://github.com/IanevskiAleksandr/sc-type	Yes
scTyper [47]	R	cluster	24	http://github.com/omicsCore/scTyper	No
Garnett [48]	R	cell	447	http://cole-trapnell-lab.github.io/garnett/	No

Tool: tool name; Impl:programming language/languages used for the implementation; AL: annotation level the tool annotates at single cell resolution; cluster: the tool can annotate only predefined clusters of cells; both: both single cell and cluster; #cited: number of times cited; repository: hyperlink to the software repository; Custom markers: yes if the tool allows the specification of custom markers, no otherwise

Table 5.

hybrid tools.

Tool	Impl	AL	#cited	Repository
GraphCS [124]	Python/R/C+ +	cell	26	http://github.com/biomed-AI/GraphCS
LAmbDA [125]	Python	cell	48	http://github.com/tsteelejohnson91/LAmbDA
scIAE [126]	Python/R	cell	21	http://github.com/JGuan-lab/scIAE
scNym [127]	Python/R	cell	96	http://github.com/calico/scnym
TripletCell [128]	Python	cell	9	http://github.com/liuyan3056/TripletCell
scSimClassify [129]	Python	cell	5	http://github.com/digi2002/scSimClassify
UNIFAN [130]	Python	cluster	1	http://github.com/doraadong/UNIFAN
scAnCluster [131]	Python	cluster	45	http://github.com/xuebaliang/scAnCluster
CellMarkerAccordion [132]	R	both	3	http://github.com/TebaldiLab/cellmarkeraccordion
scAnnoX [133]	R	cell	4	http://github.com/XQ-hub/scAnnoX
CellGrid [134]	Python	cell	10	http://github.com/Brodinlab/cellgrid
gCAnno [135]	Python	cell	2	http://github.com/xjtu-omics/gCAnno
PopV [136]	Python	cell	7	http://github.com/YosefLab/popV
scClassify [137]	R	cell	119	http://github.com/SydneyBioX/scClassify
scHPL [138]	Python	cell	36	http://github.com/lcmmichielsen/scHPL
Azimuth[139]	R	cluster	10528	http://github.com/satijalab/seurat

Tool: tool name; Impl:programming language/languages used for the implementation; AL: annotation level the tool annotates at single cell resolution; cluster: the tool can annotate only predefined clusters of cells; both: both single cell and cluster; #cited: number of times cited; repository: hyperlink to the software repository

1.2. Marker-based (MB)

Marker-based (MB) methods (Table 1 [26], [27], [28], [29], [30], [31], [32], [33], [34], [35], [36], [37], [38], [39], [40], [41], [42], [43], [44], [45], [46], [47], [48]) implement mathematical models or custom scoring systems to annotate cell types based on a predefined selection of cell-type specific marker genes (Fig. 1 A).

The definition of an optimal list of marker genes is probably the most crucial requirement for the development of marker-based methods [22]. Several approaches could be used to this end, including lists of marker genes collected from the literature; dedicated publicly available repositories such as CellMarker [49]; and/or construction of custom lists of markers through the analysis of expert curated annotations of cell types (as those available from the Human Cell Atlas [3] or PangLaoDB [50]). Importantly, most MB-methods also allow users to define custom cell types and sets of marker genes, which improves the potential for the annotation of novel/additional cell types.

Indeed, the fact that any novel list of marker genes could in theory be evaluated and scored, without the need of training and/or calibration [22], probably represents the main advantage of MB-tools over ML and SSL methods. Since in general MB-based tools are agnostic w.r.t the biological function/meaning of markers it is also theoretically possible to discretise/refine annotations by providing lists of genes that delineate different cell subtypes, states or properties. In these respects, MB-based methods are more flexible compared to ML-based methods and their range of applications can be more easily adapted to the user's specific needs.

Another important consideration is that the algorithms implemented by MB tools are completely deterministic and hence the results can be more easily assessed and inspected, compared to ML-based methods. Despite these attractive benefits, marker-based annotation also presents drawbacks which sometimes hamper their application. The most important limitation is that well characterised/validated marker genes are currently available only for a reduced number of cell types, and cell types lacking a characterization can not be annotated [21]. Additionally, since scRNA count tables are usually sparse, due to the low per cell sequencing depth, markers might not be systematically detected across every cell in the dataset [22]. Due to these considerations, annotation of cell types based on lists of marker genes might represent a double-edged sword since the accuracy strictly depends on the availability of reliable lists of marker genes and previous knowledge [21], [22]. Additionally, lists of marker genes that are either too small or do not include a sufficient number of unique markers, might limit the resolution causing an increase of the proportion of unclassified or inaccurately classified cells.

1.3. 2- Classical ML (C-ML)

By “Classical machine learning (C-ML)” here we address the wide collection of computational ML methods which predate the recent technological breakthroughs of Deep Learning (DL) [23], [24], [25]. An incomplete list includes: support vector machines (SVMs), random forest (RF), logistic regression (LR) XGBoost among the others (we refer the reader to [23] for a more comprehensive overview). C-ML algorithms provide the computation core of methods for the annotation of cellular identity from scRNA-seq data (Table 2 [51], [52], [53], [54], [55], [56], [57], [58], [59], [60], [61], [62], [63]).

Table 2.

classical machine learning (C-ML) based tools.

Tool	Impl	AL	#cited	Repository
CaSTLe [51]	R	cell	127	http://github.com/yuvallb/CaSTLe
CHETAH [52]	R	cell	263	http://github.com/jdekanter/CHETAH
CellO [53]	Python	cluster	33	http://github.com/deweylab/CellO
ELeFHAnt [54]	R	both	0	http://github.com/praneet1988/ELeFHAnt
HieRFIT [55]	R	cell	7	http://github.com/yasinkaymaz/HieRFIT
IBMR [56]	R	cell	4	http://github.com/keshav-motwani/IBMR
scAnnotatR [57]	R	cell	19	http://github.com/grisslab/scAnnotatR
scDetect [58]	R	cell	4	http://github.com/IVDgenomicslab/scDetect
SciBet [59]	R/C+ +	cell	142	http://github.com/PaulingLiu/scibet
scPred [60]	R	cell	376	http://github.com/powellgenomicslab/scPred
SingleCellNet [61]	R	cell	335	http://github.com/CahanLab/singleCellNet
Moana [62]	Python	both	79	http://github.com/yanailab/moana
scASK [63]	Matlab	cell	2	http://github.com/liubo2358/scASKcmd

Tool: tool name; Impl:programming language/languages used for the implementation; AL: annotation level the tool annotates at single cell resolution; cluster: the tool can annotate only predefined clusters of cells; both: both single cell and cluster; #cited: number of times cited; repository: hyperlink to the software repository

In C-ML, a machine learning classifier is trained using pre-labelled data (training set), to learn expert curated or high quality cell type annotations. Data are described through features, that is individual measurable properties computed from the data itself for every cell (for example: n° of expressed genes; level of expression of a specific gene; n° of expressed genes in a specific pathway or ontology).

The objective is to develop a computational model capable of consistently and automatically reproducing the annotation of the training set on novel unlabeled data based on the observation of the features (Fig. 1B). Following training, the performances of the C-ML methods are typically assessed on an independent test set, that is a dataset with known annotations/labels, but formed exclusively by new data which were not presented to the model during training. This strategy is used to measure the model's generalization and predictive accuracy (see section Benchmarking cell type annotation).

Training is a fundamental step in any machine learning application, and is common to both DL and C-ML. However, whereas in C-ML data are described through a discrete number of predetermined features selected and engineered by experts in the field [23], in DL millions of features are extracted and computed directly from the data.

Hence, since the need for the optimization of millions of features and parameters, training DL algorithms require massive datasets, is very computationally demanding and needs dedicated hardware such as GPUs or TPUs; conversely C-ML algorithms can be trained on limited numbers of training examples on virtually any hardware architecture [24], [25]. Due to these reasons C-ML methods are often used for ablation studies [23], that is experiments where one or multiple features are removed from the input during training, and their impact is estimated a posteriori by recording the accuracy of the resulting model [23]. Another important property of C-ML methods is that the underlying decision model is more easily interpretable compared to the complex architectures implemented by DL (see below). For instance, simple linear relationships can be represented using linear models. A key limitation of C-ML models is that they perform optimally only when trained to recognize a limited number of labels/classes. Applied to scRNA-seq, this limits the type of discernible cell identities and their correct annotation. More importantly, only cell types that were originally included in the training set can be recognized and annotated, and the inclusion of new cell types requires a novel round(s) of training [23]. Another potential issue is that most C-ML methods require a balanced training set with similar numbers of data points for every class [23], a requirement that is not usually met by scRNA-seq datasets where extreme fluctuations in the abundance of each cell type can be observed.

The integration of multiple datasets could ideally represent a possible solution to this limitation. Unfortunately, datasets with consistent and homogenized cell type annotations for a large number of cells and across different tissues are rather an exception than a rule in the field of scRNA-seq. Moreover, similar to any large scale omics assay, also scRNA-seq is prone to batch effects caused by difference experimental settings and/or technical limitations, (reviewed in [64]) and the integration of data obtained under different conditions might result extremely challenging.

These considerations impact the generalization of ML models and represent a crucial limitation to the development of any method for the reproducible annotation of cell identity in scRNA-seq.

1.4. 3- Unsupervised and semi-supervised learning (SSL)

Unsupervised and semi-supervised learning (SSL) combines labelled and unlabelled data in a common analytical workflow with the aim to transfer annotations from a reference dataset (labelled data) to the new instances of the data (unlabelled data) through one or more a similarity metrics (Fig. 1C) [24].

Conceptually speaking SSL stands in a middle ground between MB and C-ML: labelled data are used to infer distinctive transcriptional patterns associated with known cell identities; then similarity metrics are used to impute annotations on the unlabeled data based shared patterns of gene expression.

Several SSL strategies and algorithms have been adapted to cell type classification in the recent past [65], [66], [67], [68], [69], including for example: supervised clustering [70], similarity graphs [68], and pairwise comparison to reference annotations. In the same way a variety of different similarity metrics were applied to reconcile gene expression patterns, including (but not limited to) correlation, change in expression (logFC) of differentially expressed genes and many others [70], [71], [72]. Table 3 reports the complete list of SSL based methods for cell type annoation [73], [74], [75], [76], [77], [78], [79], [80], [81], [82], [83], [84], [85], [86], [87].

Table 3.

semi-supervised learning (SSL) based tools.

Tool	Impl	AL	#cited	Repository
CIPR [73]	R	cluster	70	http://github.com/atakanekiz/CIPR-Package
Capybara [74]	R	cell	48	http://github.com/morris-lab/Capybara
clustifyr [75]	R	both	107	http://ttps://github.com/rnabioco/clustifyR
northstar [76]	C+ +/Python	cluster	15	http://github.com/fabilab/northstar
ProjectSVR [77]	R	cell	0	http://github.com/jarninggau/ProjectSVR
RCA2 [78]	R	cell	16	http://github.com/prabhakarlab/RCAv2
scAdapt [79]	Python	cluster	24	http://github.com/biomed-ai/scAdapt
scID [80]	R	cluster	56	http://github.com/BatadaLab/scID
scLearn [81]	R	cell	27	http://github.com/bm2-lab/scLearn
scmap [82]	R	Both	628	http://github.com/hemberg-lab/scmap-shiny
scMatch [83]	Python	cell	104	http://github.com/forrest-lab/scMatch
SingleR [84]	R	cell	3548	http://github.com/dviraran/SingleR
Sincast [85]	R	cell	5	http://github.com/meiosis97/Sincast
SPEEDI [86]	R	cell	3	http://github.com/FunctionLab/SPEEDI/
Symphony [87]	R/C+ +	both	176	http://github.com/immunogenomics/symphony

Tool: tool name; Impl:programming language/languages used for the implementation; AL: annotation level the tool annotates at single cell resolution; cluster: the tool can annotate only predefined clusters of cells; both: both single cell and cluster; #cited: number of times cited; repository: hyperlink to the software repository

SSL tools provide several attractive features compared to MB and C-MB since: 1) the underlying results are easily interpretable (similar to MB); 2) no-predefined set of marker genes is required (similar to C-ML) and the complete transcriptome is evaluated at glance. However, SSL methods are affected by virtually all the limitations discussed previously for C-ML methods. Also in this case the selection of a reference dataset/reference datasets for the construction of the model is crucial and ultimately determines its range of application. Indeed, by definition only cell types for which an adequate number of labelled examples are available can be discerned and incomplete, error-prone and non accurately annotated reference data might result in error propagation. Additionally, similarly to C-ML also in this case the inclusion of novel cell types, requires a continuous refinement/repurposing of the model and might become impractical to/for non experienced users [23]. Another point of similarity with C-ML, SSL may struggle to generalize and classify cell type for which a limited number of examples are provided in the reference dataset and/or in the presence of skewed or imbalanced data. Also in this case the availability of large, homogenized and consistently annotated collection of cell types from scRNA-seq would represent the ideal solution for the development of more general and powerful methods. However, as discussed above, these requirements might be out of reach for the next few years due to inherent technical limitations (see batch effect in the previous section).

1.5. 4-Deep-learning (DL)

Deep learning (DL) methods have recently emerged as powerful tools for the analysis of scRNA-seq since the ability to model complex, high-dimensional data and capture intricate, non-linear relationships in gene expression patterns (Fig. 1D). Table 4 provides a complete list of DL methods for cell identity annotation [88], [89], [90], [91], [92], [93], [94], [95], [96], [97], [98], [99], [100], [101], [102], [103], [104], [105], [106], [107], [108], [109], [110], [111], [112], [113], [114], [115], [116], [117], [118], [119].

Table 4.

deep learning (DL) based tools.

Name	Impl	AL	#cited	Repository
ACTINN [88]	Python	cell	235	http://github.com/mafeiyang/ACTINN
CAMLU [89]	R	cluster	11	http://github.com/ziyili20/CAMLU
CellTICS [90]	Python	cell	7	http://github.com/qyyin0516/CellTICS
Cell BLAST [91]	Python	cell	121	http://github.com/gao-lab/Cell_BLAST
CIForm [92]	Python	cell	36	http://github.com/zhanglab-wbgcas/CIForm
JIND [93]	Python	cell	7	http://github.com/mohit1997/JIND
mtANN [94]	Python/R	cell	3	http://github.com/Zhangxf-ccnu/mtANN
mtSC [95]	Python	cell	16	http://github.com/bm2-lab/mtSC
N-ACT [96]	Python	cell	1	http://ttps://github.com/SindiLab/N-ACT?tab=readme-ov-file
NeuCA [97]	R	cell	14	http://github.com/haoharryfeng/NeuCA?tab=readme-ov-file
CellLM [98]	Python	cell	17	http://github.com/PharMolix/OpenBioMed
OnClass [99]	Python	cell	50	http://github.com/wangshenguiuc/OnClass
Pollock [100]	Python	cell	2	http://github.com/ding-lab/pollock
scANNA [101]	Python	cell	1	http://github.com/SindiLab/scANNA
scBERT [102]	Python	cell	288	http://github.com/TencentAILabHealthcare/scBERT
scCapsNet [103]	Python	cell	52	http://github.com/wanglf19/scCaps
scDeepHash [104]	Python	cell	0	http://github.com/bowang-lab/scHash
scDeepInsight [105]	Python	cell	26	http://github.com/shangruJia/scDeepInsight
scDeepSort [106]	Python/R	cell	105	http://github.com/ZJUFanLab/scDeepSort
scDHA [107]	R/C+ +	cell	127	http://github.com/duct317/scDHA
scGraph[108]	Python	cell	22	http://github.com/QijinYin/scGraph
scMRA [109]	Python	cell	29	http://github.com/ddb-qiwang/scMRA-torch
scPretain [110]	Python	cell	11	http://github.com/ruiyi-zhang/scPretrain
scRCA [111]	Python	cell	0	http://github.com/LMC0705/scRCA
scSHARP [112]	Python/R	cell	4	http://github.com/lewinsohndp/scSHARP
Selina [113]	Python/R	cell	4	http://github.com/wanglabtongji/SELINA.py
sigGCN [114]	Python	cell	53	http://github.com/NabaviLab/sigGCN
SIMS [115]	Python	cluster	4	http://github.com/braingeneers/SIMS
SuperCT [116]	Python/R	cell	66	http://sct.lifegen.com/ (web app)
TOSICA [117]	Python	cell	126	http://github.com/JackieHanLab/TOSICA
scGPT [118]	Python	cell	415	http://github.com/bowang-lab/scGPT
MARS [119]	Python	cell	132	http://github.com/snap-stanford/mars

Tool: tool name; Impl:programming language/languages used for the implementation; AL: annotation level the tool annotates at single cell resolution; cluster: the tool can annotate only predefined clusters of cells; both: both single cell and cluster; #cited: number of times cited; repository: hyperlink to the software repository

The rapid advancement of DL has been driven by a combination of technological and algorithmic developments that have significantly improved the performance, scalability, and accessibility of more elaborate and powerful ML architectures such as deep neural networks [24].

Neural networks (NN) are a class of machine learning models inspired by the structure and function of the human brain. NNs consist of layers of interconnected nodes (neurons) that process and transform data. Each neuron receives input data, which are weighted and summed to emit an output. Multiple layers of neurons can be arranged and interconnected to deconstruct data and learn patterns and the output of every neuron can also be weighted, depending on its importance in the decision process. The output layer -the final layer of a NN- collects the output from the other layers (inner-layers) and emits the final prediction. In simple terms, the process of training a neural network involves iteratively adjusting weights (of neurons and data) to optimize accuracy of the prediction of the model.

Deep neural networks (DNNs) are neural networks with more complex architectures, typically consisting of multiple hidden layers (whereas standard neural networks often have just one or two [23]). Due the increased complexity in their architecture DL models require larger dataset for training, and are significantly more computationally intensive compared to C-ML models. Consequently, dedicated hardware such as for example TSU and/or GPUs is needed. While these computational requirements represent a limitation for the construction of DL models, it also enables DL to process extremely large datasets, that would be computationally unfeasible for C-ML. Additionally, since DL performs optimally with large amounts of data, their accuracy is bound to increase as more data become available [120].

Differences in the implementation aside, the process of developing DL models is conceptually analogous to that of C-ML algorithms: the model learns from accurately labeled training data. Thus selecting a good reference dataset for training remains a critical step. Hence, the limited availability of consistently annotated reference datasets and challenges in the integrations of different datasets due to batch effects, pose a challenge also for DL.

However, since DL leverages sophisticated feature decomposition techniques such as autoencoders and transformers (see [121] for a more comprehensive discussion), which can effectively reduce noise and capture meaningful patterns in the data, in the presence of an adequately large body of data, DL based methods are more resilient to batch effect and noise compared to DL.

Since their intricate architecture and numerous layers of abstraction DL models are often regarded black boxes with results that are hard to explain. To mitigate these limitations DL methods can be specifically deployed for feature importance or feature attribution analysis (reviewed in [122]). These techniques help unravel the contribution of individual features or neurons in the model’s decision-making process, offering valuable insights into the data. Such methods are particularly useful for reconciling the results with biological knowledge and have been used in genomics, healthcare, and, more recently, in single-cell RNA sequencing (scRNA-seq) analysis [123].

1.6. 5- Hybrid methods (HM)

Hybrid methods (HM, Table 5 [124], [125], [126], [127], [128], [129], [130], [131], [132], [133], [134], [135], [136], [137], [138]) integrate different algorithms or computational approaches to leverage their complementary strengths and build methods with enhanced performance (Fig. 1E). For example, hybrid methods for cell type annotation in scRNA-seq might combine and integrate different SSL and C-ML approaches and/or even integrate data across different training sets to improve accuracy and generalizability. Since each distinct method builds on different assumptions and integrates different algorithms, HM represents a very broad category. Nevertheless all the methods in this category share some principles and commonalities.

Irrespective of the implementation, a key requirement for any HM is the need to efficiently combine and integrate results from different methods into a single prediction. This can be achieved through various approaches, ranging from simple majority consensus (where the final prediction is assigned based on the most frequently predicted label) to more sophisticated approaches that train an ad-hoc ML-model for label integration. Consensus based strategies help reduce bias, mitigate errors, and improve the overall reliability of cell type classification, particularly in the context of complex, high-dimensional data [23]). However, these improvements come at the cost of increased computational demands, since the need to train and optimize multiple models. Moreover- another crucial consideration- the implementation of the consensus strategy is not banal and might require continuous refinement and manual curation as new classes/data labels are introduced. Similar to any other ML based methods, also HM can correctly detect/classify only the labels that were observed in the training set, and the addition of new classes requires further rounds of training. In HM this problem is exacerbated by the need to train multiple models. As a consequence the choice of the reference dataset used for training and development remains the most critical point. In summary, although consensus based approaches might enhance the predictive power of a tool, they are still subject to the same limitations that affect the underlying methods they integrate. Finally, since multiple independent methods contribute to the final prediction, the explainability of HM tools is generally lower compared to methods based on a single predictor/approach.

1.7. Benchmarking cell type annotation

Several recent studies engaged in the task of comparing the effectiveness of a selection of the computational methods illustrated in this review (and by extension of the underlying computational approach) by evaluating their performances in a controlled environment with carefully selected high quality annotated data.

Since cell identity annotation is a classification problem, where the goal is to attach the correct label to the correct cell, metrics commonly used for the assessment of ML-classifiers are employed also for the comparison of methods for the annotation of cell identity. These include:

• •Accuracy: total proportion of correct predictions (correct annotations) made by the model;
• •Sensitivity: (or rate of TRUE positives). Considering all the cells that were annotated under a cell type, what proportion was actually of that type?
• •Specificity: (or rate of TRUE negatives). Considering all cells that were not annotated under a cell type, what proportion was not of that type?
• •F1-Score: harmonic mean of the sensitivity and the specificity. The harmonic mean is used to balance the results in the presence of classes (cell types) with different abundances.

Accuracy and F1-Score are used to summarize the overall performance of a method across all cell types in a dataset, while sensitivity and specificity can be used to evaluate/compare the efficiency in the annotation of different cell types.

In 2019, Adelaal et al. evaluated the performances of 22 distinct methods for cell type annotation from scRNA-seq data by assessing their respective median F1-scores on a carefully constructed reference benchmark [140]. This benchmark was assembled to represent different use cases through the inclusion of datasets generated with different technologies and protocols, from different model organisms, and with varying levels of complexity. Each dataset included a different number (ranging from 1500 to 65000) of murine or human cells collected from pancreas, brain and blood samples and annotated at different levels of granularity. We refer the reader to the original study for a detailed description. Since its introduction the benchmark and evaluation criteria proposed by Abdelaal et al. have been widely adopted to compare and assess newly developed methods, but also for new benchmarking studies [141], [142]. As a result these data are considered a golden standard in the field.

Table 6 collects the accuracies of 14 distinct tools discussed in this review for the classification of human pancreatic cells as reported by Abdelaal et al. and subsequent studies. While all methods demonstrated high accuracy, subtle differences can still be observed with some specific methods such as CaSTLe and SingleCellNet (C-ML), SingleR (SLL), sigGCN and scDeepHash (DL) or scMayoMap (MB) achieving an almost perfect median F1-score across all the datasets considered. However, considering the limited differences in performance, and the fact that tools based on different conceptual approaches yielded comparable results, it is hard to conclude that any of the five categories discussed in this review should have a definitive advantage.

Table 6.

F1-scores on pancreas benchmark data.

Tool	Category	Baron	Muraro	Xin	Segerstolpe
scMayoMap	MB	1.00	1.00	1.00	1.00
ACTINN	DL	0.98	0.97	0.95	1.00
Cell BLAST	DL	0.89	0.79	0.63	0.08
scBERT	DL	0.85	0.93	0.79	0.76
scDeepHash	DL	0.99	0.98	0.99	1.00
sigGCN	DL	0.97	1.00	0.99	1.00
scPred	C-ML	0.98	0.98	0.95	1.00
CaSTLe	C-ML	0.94	0.96	0.96	0.98
SingleCellNet	C-ML	0.96	0.97	1.00	0.99
CHEETAH	C-ML	0.94	0.96	0.96	0.97
scmapcell	SSL	0.98	0.97	0.73	1.00
scmapcluster	SSL	0.95	0.97	1.00	1.00
SingleR	SSL	0.97	0.95	0.99	0.97
ScID	SSL	0.59	0.95	0.80	0.85

Tool: tool name; Category: category, as defined in this review. Median F1-scores are reported in a distinct column for every dataset considered

2. Discussion and conclusion

This short review provides a concise and up to date summary of the most used computational approaches for the annotation of cellular identities in scRNA-seq. As clearly outlined in Table 1, Table 2, Table 3, Table 4, Table 5 a considerable number of methods and analytical strategies have been developed in the last few years to tackle this problem. However, in this relatively young field, the quest for the development of the “one size fits all” definitive computational method to cell type annotation in scRNA-seq remains open. This is due to limitations both in the computational methods and in the data.

Indeed, any computational strategy discussed in this study presents important limitations that are yet to be solved. For example, DL needs extensive training data, is not easily human interpretable and requires dedicated hardware. C-ML and SSL are less computationally demanding compared to DL, but require well annotated/homogenised training sets -which are not currently available; additionally they need to be constantly updated and refined to recognise and classify previously unknown cell types. MB methods are more tractable and interpretable but require a very extensive biological knowledge which is probably beyond reach at the time being. HM tries to mitigate the limitation of each individual approach by integrating different algorithms into a single “consensus” prediction, however the efficient integration of prediction/classification from different methods is not always straightforward and represents a separate problem per se. Moreover the application of different methods fragments the decision problem and results in a loss of interpretability.

From the data point of view, the relatively low resolution of single cell RNAseq, where only a handful of transcripts can be detected from each cell, is probably the most critical issue. The limit threshold of the technology for the detection of distinct cell types is not currently known and even worse, this limit might be different depending on the sequencing strategy and library preparation strategies. Another crucial limitation (see [20] for an in depth discussion) is the lack of a clear and universally accepted definition of “cell type”. Indeed, notwithstanding the recent development of standardized ontologies and taxonomies for cell type representation, these resources do not completely match/reproduce the level of granularity of expert manual annotation and large inconsistencies in the annotation of the same cell-type or dataset can often be observed [143]. These ambiguities in classification do not provide an ideal framework for the development of computational methods, and especially for the development of ML-based applications where the “correct” labels are learned from the data.

In conclusion, large scale initiatives and coordinated efforts by the scientific community for the harmonisation/homogenization of analytical protocols and the interpretation of key findings - for example the Human Cell Atlas- represent a fundamental prerequisite for the development of the field, and will be decisive to aid in the development of highly accurate and comprehensive methods for the annotation of cellular identities and the future.

In the light of the above considerations and considering the open challenges in the field we believe we can offer the following advice to guide the reader in the selection of the most appropriate tool for cell identity annotation in their scRNA-seq data:

• 1.Make sure to follow best practices in the analyses of the data. Systematic differences/mistakes might systematically confound cell type annotation;
• 2.Check carefully which cell types can be natively annotated by a tool and favour tools that were developed/trained to annotate cell type of interest (adding custom configurations might be challenging for non expert and new users);
• 3.If/when available, execute exploratory analyses on publicly available data and replicate the results. First hand experience can greatly aid the selection of the “best” method for your application;
• 4.Clearly define the key objectives. Is explainability/interpretation of the model/classification important? If so, avoid overly complex models with several layers of abstraction. Would you like to replicate the annotation as in a specific study? Probably the best option is to select the same exact method;
• 5.Is customization required? At the time being probably MB based methods are more easily customized and adapted while training or modifying C-ML models might be more complex. DL methods cannot be modified/installed/configured by an end user. Dedicated hardware is required;
• 6.Manual annotation might still be required for the identification of novel cell types not represented in publicly available databases. Do not blindly trust automatic classification.

Statement

We the undersigned declare that this manuscript is original, has not been published before and is not currently being considered for publication elsewhere.

Funding and acknowledgements

This work was supported by Ministero dell’Università e della Ricerca, National Center for Gene Therapy and Drugs Based on RNA Technology [project number CN_00000041 to M.C.]; European Life-Science Infrastructure—Italy (ELIXIR-IT), ELIXIR x NextGenerationIT (ELIXIRxNextGenIT) project [grant code IR0000010 to M.C].

CRediT authorship contribution statement

Traversa Daniele: Writing – original draft, Investigation, Conceptualization. Chiara Matteo: Writing – review & editing, Supervision, Conceptualization.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.