Ensemble machine learning-based pre-trained annotation approach for scRNA-seq data using gradient boosting with genetic optimizer

Abstract

Single-cell RNA sequencing (scRNA-seq) has revolutionized the study of gene expression by allowing researchers to analyze the transcriptomes of individual cells. This technology provides unprecedented insights into cellular heterogeneity, cellular states, and biological processes at a single-cell resolution. The problem of single-cell RNA annotation involves assigning meaningful labels or annotations to each cell in the scRNA-seq dataset, indicating its corresponding cell type, state, or biological function. Current annotation methods are often challenged by limited source data quality, sensitivity to batch effects, and poor adaptability to uncharacterized cell types. We propose an ensemble machine learning-based pre-trained annotation framework that integrates gradient boosting and genetic optimization for robust feature selection. The proposed method uses ensemble learning to enhance annotation accuracy under data scarcity, addressing limitations in existing supervised methods by leveraging a combination of multiple annotated datasets and feature alignment strategies. Through comprehensive benchmarking across varied biological contexts, we demonstrate that the proposed approach significantly improves annotation accuracy and generalization across different scRNA-seq platforms, especially under conditions of reduced reference data. Results confirm its versatility and resilience in accurately annotating cell types, even under reduced data conditions, establishing it as a powerful tool for cell-type classification in scRNA-seq data.

Introduction

Single-cell RNA sequencing (scRNA-seq) is a transformative technology that enables researchers to analyze transcriptomes at the individual cell level, unlike bulk RNA-seq, which examines mixed cell populations. Rapid advancements in single-cell omics have significantly impacted multiple biological fields, including developmental biology [1], neurology [2], oncology [3], immunology [4], cardiovascular research [5], and infectious disease [6]. These developments have facilitated deeper exploration into these disciplines, allowing for substantial progress in understanding complex cellular dynamics [7]. A primary objective of scRNA-seq data analysis is accurately identifying cell types for each cell [8, 9]. This objective has led to the development of numerous computational methods, which can be broadly categorized into marker gene-based methods and annotated dataset-based methods for cell-type identification [10, 11].

Marker gene-based methods rely on identifying and labeling clusters formed by known marker genes following the clustering of scRNA-seq data [12–15]. This approach has played a pivotal role in early-stage cell-type annotation in scRNA-seq analysis. However, advancements in sequencing technology now allow for the processing of hundreds of thousands of cells per experiment [16]. Alongside the proliferation of published research and the advent of large-scale projects, such as mouse and human cell atlases [17–20], scRNA-seq studies have amassed an extensive collection of annotated datasets. Despite these advances, manual annotation based on marker genes has become time-consuming and is less effective when identifying cell types without canonical markers [8].

Annotated dataset-based methods have emerged as alternatives, involving two primary approaches. The first approach calculates gene expression profile correlations between query and reference datasets to determine cell labels [21–23]. The second employs machine-learning models trained on annotated datasets, which are subsequently used to label cells in new datasets [24–26]. In recent years, deep learning models have been widely applied to cell-type annotation in scRNA-seq, leveraging architectures such as autoencoders [27], variational autoencoders [28], and generative adversarial networks [29] for tasks like clustering, cell-type identification, and pseudo-time inference. Additionally, graph-based models have gained prominence, utilizing gene-gene or cell-cell interactions to enhance annotation accuracy [30–32]. One common approach involves clustering cell populations, identifying cluster-specific marker genes through differential expression analysis, and annotating cells based on gene ontologies.

While existing methods have contributed significantly to the advancement of scRNA-seq annotation, they each exhibit notable limitations. For instance, methods like scmap and Seurat rely heavily on high-quality reference datasets and often struggle with batch effects or heterogeneous sequencing technologies, limiting their generalizability [33]. Deep learning-based approaches like MARS and ItClust have improved adaptability and representation learning but often require large training datasets and still suffer from information loss during domain adaptation [38]. Furthermore, graph-based techniques, while powerful, can be computationally intensive and sensitive to graph construction choices. These limitations highlight the need for more robust and flexible annotation methods that can perform reliably even in low-data regimes, handle high-dimensional data, and generalize across diverse biological contexts. This motivates the development of the ensemble-based framework proposed in this work.

Related work

In recent years, numerous single-cell clustering algorithms have emerged. For example, Seurat utilizes the Louvain community discovery algorithm to partition the shared neighbor graph among cells [33]. Another notable method, SC3, employs the k-means algorithm on a cell similarity matrix derived from consensus learning to achieve cluster segmentation [34]. However, the effectiveness of marker gene identification varies significantly across different experiments, which poses challenges in the generalization performance of these methods [12]. Moreover, as the scale of sequencing data increases, the process of annotating cells through marker gene identification becomes increasingly burdensome and time-consuming [35].

In the evolving field of single-cell RNA sequencing (scRNA-seq) data annotation, the use of cell-type labels from well-annotated reference datasets has gained prominence recently. This way signifies a shift toward utilizing previously established knowledge to enhance the annotation accuracy of newly generated datasets. One significant method in this area is scmap, which projects query cells onto reference data and evaluates their similarity to known cluster centroids or existing cells in the reference dataset. According to Kiselev et al. [36], scmap offers a robust framework for managing large-scale datasets, allowing for effective cell-type annotation. However, this method primarily relies on the cell-type information contained within the reference datasets, which can lead to performance issues when faced with technical variations or batch effects. Such challenges are particularly pronounced when the target and reference data arise from disparate sequencing technologies.

Another prominent tool, Seurat enhances the annotation process by performing supervised classification and prediction on target datasets. It achieves this by identifying anchoring cell pairs and constructing shared neighbor networks between well-annotated reference data and unlabeled target data [37]. While Seurat’s approach is effective, it does not fully leverage specific cell-type knowledge embedded within the reference datasets. Furthermore, the computational demands associated with feature alignment and realignment for mixed query and reference cells can hinder its scalability and efficacy, particularly as the size of the datasets grows.

To address these limitations, recent developments have turned to deep learning methodologies that aim to improve the accuracy and efficiency of cell-type identification. For instance, MARS and ItClust are two notable deep clustering methods designed for cell-type identification from single-cell RNA-seq data. MARS, as described by Brbić et al. [38], employs a deep autoencoder to pretrain on unlabeled target datasets, learning cell-type landmarks and non-linear embeddings. While MARS can effectively identify novel cell types, its reliance on distance metrics for predicting cell types can overlook critical batch effects between reference and target datasets, resulting in potential overfitting. Similarly, ItClust, introduced by Hu et al. [39], utilizes transfer learning to capitalize on cell-type-specific gene expression information from reference data. However, ItClust also grapples with challenges related to batch effects and information loss, which can compromise its predictive accuracy.

In addition, scMRA integrates a knowledge graph to represent cell-type characteristics across various datasets. This approach, highlighted by Yuan et al. [40], employs a graph convolutional network to maintain the relative positioning of cell types while preserving intra-cell-type relationships. Although scMRA offers advancements in handling multiple reference datasets, it still contends with issues of annotation consistency and computational efficiency.

Motivation and contribution

Despite the advances in scRNA-seq data annotation methods, significant challenges remain, particularly in dealing with the variability introduced by different sequencing platforms and batch effects. Many existing methods, such as scmap and Seurat, rely heavily on well-annotated reference datasets but struggle with generalization when faced with heterogeneous data sources. This can lead to diminished performance, particularly when new cell types are present that were not represented in the reference data. Furthermore, the reliance on extensive annotated reference datasets limits their applicability, particularly in scenarios of data scarcity. From our review of the relevant literature, the proposed framework aims to address the inadequacies observed in current methodologies. In this paper, we introduce an Ensemble Machine Learning-Based Pre-Trained Annotation Framework (ScEMLA) that leverages genetic optimization for feature selection. By integrating an ensemble machine learning approach that incorporates genetic optimization for feature selection, the prposed framework is designed to enhance the performance of scRNA-seq data annotation, especially under conditions of data scarcity and batch effects. The key contributions of this paper can be outlined as follows:

• Development of a Hybrid Annotation Framework: this paper introduces a novel hybrid framework that integrates multiple machine learning models through an ensemble approach. This framework effectively combines the strengths of different models to achieve higher annotation accuracy in scRNA-seq data, particularly when dealing with complex and heterogeneous datasets.

• Genetic Algorithm-Driven Feature Selection: a significant contribution is the incorporation of a genetic algorithm for feature selection within the annotation process. This approach optimizes the selection of relevant gene markers, reducing dimensionality while maintaining critical biological information. The genetic optimizer enhances model performance by focusing on the most informative features, leading to improved annotation outcomes.

• Enhanced Performance under Data Scarcity: the proposed method is extensively validated under conditions of varying data availability, including scenarios with mild, moderate, and severe reductions in training data. The framework consistently demonstrates superior performance, maintaining high annotation accuracy even when data is limited, thus proving its robustness in real-world applications where data scarcity is a common challenge.

• Comprehensive Benchmarking Against Existing Methods: the research conducts a thorough comparative analysis, benchmarking the proposed framework against leading annotation methods such as scMRA, ItClust, Scmap, and Seurat. The results clearly show that the proposed ensemble model outperforms these methods across various metrics, establishing a new standard for scRNA-seq data annotation accuracy and reliability.

• Versatility Across Different Biological Contexts: while the primary focus of the framework is on scRNA-seq data annotation, its underlying principles are adaptable to other types of omics data. The modular design of the ensemble and genetic optimization approach makes it a versatile tool that can be applied to various biological datasets, potentially extending its impact beyond single-cell transcriptomics.

• Improvement in Annotation Accuracy Across Clusters: the framework’s ability to improve annotation accuracy as the number of clusters and source domains increases is a noteworthy contribution. This characteristic is particularly important for complex biological datasets where the accurate classification of distinct cell types is critical. The research demonstrates that the proposed method can effectively scale with increasing data complexity, providing accurate and reliable annotations across a wide range of experimental conditions.

In contrast to existing approaches such as scmap, Seurat, ItClust, and MARS, which suffer from challenges like reliance on canonical markers, sensitivity to batch effects, and performance degradation under data scarcity, the proposed method uniquely integrates an ensemble learning framework with a genetic optimizer for robust feature selection. The core innovation lies in combining weak learners to boost prediction accuracy while dynamically selecting the most informative genes using an evolutionary strategy. This dual design allows the proposed model to generalize better across diverse biological conditions and perform reliably even when reference datasets are sparse or partially annotated. These distinguishing elements position the proposed approach as a scalable and adaptive alternative to conventional single-model or marker-dependent annotation pipelines.

The proposed scheme significantly advances the field of scRNA-seq data analysis by introducing a robust, scalable, and highly accurate annotation framework. The combination of ensemble learning with genetic optimization offers a powerful solution to the challenges of data scarcity and high dimensionality, setting a new benchmark for future research in bioinformatics and single-cell analysis.

Materials and methods

In this study, we propose a robust scheme aimed at enhancing the training phase of deep learning models when applied to single-cell data sequences. Our approach involves a series of preprocessing steps, explained in the following, which collectively prepare the data for improved model training. Then the proposed scheme is examined to annotate the target dataset as shown in Fig. 1

Fig. 1.

Proposed Model Framework

Data preprocessing

Recent studies have pointed out that scRNA-seq datasets often exhibit biological heterogeneity and technical biases. To mitigate the effects of these factors, the raw scRNA-seq read counts were pre-processed using the DESeq2 R package.” Let’s denote the raw single-cell RNA sequencing data as X. The data preprocessing step can be represented as a transformation function $f_1$ that maps the raw data X to a preprocessed dataset $X^{\prime }$.

$$\begin{array}{c}\hfill X^{\prime }=f_1(X),\end{array}$$ 1

where f1 is the Preprocessing function (e.g., removing contaminants, initial filtering) Input: Raw scRNA-seq data $X\in R^{n\times p}$

Output: Preprocessed data $X^{\prime }\in R^{n^{\prime }\times p^{\prime }}$

Where n, p, n’, p’ represent the number of cells, the number of genes, the number of filtered cells, and the number of filtered genes, respectively.

Quality control

Quality control Ensuring that only high-quality cells and genes are used in the analysis. It consists of two steps Cell filtering and Gene filtering. Cell filtering Remove cells with low gene counts or high mitochondrial gene content, which can indicate poor-quality or dying cells. While Gene Filtering exclude genes not expressed in a significant number of cells to reduce noise. In the proposed scheme, quality control is implemented as a filtering function $f_2$, which is applied to the preprocessed data, $X^{\prime }$ to generate a refined, high-quality dataset $X_{\mathit{QC}}$.

$$\begin{array}{c}\hfill X_{\mathit{QC}}=f_2(X^{\prime }),\end{array}$$ 2

where $f_2$ represents a quality control function, such as filtering cells with low gene counts and high mitochondrial content.

Gene marker identification

Following the quality control step, gene marker identification is performed on the quality-controlled dataset. Gene marker identification entails selecting key genes from the quality-controlled dataset. This process focuses on identifying marker genes that are differentially expressed between distinct cell populations, allowing for the characterization and differentiation of those populations based on specific gene expression profiles. Let M be the set of marker genes, which is derived from $X_{\mathit{QC}}$ using a marker identification function $f_3$ to produce set of marker genes $M\subset \left\{1,,,2,,,\cdots ,,,p^{\prime }\right\}$.

$$\begin{array}{c}\hfill M=f_3(X_{\mathit{QC}}),\end{array}$$ 3

where $f_3$ is the differential expression analysis as the Marker identification function.

Data normalization

The data normalization step is added to adjust the expression levels across to make them comparable. In the proposed scheme, the data normalization is represented by a normalization function $f_4$, that transforms $X_{\mathit{QC}}$ into a normalized dataset $X_{\mathit{norm}}$. We scale the data to ensure that comparisons across cells are meaningful, typically by normalizing the total gene expression. Moreover, a log transformation is applied to stabilize the variance and reduce skewness.

$$\begin{array}{c}\hfill X_{\mathit{norm}}=f_4(X_{\mathit{QC}}),\end{array}$$ 4

where $f_4$ is the normalization function, e.g., log normalization.

Dimensionality reduction

After applying the quality control function and identifying key gene markers, dimensionality reduction is employed to manage the complexity of the high-dimensional data. This step is essential to streamline subsequent analysis, reducing computational load while retaining the most relevant features of the dataset. Principal Component Analysis (PCA) is used as the initial method for dimensionality reduction, capturing the key variations in the data. Following this, t-distributed Stochastic Neighbor Embedding (t-SNE) is applied to further reduce dimensions, particularly for visualization purposes, allowing us to intuitively interpret the clustering of cells or features in two dimensions. Let Z be the low-dimensional representation of the normalized data $X_{\mathit{norm}}$, obtained using a reduction function $f_5$.

$$\begin{array}{c}\hfill Z=f_5(X_{\mathit{norm}}),\end{array}$$ 5

Principal component analysis (PCA)

PCA was selected to reduce the dimensionality of the data by transforming it into a set of orthogonal components that capture the highest variance. Given the dataset $X_{\mathit{norm}}\in {\mathbb{R}}^{n^{\prime }\times p^{\prime }}$, where $n^{\prime }$ represents the number of observations (e.g., cells) and $p^{\prime }$ represents the number of variables (e.g., gene expression values), PCA computes a new set of variables (principal components) that are linear combinations of the original features. This transformation reduces the dimensionality while preserving as much variability in the data as possible.

Mathematically, the principal components are obtained by solving the eigenvalue problem for the covariance matrix $\mathrm{\Sigma }$ of the centered data $X_{\mathit{norm}}$:

$$\begin{array}{c}\hfill \mathrm{\Sigma }=\frac{1}{n^{\prime }}{X}_{\mathit{norm}}^T{X}_{\mathit{norm}}\end{array}$$

Here, $\mathrm{\Sigma }$ represents the covariance matrix of the data. The eigenvalues ${\lambda }_1,{\lambda }_2,\cdots ,{\lambda }_p$ correspond to the amount of variance explained by each principal component, and the eigenvectors $w_1,w_2,\cdots ,w_p$ are the directions (or principal components) in which the data varies the most. The data is then projected onto these principal components to form the reduced-dimensional dataset:

$$\begin{array}{c}\hfill Z=X_{\mathit{norm}}{W}_k\end{array}$$

where $W_k$ consists of the top k eigenvectors associated with the largest eigenvalues. The number of principal components k is chosen to retain the majority of the dataset’s variance, while reducing the noise and redundant information.

t-SNE for visualization

While PCA effectively reduces dimensionality for computational purposes, it may not always provide the most interpretable results for visualization. To address this, t-SNE is subsequently applied to the PCA-reduced data, primarily for visualization in two dimensions. t-SNE is particularly well-suited for capturing local similarities and revealing structure in the data, which can be lost when using linear methods like PCA.

By combining PCA and t-SNE, we ensure that the essential variance in the data is preserved, while still obtaining an interpretable two-dimensional representation for exploratory data analysis and visualization.

Clustering of reduced data

After performing dimensionality reduction, a clustering step was introduced to group the reduced data Z into meaningful clusters. The reduced data from PCA and t-SNE retains the most significant features, which facilitate accurate clustering of the dataset. In this study, we applied clustering techniques to identify potential subgroups or patterns within the data.

Given the reduced representation Z, a clustering algorithm was employed to assign each data point to one of C clusters. The clustering approach segments the data based on similarities between observations in the lower-dimensional space, allowing us to identify distinct groups that may correspond to different biological states or cell types.

The clustering step enhances the interpretability of the data by organizing it into coherent subgroups. These clusters can then be used for further analysis, such as identifying marker genes or functional pathways that are specific to each group.

$$\begin{array}{c}\hfill \widehat{C}=f_6(Z)\end{array}$$

where $f_6$ represents clustering function, and $\widehat{C}$ represents the set of clusters obtained from the reduced data Z. The resulting clusters provide valuable insight into the structure of the dataset, supporting downstream analysis and interpretation. The detailed steps of the proposed scheme are outlined in Algorithm 1.

To clarify the full workflow of the proposed annotation pipeline, we have formulated Algorithm 1 to explicitly describe each stage from raw data preprocessing to final label prediction. The pipeline integrates quality control, marker gene identification, feature reduction, and model training, followed by a genetic algorithm-based feature selection to enhance gradient boosting performance.

Algorithm 1.

Proposed Framework

Hyperparameter selection strategy

To ensure the proposed framework achieves optimal performance while remaining computationally efficient, we adopted the following strategy for determining model parameters:

Gradient boosting parameters

We performed an empirical grid search over a predefined set of parameters. The key hyperparameters and their selected values are as follows: the learning rate ($\eta$) was set to 0.003 to ensure stable convergence and avoid overfitting. The number of estimators was set to 100 to achieve a balance between predictive performance and training time. The maximum tree depth was limited to 3 in order to control model complexity and reduce the risk of overfitting. Finally, the subsample ratio was set to 0.8 to introduce randomness during training and enhance the model’s generalization capability. The optimal parameter combination was selected based on 5-fold cross-validation, with evaluation metrics including accuracy and F1-score.

Genetic algorithm parameters

The hyperparameters of the genetic algorithm were determined based on prior studies and further fine-tuned through empirical testing. The population size was set to 50, and the number of generations was also fixed at 50. A crossover rate of 0.8 and a mutation rate of 0.02 were chosen to maintain genetic diversity and convergence speed. Additionally, we incorporated a feature penalty term ($\lambda$) set to 0.01 to balance between model sparsity and performance. These values provided stable and consistent results across a variety of datasets while maintaining computational feasibility.

Dimensionality reduction

To reduce the dimensionality of the input data, we applied Principal Component Analysis (PCA) and retained the components that explain 95% of the total variance. This typically resulted in retaining 30 to 50 principal components, depending on the variability and size of the dataset being analyzed.

Clustering parameters

For the clustering step, we used the k-means algorithm. The number of clusters was determined based on the expected number of cell types present in the dataset. This estimation was guided by prior biological knowledge, available metadata, and exploratory data analysis using t-SNE plots.

Each set of parameters was validated by measuring annotation accuracy and assessing robustness, particularly in data scarcity scenarios. This helped ensure that the chosen configuration generalized well across different conditions.

Proposed scheme framework

The proposed framework introduces a novel ensemble machine learning-based approach to accurately annotate single-cell RNA sequencing (scRNA-seq) data. A pivotal aspect of this framework are two key components: Gradient Boosting and Genetic Optimization, which collectively enhance both the accuracy and robustness of the annotation process. Gradient Boosting serves as the core machine learning model, leveraging its ability to iteratively optimize prediction errors by combining weak learners into a strong predictive ensemble. To further refine this process, Genetic Optimization is employed to select the most relevant features and hyperparameters, ensuring the model’s adaptability to varying biological contexts and datasets. This synergistic integration addresses the challenges posed by data heterogeneity and scarcity, establishing a foundation for reliable cell-type annotation.

Gradient boosting

Gradient Boosting forms the cornerstone of the proposed framework, leveraging its capacity to build a strong predictive model by iteratively combining weak learners. At its core, Gradient Boosting minimizes a differentiable loss function, $L(y,\widehat{y})$, through gradient descent in functional space.

Model formulation

Let the dataset consist of $n$ samples ${\{x_i,y_i\}}_{i=1}^n$, where $x_i\in {\mathbb{R}}^d$ represents the feature vector, and $y_i$ is the target variable. The goal is to find a function $F(x)$ that minimizes the expected value of the loss function:

$$\begin{array}{c}\hfill \mathbb{E}[L(y,F(x))]=\frac{1}{n}\sum _{i=1}^{n}L(y_i,F(x_i)).\end{array}$$

Gradient Boosting starts with an initial model $F_0(x)$, often a constant:

$$\begin{array}{c}\hfill F_0(x)=arg\underset{c}{min}\sum _{i=1}^{n}L(y_i,c).\end{array}$$

Then, it iteratively updates $F(x)$ by adding new models $h_m(x)$, known as weak learners, weighted by ${\alpha }_m$:

$$\begin{array}{c}\hfill F_m(x)=F_{m-1}(x)+{\alpha }_m{h}_m(x).\end{array}$$

Here, $h_m(x)$ is selected to approximate the negative gradient of the loss function at $F_{m-1}(x)$:

$$\begin{array}{c}\hfill h_m(x)=arg\underset{h}{min}\sum _{i=1}^n{\left[-,\frac{\partial L(y_i,F_{m-1}(x_i))}{\partial F(x_i)},-,h,(x_i)\right]}^2.\end{array}$$

Implementation in scRNA-seq annotation

For scRNA-seq data, Gradient Boosting handles the heterogeneity by learning complex relationships between the input features (e.g., gene expression profiles) and the target cell-type annotations. The iterative optimization ensures that the model learns subtle patterns while correcting for previous errors, making it well-suited for noisy and high-dimensional datasets.

Genetic optimization for feature selection

Genetic Optimization is a bio-inspired algorithm that optimizes the feature selection process by simulating natural evolution. In the proposed framework, the proposed genetic optimization enhances Gradient Boosting by identifying the most informative features and optimal hyperparameters, improving both accuracy and computational efficiency.

Genetic algorithm components

1. Population Representation: Each individual in the population represents a candidate solution encoded as a binary vector $\mathbf{v}\in {\{0,1\}}^d$, where $v_j=1$ indicates that the $j$-th feature is selected.

2. Fitness Function: The fitness of an individual is evaluated based on the cross-validation accuracy $A(\mathbf{v})$ of the Gradient Boosting model trained on the selected features:

   $$\begin{array}{c}\hfill \text{Fitness}(\mathbf{v})=A(\mathbf{v})-\lambda ·{\Vert \mathbf{v}\Vert }_1,\end{array}$$

   where $\lambda$ is a regularization parameter penalizing the number of selected features.

3. Selection: Individuals with higher fitness are more likely to be selected for reproduction, modeled using roulette-wheel or tournament selection.

4. Crossover and Mutation: Genetic diversity is maintained through crossover, which combines two parent solutions, and mutation, which randomly flips bits in the vector $\mathbf{v}$.

Optimization procedure

The algorithm iteratively evolves the population over $g$ generations, refining the selection of features. At each generation:

• The best individuals are retained as elites.

• A new population is generated using crossover and mutation.

• The fitness of the updated population is evaluated.

Mathematically, the optimization process can be expressed as:

$$\begin{array}{c}\hfill {\mathbf{v}}^{\ast }=arg\underset{\mathbf{v}}{max}\text{Fitness}(\mathbf{v}),\end{array}$$

where ${\mathbf{v}}^{\ast }$ is the optimal feature selection vector.

Integration with scRNA-seq data

In scRNA-seq annotation, Genetic Optimization ensures the framework’s adaptability by selecting features that maximize cell-type classification performance while reducing irrelevant or redundant genes. The synergy between Genetic Optimization and Gradient Boosting creates a robust and efficient annotation pipeline. The detailed steps of Gradient Boosting and Genetic Optimization are outlined in Algorithm 2.

Results and discussion

The proposed ensemble machine learning-based pre-trained annotation framework operates with two primary datasets: a training dataset comprising cells with well-annotated cell type labels and a target dataset containing cells that require clustering and annotation. The framework begins by building an ensemble model that combines multiple machine learning algorithms to capture complex relationships in the annotated data. This is followed by a genetic algorithm-driven feature selection process, which extracts the most informative gene markers for accurate cell type classification. The resulting model effectively optimizes the annotation accuracy of the target dataset, even in scenarios with reduced training data.

The proposed framework then applies a clustering algorithm to form coherent clusters within the target dataset, leveraging the ensemble model to predict annotations with high confidence. To ensure resilience in data-limited settings, the framework is validated under varying data availability scenarios, consistently demonstrating superior accuracy and adaptability. The ensemble model’s adaptability to diverse biological contexts and its ability to improve annotation accuracy across clusters establish a new standard for scRNA-seq data annotation.

We extensively evaluate our proposed framework against several state-of-the-art annotation methods, including scMRA, ItClust, Scmap, and Seurat. All methods are executed using their default configurations and parameter settings to ensure a fair comparison. For our ensemble model, the key hyperparameters were fine-tuned to optimize performance, with a fixed learning rate of 0.003 and an ensemble size of 10 models. The genetic algorithm employed a crossover probability of 0.8 and a mutation rate of 0.02, evolving over 50 generations to identify the most informative features.

For our experiments, we utilized six scRNA-seq datasets spanning multiple human tissues and sequencing protocols: Enge (Smart-seq2), Segerstolpe (Smart-seq2), Muraro (CEL-seq2), as well as additional datasets from spleen, tongue, and trachea. Due to its comprehensive cell type annotations, the Muraro dataset was designated as the reference, while the other datasets served as targets in our annotation experiments. Our framework’s performance was evaluated across these datasets and compared with existing methods, demonstrating enhanced accuracy and scalability. Even under scenarios of severe data scarcity, the framework demonstrates significant robustness, maintaining high performance. These results underscore the effectiveness of our ensemble-based approach in achieving reliable and accurate annotations in complex scRNA-seq datasets. Detailed annotation results for Enge, Segerstolpe, spleen, tongue, and trachea datasets, alongside comparisons with competing approaches, are presented in the following subsections.

Annotation accuracy

Table 1 shows the annotation accuracy of the proposed ensemble machine learning-based pre-trained annotation framework in comparison to scMRA, MARS, ItClust, scmap, and Seurat methods across six datasets: Enge, Segerstolpe, Muraro, spleen, tongue, and trachea. The proposed method demonstrates superior performance, consistently achieving the highest accuracy rates across all datasets. Notably, the proposed framework achieves accuracy over $97\%$ in five out of six datasets, with the highest accuracy of $98.1\%$ on the Enge dataset, significantly outperforming the other methods.

Table 1.

Annotation Accuracy Comparison

Dataset	ScEMA	ScMRA	MARS	ItClust	Scmap	Seurat
Enge	0.981	0.961	0.891	0.876	0.851	0.882
Segerstolpe	0.975	0.967	0.884	0.864	0.843	0.871
Muraro	0.972	0.967	0.90	0.882	0.864	0.898
Spleen	0.967	0.964	0.885	0.861	0.851	0.881
Tongue	0.973	0.964	0.885	0.861	0.851	0.881
Trachea	0.952	0.931	0.887	0.886	0.844	0.873

For the Segerstolpe dataset, our framework reaches an accuracy of $97.5\%$, compared to scMRA’s $96.7\%$, and substantially surpasses MARS, ItClust, scmap, and Seurat, each of which shows accuracies below $89\%$. Similarly, in the Muraro dataset, our approach achieves an accuracy of $97.2\%$, again outperforming scMRA’s $96.7\%$ and maintaining a clear advantage over the remaining methods, with MARS reaching $90.0\%$, ItClust $88.2\%$, scmap $86.4\%$, and Seurat $89.8\%$.

The proposed method’s robustness is particularly evident in the more complex and heterogeneous datasets, such as spleen, tongue, and trachea. In the spleen dataset, our framework attains $96.7\%$ accuracy, outperforming scMRA’s $95.4\%$ and other methods, which show accuracies below $89.5\%$. In the tongue dataset, our method achieves $97.3\%$ accuracy, while scMRA reaches $96.4\%$. Finally, in the trachea dataset, the proposed framework achieves an accuracy of $95.2\%$, exceeding scMRA’s $93.1\%$ and showing resilience in data environments with increased complexity.

In comparison, competing methods such as MARS, ItClust, scmap, and Seurat demonstrate a notable decline in accuracy in complex datasets. This difference underscores the enhanced adaptability and scalability of the proposed framework in handling diverse scRNA-seq data. These results confirm the framework’s effectiveness.

As illustrated in Fig. 2, which shows the change in annotation accuracy with increasing cluster numbers and the number of source domains, we evaluated the performance of the proposed framework alongside scMRA, MARS, ItClust, scmap, and Seurat across 20 groups of simulated datasets. For each method and each group, we computed the mean annotation accuracy across 10 different datasets within that group, enabling us to observe the sensitivity of each method to both the number of clusters and the number of source domains.

Fig. 2.

Change of annotation accuracy with increasing cluster numbers and number of source domains

The results show that our proposed framework maintains a clear advantage across various parameter settings, particularly when multiple reference datasets are incorporated. As seen in Fig. 2, the annotation accuracy of all methods tends to decrease as the number of clusters increases, but the proposed framework demonstrates a more stable decline compared to the other methods, with the accuracy decreasing gradually and consistently.

With the addition of more source domains, the proposed framework’s performance steadily improves, highlighting its scalability and robustness in scenarios involving multiple reference datasets. In contrast, methods like Seurat and ItClust show fluctuating accuracy as the number of clusters and domains increases. Specifically, for settings with seven clusters, both Seurat and ItClust exhibit a decrease in accuracy as the number of source domains rises from two to three, while scmap’s accuracy declines when domains increase from four to five. Notably, only our proposed method consistently improves with additional source domains, underscoring its effectiveness in handling complex datasets with multiple sources. Overall, these findings underscore our framework’s superior performance in both stability and accuracy across varying cluster numbers and source domains, demonstrating its robustness in complex multi-reference dataset annotation tasks.

Robustness under insufficient data conditions

In these experiments, we tested the robustness of the proposed framework and four competing methods when the reference data was limited. Specifically, we used three experimental groups: Group 1 (30% data reduction), Group 2 (50% data reduction), and Group 3 (70% data reduction). Annotation accuracy results for each method across these conditions are presented in Table 2.

Table 2.

Annotation accuracy values on three groups of insufficient data experiments

Method	Group 1	Group 2	Group 3
ScEMA	0.845	0.838	0.813
ScMRA	0.832	0.828	0.8057
MARS	0.691	0.671	0.654
ItClust	0.84	0.794	0.786
Scmap	0.757	0.747	0.739
Seurat	0.796	0.773	0.762

The proposed method displayed considerable resilience to data reduction, achieving accuracy rates of 0.845, 0.838, and 0.813 for Groups 1, 2, and 3, respectively. This stability stems from the framework’s ensemble model combined with genetic optimization, which selectively emphasizes key features despite reduced data availability. In contrast, while scMRA maintained relatively high accuracy (0.832, 0.828, and 0.8057), other methods demonstrated more substantial declines as data reduction increased.

For instance, both Seurat and scmap showed moderate declines across groups, with accuracies ranging from 0.796 in Group 1 to 0.762 in Group 3 for Seurat, and from 0.757 to 0.739 for scmap. ItClust initially performed well under mild data reduction (0.84 in Group 1) but saw a notable drop by Group 3, with accuracy decreasing to 0.786. The MARS method was the most affected, with performance dropping from 0.691 to 0.654.

These results underscore the effectiveness of the proposed method in handling limited reference data, maintaining accuracy with minimal drop-off even under severe data scarcity. This characteristic makes the framework particularly suited for real-world applications where high-dimensional biological datasets often suffer from data limitations.

Algorithm 2.

Gradient Boosting and Genetic Optimization

Ablation studies and component analysis

To assess the individual contributions of each component in the proposed framework, we conducted comprehensive ablation studies by isolating key features of the model. The ablation study examined four distinct configurations: (1) Baseline model without genetic optimization, (2) Single model without ensemble learning, (3) Omission of dimensionality reduction, and (4) Variation in clustering algorithms. We evaluated the performance of these configurations across three experimental groups 30% (Group 1), 50% (Group 2), and 70% (Group 3) data reduction as shown in Fig. 3.

Fig. 3.

Annotation accuracy values in ablation studies across varying data reductions

Baseline model (without genetic optimization)

In this setup, the ensemble model was used without the genetic optimization for feature selection, thereby including all available features (genes) without filtering. This baseline configuration yielded annotation accuracies of 0.75, 0.73, and 0.70 across Groups 1, 2, and 3, respectively. The significant drop in accuracy, compared to the proposed framework, underscores the importance of genetic optimization in enhancing the model’s performance by eliminating irrelevant or redundant features.

Single model (without ensemble learning)

This setup replaced the ensemble approach with a single machine learning model (e.g., Gradient Boosting), which was trained and tested independently. The annotation accuracy achieved was 0.78 for Group 1, 0.76 for Group 2, and 0.74 for Group 3. The ensemble model consistently outperformed this single-model approach, highlighting the benefits of combining multiple models to capture diverse data characteristics and reduce both bias and variance.

Removal of dimensionality reduction

Here, dimensionality reduction (e.g., PCA) was omitted, and raw features were directly used for annotation. Annotation accuracies for Groups 1, 2, and 3 were 0.76, 0.74, and 0.72, respectively. The lower performance observed in this configuration demonstrates that dimensionality reduction aids in capturing the most informative features, thereby improving model performance, especially as data complexity increases.

Variation in clustering algorithms

In this setup, different clustering algorithms (e.g., k-means and hierarchical clustering) were applied. The annotation accuracy achieved with hierarchical clustering was 0.79, 0.77, and 0.75, while with k-means it was 0.80, 0.79, and 0.77 for Groups 1, 2, and 3, respectively. While the choice of clustering algorithm did impact the results, the framework maintained robustness across methods, with the ensemble approach effectively reducing variability in performance due to different clustering techniques.

These ablation studies clearly demonstrate that each component in the proposed framework significantly contributes to its overall performance. The genetic optimization step is particularly critical for improving annotation accuracy by selecting the most relevant features, while ensemble learning consistently outperforms single models, highlighting the value of a multi-model approach for robustness and accuracy. Dimensionality reduction is another key component, effectively managing high-dimensional scRNA-seq data. Finally, the choice of clustering algorithm, although having a smaller impact, shows the adaptability of the framework when using ensemble methods.

Conclusion

Cell-type annotation remains a crucial yet complex step in scRNA-seq analysis, often requiring high levels of computational precision to achieve accurate and robust labeling under varying data conditions. In this work, we introduce a novel ensemble machine learning-based framework that leverages pre-trained models and genetic optimization to annotate scRNA-seq data. Unlike traditional approaches that depend on a single learning algorithm or predefined marker genes, the proposed approach harnesses multiple learners in an ensemble to capture diverse data characteristics and employs a genetic algorithm to identify the most discriminative features. This dual strategy enables the model to address key challenges in scRNA-seq analysis, including high dimensionality, data scarcity, and biological heterogeneity, without requiring explicit batch correction, making it adaptable across different biological environments and data constraints. By incorporating a genetic algorithm for feature selection and a clustering step for unsupervised learning, the proposed model achieves high annotation accuracy and robustness in data-limited scenarios without specific batch effect correction.

In benchmark comparisons and simulation studies, the proposed method consistently outperformed other state-of-the-art annotation frameworks, including scMRA, ItClust, Seurat, and MARS. The resilience of the proposed framework to varying data reductions tested across mild, moderate, and severe data scarcity settings demonstrates its robustness, with only minor accuracy reductions even as the available reference data decreased by $70\%$. This stability stems from the ensemble model's capacity to integrate information across diverse model architectures, while genetic optimization ensures the selection of the most relevant features, ultimately enabling accurate annotation with minimal reference data.

The proposed framework also demonstrates significant adaptability in multi-domain annotation scenarios, where it effectively handles clusters of varying sizes and structures without being negatively impacted by the presence of unmatched cell types in the target data. Through rigorous ablation studies, we found that each component genetic optimization, ensemble learning, dimensionality reduction, and choice of clustering algorithm contributed substantially to the model’s overall performance, reinforcing the value of each in managing the complexity of scRNA-seq data.

Compared to existing methods such as scMRA, MARS, ItClust, Seurat, and scmap, our framework consistently achieves higher accuracy and robustness under varied and challenging conditions. Its adaptability to multiple reference domains, resistance to overfitting in small-data scenarios, and capacity to generalize across platforms highlight its superiority. We believe that this framework’s ability to maintain performance under both data-rich and data-limited conditions, along with its adaptability to partial and diverse reference datasets, highlights its potential as a practical and powerful tool for scRNA-seq annotation. Future work may explore further optimizations for the partial setting and investigate its applications across a broader range of biological and clinical datasets.

Author contributions

Qiu.Y. conceived the project. Elnahas O., Qiu, Y. conducted the formal analysis. Elnahas O., Ead, W., and Qiu, Y. developed the methodology. Qiu. Y., and Lu, J., supervised the project. All the authors wrote and approved the manuscript.

Funding

The work was supported in part by the National Natural Science Foundation of China under Grant numbers 62372303, 62002234, U21 A20455, 12326619; in part by the Natural Science Foundation of Guangdong Province of China under Grant 2024 A1515011913, 2023 A1515011691; in part by the Guangdong Basic and Applied Basic Research Foundation under Grant 2024 A1515010113; in part by the Shenzhen Basis Research Project of China under Grant JCYJ20210324094006017; in part by the Shenzhen Science and Technology Program under Grant RCYX20231211090244048.

Availability of data and materials

We selected four sequencing datasets from human pancreatic tissue for experiments: Baron_human (inDrop sequencing) (Klein and Macosko, 2017) https://pubmed.ncbi.nlm.nih.gov/28721415/, Enge (Smart-seq2) (Picelli et al., 2013) https://www.nature.com/articles/nmeth.2639, Muraro (CEL-seq2) (Hashimshony et al., 2012) https://www.sciencedirect.com/science/article/pii/S2211124712002288, and Segerstolpe (Smart-seq2) https://bioconductor.org/books/3.20/OSCA.workflows/segerstolpe-human-pancreas-smart-seq2.html. The source code, usage instructions, and example datasets will be made publicly available upon acceptance of this manuscript, in accordance with journal guidelines for reproducibility.

Declarations

Ethics approval and consent to participate

Not applicable

Consent for publication

Not applicable

Competing interests

The authors have no Conflict of interest to declare.