Benchmarking RNA velocity methods across 17 independent studies

Summary

RNA velocity techniques offer great potential for unveiling trajectories of cell state transitions in different biological contexts. While diverse computational methods have been developed, there are no evidence-based guidelines for best-practice in RNA velocity inference. Here, we conduct a benchmark study of 15 existing RNA velocity methods across 17 independent datasets, incorporating multiple validation strategies. We evaluate performance across three key dimensions: accuracy, stability, and usability. Our data showed no single method exhibited superior performance in all the assessments, and unexpected underperformance was observed in certain cases. Based on these findings, we establish scenario-based suggestions of best-practice to assist users in selecting the method best suited to their data and analytical needs.

Graphical abstract

Highlights

• •Multi-dimensional evaluation covering accuracy, stability, and computational usability
• •Widespread discrepancies in velocity fields inferred by different methods
• •Best-practice for method selection in three specific scenarios

Motivation

RNA velocity has emerged as a pivotal technique for reconstructing cell differentiation trajectories by leveraging the temporal kinetics of unspliced and spliced transcripts. However, the rapid expansion of the algorithmic ecosystem has introduced uncertainty and challenges in selecting best-practice methods for specific analyses. Here, we systematically evaluated 15 RNA velocity methods, providing users with actionable, scenario-based suggestions to guide optimal tool selection.

In a benchmark study utilizing both real and simulated single-cell RNA sequencing datasets, Luo et al. systematically evaluate 15 RNA velocity methods across three dimensions and propose scenario-based recommendations of best-practice.

Introduction

Single-cell RNA sequencing (scRNA-seq) has enabled a high-resolution snapshot of the cellular status and transitions, facilitating the development of computational approaches unveiling the transcriptional dynamics in diverse biological contexts.^1,2,3 RNA velocity has become a powerful technique in investigating cell fate determination by offering comprehensive views of the trajectories of cells during state transitions.

RNA velocity recovers information about cellular dynamics by measuring the abundance of unspliced and spliced mRNA at single-cell level,^4,5 which can be explicitly quantified using various established tools, such as velocyto,⁴ STARsolo,⁶ and dropEst.⁷ RNA velocity inference generally comprises three stages: preprocessing, velocity estimation, and postprocessing (Figure 1A). In the preprocessing stage, spliced and unspliced mRNA abundance matrices are initially filtered to keep only highly variable genes, which are subsequently normalized, projected to a lower dimension and smoothed. The velocity estimation steps vary substantially among different methods. Many methods infer RNA velocity based on the steady-state assumption, as implemented in velocyto.⁴ The other methods infer RNA velocity by regressing spliced and unspliced mRNA abundance to the trajectories defined by optimized dynamic models, as seen in scVelo.⁵ The steady-state model, such as velocyto, assumes: (1) gene transcription and degradation rates are constant, and splicing rates are uniform for all genes; (2) a subset of cells reside in steady-state. Both assumptions are circumscribed to large, homogeneous cell populations. Then, the method such as scVelo offers a dynamical model based on maximum likelihood estimation that relaxes the steady-state assumption by introducing cell-specific latency time for each gene, and the kinetic parameters are jointly estimated to enhance the flexibility and accuracy. In the postprocessing stage, gene-wise RNA velocity vectors are projected into a low-dimensional space using Uniform Manifold Approximation and Projection (UMAP), t-distributed stochastic neighborhood embedding (t-SNE), or principal component analysis (PCA), to ultimately demonstrate the trajectory map.

Figure 1.

An overview of RNA velocity methods

(A) Summary of three-step RNA velocity analysis.

(B) Timeline of RNA velocity methods.

(C) Schematic plot of the benchmarking workflow for 15 RNA velocity methods in the 17 scRNA-seq datasets.

Up-to-date, more than 20 RNA velocity methods are published, which explicitly adopted either of the following methodologies (Figure 1B). The first is to improve the accuracy of inference by incorporating additional biological information. For examples, protaccel⁸ integrates protein translation processes into the model; Chromatin Velocity⁹ and MultiVelo¹⁰ incorporate epigenomic features like chromatin accessibility; PhyloVelo¹¹ combines genealogical information with single-cell transcriptome data for velocity field reconstruction; Dynamo¹² integrates time-resolved metabolic labeling data, overcoming traditional methods’ reliance on steady-state assumptions, and establishes a computable quantitative theoretical framework to reconstruct continuous transcriptomic vector fields. The second methodology focuses on improving the algorithms to better control the noises and specifically recapitulate the dynamics of the transcription process. These include velvet¹³ improves RNA velocity inference through a neighborhood constraint mechanism; VeloVAE,¹⁴ veloVI,¹⁵ and Pyro-Velocity¹⁶ that adopt Bayesian inference framework to cope with uncertainty; then, veloAE¹⁷ smooths low-dimensional velocities with an autoencoder; DeepVelo¹⁸ applies a variational autoencoder to predict continuous changes in intracellular gene expression; UniTVelo¹⁹ develops a uniform regularization of velocity estimation to control for biases in timescale and directionality; DeepVelo²⁰ utilizes graph convolutional networks to infer gene-specific and cell-specific kinetic parameters; LatentVelo²¹ constructs a neural ODE model in the presence of a multi-branching temporal process; cellDancer²² uses deep neural network to predict cell-specific kinetic parameters for each gene, thereby enabling accurate RNA velocity estimation across different cell types; finally, cell2fate²³ decomposes the velocity matrix into resolvable modules representing the biological factors underlying transcriptional dynamics.

The rapid advancement of RNA velocity methods has provided researchers with diverse analytical tools, yet it also leads to the uncertainty and difficulty in choosing the best-practice for specific analyses. Previous studies^24,25,26 of RNA velocity methods have largely focused on single performance dimensions or specific application scenarios, resulting in limitations in the assessment criteria. Here, we conducted a systematic evaluation of 15 RNA velocity methods, assessing the performance in full-scale including inference accuracy, algorithmic stability, and computational resource usage. Our benchmark study utilized 17 independent datasets, in which most methods were tested for the first time. By rigorous comparison of the performance across multiple datasets and conditions, our data offer insights for the current RNA velocity methods and inform the best-practice for future studies.

Results

Design of the benchmark study

To fully represent the state-of-the-art in RNA velocity methods, our benchmark study included 15 algorithms that are publicly available. These methods are categorized into three groups: (1) ODE-based methods including velocyto,⁴ scVelo-stochastic,⁵ scVelo-dynamic,⁵ MultiVelo,¹⁰ and CellRank²⁷; (2) machine learning-based methods such as UniTVelo,¹⁹ Dynamo-stochastic,¹² Pyro-Velocity,¹⁶ and cell2fate²³; and (3) deep learning-based methods like veloAE,¹⁷ veloVI,¹⁵ veloVAE,¹⁴ LatentVelo,²¹ cellDancer,²² and DeepVelo²⁰ (Figure 1C).

As for the datasets, we collected 17 published single-cell RNA sequencing (scRNA-seq) datasets, covering a diverse biological contexts and sequencing technologies (Table S1, methods). Particularly, certain datasets (dataset 1–5 and dataset 10) were extensively utilized for evaluating the performance of RNA velocity methods, while other datasets (dataset 9, dataset 12, and dataset 17) were tested only once (Figure S1A). For performance evaluation, we used four metrics. Cross-boundary direction correctness (CBDir),¹⁷ in-cluster coherence (ICCoh),¹⁷ and velocity consistency⁵ measure the correctness and coherence of inference at cell level by each method. Then, method agreement A1 and A2²⁵ measure the consistency of inference among different methods. We systematically benchmarked the performance of 15 RNA velocity methods in all 17 datasets. In addition, we conducted cellular downsampling experiments on four representative datasets, selected varying numbers of highly variable genes during preprocessing, and generated simulated datasets covering three distinct developmental topologies to evaluate the stability of these methods. Lastly, we conducted a comparative analysis of the computational resource consumption of each method in practical applications, including time and memory requirements (Figure 1C).

The performance of RNA velocity inference varies substantially among different methods

We compare the performance of 15 methods on velocity estimation in 17 independent datasets. The implementation of each algorithm followed strictly the description of the original study (“methods”). We examined the results of each method measured on different datasets. As a result, the test performance of the RNA velocity methods showed substantial variation in all aspects.

For CBDir, with an average of 0.1, the test results suggested that for most of the RNA velocity methods, there is still a large room for improvement. veloVI demonstrated the highest overall performance (CBDir = 0.23), followed by Pyro-Velocity (CBDir = 0.17) (Figure 2A). Nevertheless, we noticed that most of the inferred transitions by veloVAE were reversed, as evidenced by negative CBDir values (Figures 2A and S2A). Besides, the accuracy for RNA velocity prediction tended to decline with the complexity. In the human bone marrow cells dataset (i.e., dataset4) with multiple transcriptionally enhanced cell trajectories, the average CBDir was minimized to −0.193 (Figure S1B). In the mature-state peripheral blood mononuclear cell dataset (i.e., dataset11), most methods yielded a number of erroneous directions, such as the transition from CD8+Cytotoxic T cells to CD4+/CD45RA+/CD25-Naive T cells, which is contrary to the known biology²⁸ (Figures S1B and S2B). In addition, some of the methods yielded higher CBDir values in datasets tested in the original studies (“tested on”) than in datasets tested for the first time (“unseen”) (Figure S1C).

Figure 2.

The performance of 15 RNA velocity methods across 17 scRNA-seq datasets

(A) Cross-boundary Direction Correctness (CBDir) scores. Each point represents the directional correctness score for a source-to-target cluster transition (e.g., cluster A → cluster B) within a dataset.

(B) Intra-cluster coherence (ICCoh) scores.

(C) Velocity consistency scores. Individual points represent the respective metric’s mean score within a single dataset (for B and C). Yellow symbols denote the grand mean of each method’s scores for the corresponding metric, calculated across all datasets.

On the other hand, most methods achieved relatively high ICCoh values (≥0.7), especially latentvelo (ICCoh = 0.99), UniTVelo (ICCoh = 0.96), and MultiVelo (ICCoh = 0.96). Likewise, these methods also demonstrated high velocity consistency (≥0.6). Both metrics indicated that the inferred velocity fields were quite smooth among neighboring cells (Figures 2B and 2C). Nevertheless, veloVAE underperformed in both intra-cluster coherence and velocity consistency compared to other methods (Figures 2B, 2C, S1D, and S1E). Moreover, for all the methods, the mean ICCoh showed no biases for different datasets (Figure S1C). Whereas for velocity consistency, most methods performed significantly better in the datasets tested in the original studies (“tested on”) (Figure S1C). To summarize, our data suggested that existing methods infer consistent velocities at single-cell level, but the results demonstrate substantial inadequacy in resolving the transitions among different cell states hence limited ability to distinguish truly relevant transcriptional dynamics.

Discrepancies in the velocity fields inferred by different methods

Then, we evaluated the consistency in the velocity fields inferred by 15 different methods using two metrics of agreement, A1 and A2.²⁵ Our data showed that the agreement scores (A1) of the 15 methods were generally low, with most methods’ A1 below 0.4, indicating substantial discrepancies in RNA velocity estimations among these methods. In particular, latentvelo and cell2fate manifested quite low A1 to all the other methods and between each other (Figures 3A and S3). We further investigated method agreement measured by pairwise A1 within different cell types in four widely used benchmark datasets (dataset1-dataset4). In the pancreas endocrinogenesis dataset (dataset1), these methods (velocyto, scVelo-dyn, veloVAE, DeepVelo, and CellRank) showed robustness in method agreement (A1) for early cell types, but decreased by approximately 56% in terminally differentiated cell types (Figures S4A–S4C). In the human bone marrow dataset (dataset4), only veloVI and DeepVelo maintained stability across all cell types (Figure S4D), while consensus levels were generally low in the other two datasets (Figures S4E and S4F). Next, we assessed the systematic differences (A2) among these methods. As a result, four methods (Pyro-Velocity, scVelo-sto, velocyto, and Dynamo-sto) performed relatively better (A2 > 0.5) than others, and latentvelo, again, exhibited lowest consistency (A2 < 0.3) (Figures 3B and S4G). In the four widely used benchmark datasets, scVelo-sto and Pyro-Velocity demonstrated higher consistency (A2 > 0.7), while cell2fate and latentvelo were the least consistent (A2 < 0.35) (Figure S4G).

Figure 3.

Comparison of cell velocity field agreement for RNA velocity methods

(A) Pairwise comparison of method agreement by mean-A1 scores.

(B) Method agreement by mean-A2 scores of 15 RNA velocity methods across all datasets. Individual data points (circles) represent the mean-A2 score per dataset for each method. Grand means (±SD) across datasets are indicated by yellow symbols.

In summary, our data suggested that discrepancies in RNA velocity estimation are quite common among the tested methods, especially in certain methods and cell types. Consequently, to mitigate potential bias introduced by reliance on a single method, we recommend comparing RNA velocity results across multiple methods and focusing on their consistency, rather than depending on a single approach.

Stability of RNA velocity inference performance is subject to sampling biases and complexity

Current methods infer RNA velocity based on measures of RNA abundance from a large number of genes and cells hence the results are subject to all technical and sampling biases. Here, we assessed the stability of existing methods by downsampling cells and controlling the number of highly variable genes (HVGs) for RNA velocity inference across four widely used datasets.

We evaluated the stability of the performance of each method in the downsample and different sets of HVGs using the same metrics above. Results of downsamples showed that the method stability is mostly affected by the sampling rates. For most of the methods, CBDir varied drastically at different sampling rates and the tendencies of the change were largely unpredictable (Figures 4A, S5A, and S5C). Specifically, our data suggested that Pyro-Velocity and cell2fate were the most unstable for CBDir (range [−0.11, 0.403]). On the other hand, ICCoh were relatively stable at different downsample rates (above 0.7), except for some outliers (velocyto and cellDancer; Figures 4B, S5B, and S5D). As for the velocity consistency, the performance of the methods tended to polarize in the downsampling. Methods such as UniTVelo and latentvelo remained stable and performed well at different sampling rates, while other methods tended to be unstable and underperformed (Figures 4C and S5E). Regarding the inter-method agreement A1, despite the different sampling rates, the overall agreement of inferred RNA velocity by these methods remains unchanged. Notably, some methods, including DeepVelo, scVelo-sto, veloVI, and velocyto, showed highly stable agreement (Figure 4D). As for the agreement A2, most of the methods exhibited stable inter-method agreement, dropping only slightly in the downsampling data. Nevertheless, methods achieved high agreement in full data also tend to agree in the downsamples, such as scVelo-sto, Pyro-Velocity, and Dynamo-sto (A2 > 0.5) (Figures 4E and S5F). Test results based on different numbers of HVGs indicated that CBDir is highly sensitive to the choice of HVG. For latentvelo and UniTVelo, optimal performance occurred at 1,000 highly variable genes (Figure S6A). ICCoh and velocity consistency demonstrated notable robustness to feature selection across most methods, with an exception for veloVAE (Figures S6B and S6C). Regarding method agreement A1, the overall consistency of inferred RNA velocity across different HVGs remained unchanged (Figure S6D). Additionally, the A2 scores for scVelo-sto, Dynamo-sto, and Pyro-Velocity increased with the size of highly variable genes included (Figure S6E).

Figure 4.

Comparing the performance of RNA velocity methods on downsampled datasets

(A) The average CBDir score, (B) ICCoh score, and (C) velocity consistency score of four benchmark datasets were analyzed by 15 RNA velocity methods with different sampling rates.

(D) The average of the method agreement A1 score of the four benchmark datasets processed by the 15 RNA velocity methods with different sampling rates.

(E) And the average of method agreement A2 score. “1.0” represents the ground truth of the original dataset.

To assess the stability of performance of RNA velocity methods in resolving complex transcriptional dynamics, we subsequently evaluated these methods using simulated datasets of different trajectorial features (“methods”). We employed dyngen²⁹ to generate three simulated datasets, each comprising 1,000 cells. In three simulated datasets of 1000 cells, each featuring a unique trajectorial type of bifurcation (185 genes), cycle (171 genes), and linear (171 genes), veloVI, DeepVelo, and Dynamo-sto achieved high CBDir scores in resolving bifurcating and cycling trajectories (Figure 5A). Latentvelo and cell2fate exhibited the highest ICCoh and velocity consistency, indicating that these methods generate smooth and stable velocity vector fields within local neighborhoods (Figures 5B and 5C). Nevertheless, the pairwise A1 revealed that agreement of results only exists between certain methods such as velocyto and scVelo-sto, CellRank and cellDancer, DeepVelo and veloVI (Figures 5D and S6F). Then, DeepVelo, velocyto, veloVI, and Dynamo-sto achieved high agreement scores (A2), suggesting they may capture some consensus features of transcription dynamics (Figure 5E).

Figure 5.

Benchmarking RNA velocity methods on simulated datasets

(A) CBDir score, (B) ICCoh score, and (C) velocity consistency score of three simulated datasets.

(D) The average of the method agreement A1 score of three simulated datasets.

(E) Method agreement A2 score of three simulated datasets.

Overall performance and suggested best-practice for RNA velocity inference

In summary, we evaluated each method across three core aspects: (1) accuracy of RNA velocity estimation, assessed across all methods and real datasets; (2) stability of prediction under different conditions (cell downsampling, the number of highly variable genes included, and simulated trajectories); and (3) computational efficiency. Based on the benchmark results across all methods, we provide a comprehensive summary of performance of RNA velocity methods across these three aspects (Figure 6).

Figure 6.

Summary of performance of all methods

(A) Each method’s performance on four benchmark metrics across 17 real datasets (red).

(B) Each method’s performance on four benchmark metrics across downsampled datasets, varying HVG sets, and simulated datasets (blue).

(C) Each method’s average execution time and average memory increment when testing real datasets (green). Among these, 10 methods (MultiVelo, UniTVelo, Pyro-Velocity, cell2fate, veloAE, veloVI, veloVAE, LatentVelo, cellDancer, and DeepVelo) were tested on GPUs, while the remaining 5 methods (velocyto, scVelo-sto, scVelo-dyn, Dynamo-sto, and CellRank) were tested on CPUs. The “overall” metric represents the geometric mean of four metrics: cross-boundary direction correctness (CBDir), intra-cluster coherence (ICCoh), velocity consistency (Vcs), and method agreement A2 (A2). And the longer the bar chart, the darker the color, indicating a higher score for this metric. Similarly, the larger the bubble and the darker the color, the higher the score for this metric.

(D) Scenario-specific suggestions for method users.

In terms of accuracy, veloVI, Pyro-Velocity, and DeepVelo demonstrated the best overall performance, characterized by high and balanced scores across all four metrics (Figure 6A). Specifically, veloVI and Pyro-Velocity led in Cross-Boundary Direction correctness (CBDir), while LatentVelo, UniTVelo, MultiVelo, and Dynamo-sto excelled in In-Cluster Coherence (ICCoh) and Velocity Consistency (Vcs). Furthermore, Pyro-Velocity, Dynamo-sto, scvelo-sto, and velocyto showed distinct advantages in method agreement A2 (A2). Regarding stability, LatentVelo and UniTVelo exhibited robust performance in the downsamples, whereas velocyto showed a significant decline. UniTVelo, LatentVelo, and veloVI demonstrated relatively stable performance across varying numbers of HVGs. In the simulated datasets, DeepVelo, veloVI, and LatentVelo achieved excellent overall performance (Figure 6B). The usability assessment revealed that among GPU-based methods, DeepVelo and veloVI were notable for their faster execution times and lower memory requirements. In contrast, cell2fate and Pyro-Velocity demanded higher memory, while cellDancer and MultiVelo required longer execution times. CPU-based methods generally exhibit lower memory requirements, with velocyto, scVelo-sto, and Dynamo-sto achieving shorter execution times (Figure 6C).

Based on our benchmarking results, we propose a suggestion framework tailored to three specific scenarios to guide users in method selection (Figure 6D). First, for large-scale cell atlas construction (million-cell scale) where scalability and accuracy are paramount, we recommend veloVI, DeepVelo, Dynamo-sto, and scVelo-sto. These methods offer a favorable balance of high accuracy and computational efficiency, featuring low memory footprints and CPU compatibility suitable for massive datasets. Second, for datasets with compromised quality (characterized by low sequencing depth, high noise, or sparsity), UniTVelo, LatentVelo, veloVI, and Pyro-Velocity are preferred. These approaches demonstrate superior robustness against downsampling while maintaining reliable inference. Third, for dissecting complex lineage dynamics (e.g., non-linear trajectories or multi-branching lineages), we advocate for DeepVelo, veloVI, and LatentVelo, as they exhibit high concordance in real-world datasets and resilience to simulated perturbations.

Discussion

RNA-velocity analysis is gaining traction in better resolving the complex dynamics of cell-state transitions implicated in diverse biological contexts. In this benchmarking study, we tested 15 RNA velocity methods in 17 published datasets, conducting a comprehensive performance evaluation in the dimensions of inference accuracy, algorithm stability, and computational efficiency. Our results suggested that the performance of RNA velocity methods is significantly influenced by differences in cell characteristics. Our data showed that, with increasing complexity of transcriptional dynamics, the inference performance of the methods tended to decline. For example, in a peripheral blood mononuclear cell dataset at a mature state, many methods inferred incorrect directions; in a human bone marrow cell dataset, due to the presence of multiple transcriptionally enhanced cell trajectories, most methods exhibited low cross-boundary direction correctness (CBDir). Additionally, cell lineage and differentiation state also influence the performance of RNA velocity methods. For example, in the pancreatic endocrinogenesis dataset, early progenitor cells with gradual transcriptional changes were accurately resolved by most of the methods, whereas consistency among the methods decreased drastically in the same population of cells but at highly differentiated levels, highlighting the challenges posed by the complexity of differentiation lineages for velocity inference. Based on our benchmarking data, we propose the following scenario-specific recommendations: (i) Extra-large dataset. For datasets of million-scale, methods of high accuracy and computational efficiency, such as veloVI, DeepVelo, Dynamo-sto, and scVelo-sto are recommended due to high compatibility and low memory usage. (ii) Low-quality data. For sparse, noisy, or low-depth data, UniTVelo, LatentVelo, veloVI, and Pyro-Velocity are optimal, showing high accuracy and strong resistance to biases caused by under-sampling. (iii) Complex dynamics. For resolving non-linear or multi-branching lineages, DeepVelo, veloVI, and LatentVelo are preferred, distinguishing themselves through high accuracy in real datasets and stability under simulated perturbations.

Nevertheless, the current benchmarking analysis has some limitations. On the one hand, due to the fast advances in the field, some new methods were not included in our benchmarking study. On the other hand, while our metrics provide multi-dimensional insights of the performance of the inference, there are some key trade-offs. First, CBDir is used to measure the accuracy of cell directional transitions. However, CBDir relies on predefined ground truth, which for most single-cell datasets is constructed based on curated cell trajectories or existing biological knowledge. Such ground truth definitions may be affected by incomplete prior information or human assumptions. Moreover, ground truth definitions vary across datasets, reflecting differences in biological context, experimental design, and data availability. This heterogeneity may emphasize different aspects of RNA velocity inference and consequently influence benchmarking outcomes. Therefore, the main conclusions of this study are not derived from any single dataset or ground-truth definition, but are based on consistent trends observed across multiple datasets with diverse types of ground truth. Second, we used ICCoh and velocity consistency to measure the coherence of inferences, for which most methods achieving high scores (ICCoh ≥ 0.7, and velocity consistency ≥ 0.6). However, such high scores can result from over-smoothing of the trajectory rather than real biological continuity and introduce bias against bifurcative and multifurcative trajectories. Then, the method consistency metrics (A1 and A2) assesses the consistency of inferred velocity fields across different methods. Our data showed that discrepancy is very common in the RNA velocity inferred by different methods, particularly for certain cell types. However, such discrepancy can reflect different biology contexts embedded in the different algorithms. For example, LatentVelo and cell2fate exhibit significant discrepancy, which is attributed to the unique model architecture.

Furthermore, we performed simple downsampling (ratio) of the raw data to simulate sampling biases and insufficient observations, but the actual biases in real data are more complex. Nevertheless, our data showed that sampling biases, such as insufficient sampling of rare cell types or intermediate transitional cells, lead to erroneous or incomplete trajectory reconstruction. Data sparsity is a fundamental limitation of traditional scRNA-seq technology. Integrating metabolic labeling single-cell sequencing technology^30,31,32 or other multi-omics data (e.g., chromatin accessibility or protein translation) will improve the quantification of spliced and unspliced mRNA. Again, integration of other data in RNA velocity study is constrained by low throughput, complex experimental design, and high costs.

In summary, our benchmarking results informed future directions to improve RNA velocity methods. First, large, well-balanced, multi-center datasets by combining lineage tracing or metabolite labeling techniques will mitigate technical biases and provide foundational benchmarking data for validation of the methods. Second, parallel benchmarking of the methods is needed to suggest best-practice in RNA velocity inference. Third, better control of the imbalance in the cell populations will improve the inference performance. Finally, to mitigate biases introduced by reliance on any single RNA velocity method, we recommend adopting a multi-method comparison strategy that emphasizes cross-method consistency in downstream biological interpretations, thereby enabling more robust velocity inference.

Limitations of the study

The evaluation metrics entail inherent trade-offs. First, the accuracy assessment relies on predefined ground truth, which may be subject to incomplete prior information or annotation bias. Second, high coherence scores may reflect trajectory over-smoothing rather than biological fidelity. Finally, observed inconsistencies between methods (e.g., LatentVelo vs. cell2fate) likely stem from distinct model architectures and underlying assumptions rather than inference errors.

Resource availability

Lead contact

Further information and requests for resources should be directed to and will be fulfilled by the lead contact, Qiyuan Li (qiyuan.li@xmu.edu.cn).

Materials availability

This study did not generate new unique reagents.

Data and code availability

• •The datasets that are analyzed in the study are publicly available. The accession numbers are listed in the key resources table.
• •All original code used for the comparative analysis has been deposited on GitHub (https://github.com/luoy-cloud/veloBench/tree/main). The DOI is listed in the key resources table.
• •Any additional information that may be required to reanalyze the data reported in this paper is available from the lead contact upon request.

Acknowledgments

This work was supported by National Key Research and Development Program of China (2022YFC2704801 and 2022YFC2704800 to Q.Y.L.) and the National Natural Science Foundation of China (no. 82272944 to Q.Y.L.).

Author contributions

The research design and manuscript writing were completed by Y.L., J.R., and Q.L.; the code development and benchmark experiment implementation were carried out by Y.L. and Q.Y.; data and methodology collection were performed by Y.L., J.R., and Z.Y.; manuscript revision and review were conducted by Y.L., J.R., Q.Y., and Q.L., with assistance from Y.Z. and Q.Q. All authors read and approved the final manuscript.

Declaration of interests

The authors declare no competing interests.

STAR★Methods

Key resources table

REAGENT or RESOURCE	SOURCE	IDENTIFIER
Deposited data
Pancreatic endocrinogenesis	Bastidas et al.33	GEO: GSE132188
Dentate gyrus	Hochgerner et al.34	GEO: GSE95753
Mouse Erythroid Maturation	Dong et al.35	GEO: GSE87038
Human Bone Marrow	Setty et al.36	INSDC: ERP120467
Intestinal organoid	van den Berg et al.37	GEO: GSE128365
Mouse retina development	Lo Giudice et al.38	GEO: GSE122466
Mouse hindbrain (GABA, Glial)	Vladoiu et al.39	GEO: GSE118068
Mouse organogenesis	Cao et al.40	GEO: GSE119945
Developing human brain	Trevino et al.41	GEO: GSE162170
Human postconception forebrain	La Manno et al.4	SRA: SRP129388
PBMC-68k	Zheng et al.42	SRA: SRP073767
Human HSPC	Li et al.10	GEO: GSE209878
Mouse cortical neuronal	Qiu et al.32	GEO: GSE141851
Mouse hindbrain (Oligo)	Zeisel et al.43	SRA: SRP135960
Mouse bone marrow	Petukhov et al.7	GEO: GSE109989
Embryonic mouse brain (5k)	10× Genomics	https://www.10xgenomics.com/resources/datasets/fresh-embryonice-18-mouse-brain-5-k-1-standard-1-0-0
Mouse hematopoiesis	Nestorowa et al.44	GEO: GSE81682
Software and algorithms
velocyto	La Manno et al.4	http://velocyto.org
scVelo-stochastic	Bergen et al.5	https://github.com/theislab/scvelo
scVelo-dynamical	Bergen et al.5	https://github.com/theislab/scvelo
MultiVelo	Li et al.10	https://github.com/welch-lab/MultiVelo
veloAE	Qiao et al.17	https://github.com/qiaochen/VeloRep
CellRank	Lange et al.27	https://github.com/theislab/cellrank
veloVI	Gayoso et al.15	https://github.com/YosefLab/velovi
veloVAE	Gu et al.14	https://github.com/welch-lab/VeloVAE
UniTVelo	Gao et al.19	https://github.com/StatBiomed/UniTVelo
Pyro-Velocity	Qin et al.16	https://github.com/pinellolab/pyrovelocity
LatentVelo	Farrell et al.21	https://github.com/Spencerfar/LatentVelo
Dynamo	Qiu et al.12	https://github.com/aristoteleo/dynamo-release
cell2fate	Aivazidis et al.23	https://github.com/BayraktarLab/cell2fate
cellDancer	Li et al.22	https://github.com/GuangyuWangLab2021/cellDancer
DeepVelo	Cui et al.20	https://github.com/bowang-lab/DeepVelo
Other
Data analysis scripts	This paper	https://doi.org/10.5281/zenodo.18699599

Method details

Datasets acquisition, processing, and availability

We collected real datasets by searching and selecting 15 existing RNA velocity methods where cells are in dynamic development and differentiation. We incorporated 17 scRNA-seq datasets (Table S1) for evaluation, including datasets used in at least 3 RNA velocity methods, such as pancreatic endocrinogenesis (Dataset1), mouse dentate gyrus (Dataset2), erythrocyte maturation (Dataset3), intestinal organoids (Dataset5), human first post-conception 10th week forebrain (Dataset10), mouse cortical neurons (Dataset13), mouse retinal development (Dataset6), and oligodendrocyte differentiation in the mouse hindbrain (Dataset14). To assess the ability to capture complex transcriptional dynamics, we analyzed a mouse hindbrain dataset with diverse lineage-specific cell populations (Dataset7), a human bone marrow dataset displaying transcriptional enhancements in cell trajectories (Dataset4), a mouse chondrocyte cell development dataset illustrating multiple differentiation pathways (Dataset8), and a human peripheral blood cell dataset representing maturation state (Dataset11). Furthermore, we evaluated the model’s performance on a mouse bone marrow dataset with notably lower UMI counts (Dataset15) to assess its performance with low coverage data. Additionally, we included datasets on developmental and hematopoietic cell dynamics processes: the developing human cortex dataset (Dataset9), the mouse embryonic brain dataset (Dataset16), the human hematopoietic stem cell differentiation dataset (Dataset12), and the mouse hematopoietic stem cell differentiation dataset (Dataset17) to further evaluate the efficacy of different RNA velocity methods.

All scRNA-seq data we used in this study were publicly available. The original accession numbers and the number of cells and genes in the datasets are described in Table S1. For instance, the mouse dentate gyrus neurogenesis data, we followed the gene and cell filtering methods by Bergen et al. in the scVelo study⁵ and selected 2930 cells with 13913 genes. In the preprocessing phase of RNA velocity analysis, all datasets were log- normalized for the first 3000 highly variable genes, and first and second moments were computed with 30 principal components and 30 nearest neighbors. It should be noted that the known differentiation orders observed in some or all cell clusters across datasets in this study are summarized in Table S2 and applied to the cross-boundary direction correctness (CBDir) metric. The download sources for all datasets and the ground-truth orders of cell development within the datasets are further elaborated as follows:

The pancreatic endocrinogenesis data (Dataset1) is integrated in scVelo via scv.datasets.pancreas() or the original work³³ under GEO accession number GSE132188. The dataset contains multiple lineages of cellular development. We evaluated velocity on these transition directions: (Ngn3 high EP→Pre-endocrine), (Pre-endocrin→Alpha), (Pre-endocrine →Beta), (Pre-endocrine → Delta), (Pre-endocrine → Epsilon).

The dentate gyrus neurogenesis data (Dataset2) is integrated in scVelo via scv.datasets.dentategyrus() or the original work³⁴ under GEO accession number GSE95753. The known cell differentiation direction in this dataset is from OPC cells to OL cells. We evaluated velocity on these transition directions: (OPC→OL), (nIPC→Neuroblast), (Neuroblast→ Granule immature),(Granule immature→ Granule mature), (Radial Glia-like→ Astrocytes).

The erythroid maturation data (Dataset3) is integrated in scVelo via scv.datasets.gastrulation_erythroid() or the original work³⁵ under GEO accession number GSE87038. The dataset annotated red lineage cells at developmental stages 1, 2, and 3, corresponding to the order of ground-truth transition. We evaluated velocity on these transition directions: (Blood progenitors 1 → Blood progenitors 2), (Blood progenitors 2 → Erythroid1), (Erythroid1 →Erythroid2), (Erythroid2 → Erythroid3).

The human bone marrow data(Dataset4) is integrated in scVelo via scv.datasets.bonemarrow() or the original work³⁶ through the Human Cell Atlas data portal under INSDC project accession number ERP120467. The transitions tested with CBDir are (HSC_1 → Ery_1), (HSC_1 → HSC_2), (Ery_1→ Ery_2).

The Intestinal organoid data(Dataset5) can be accessed at https://www.dropbox.com/s/ 25enev458c8egn7/organoid.h5ad? dl = 1 or the original work³⁷ under GEO accession number GSE128365. The scEU-seq dataset groups all cells into two monocle branches. The transitions tested with CBDir are (Stem cells → TA cells), (Stem cells → Goblet cells).

The mouse retina development data (Dataset6) can be accessed at http://pklab.med.harvard.edu/peterk/review2020/examples/retina/ or the original work³⁸ under GEO accession number GSE122466. Transcriptome analysis revealed cellular lineage progression encompassing a proliferative phase followed by differentiation into three terminal states. We evaluated velocity on these transition directions: (Neuroblast → PR), (Neuroblast → AC/HC), (Neuroblast → RGC).

The mouse hindbrain data(Dataset7) can be accessed at https://doi.org/10.6084/m9.figshare.24716592 or the original work³⁹ under GEO accession number GSE118068. We employed the putative developmental order of six cell types in early mouse hindbrain development provided by DeepVelo.²⁰ We evaluated velocity on these transition directions: (Neural stem cells → Proliferating VZ progenitors), (Proliferating VZ progenitors → VZ progenitors), (VZ progenitors → Differentiating GABA interneurons), (VZ progenitors → Gliogenic progenitors), (Differentiating GABA interneurons → GABA interneurons).

The mouse organogenesis data(Dataset8) can be accessed at https://doi.org/10.6084/m9.figshare.24716592 or the original work⁴⁰ under GEO accession number GSE119945.We employed the cell differentiation orders described by DeepVelo,²⁰ which are: (Early mesenchyme → Chondrocyte progenitors), (Early mesenchyme → Osteoblasts), (Early mesenchyme →Intermediate Mesoderm), (Early mesenchyme → Myocytes), (Early mesenchyme →Connective tissue progenitors), (Early mesenchyme → Limb mesenchyme), (Early mesenchyme →Jaw and tooth progenitors), (Chondroctye progenitors → Chondrocytes & osteoblasts).

The development of human cerebral cortex data(Dataset9) can be accessed at https://figshare.com/articles/dataset/Developing_ Human_Cortex_RNA_Data/22575376 or the original work⁴¹ under GEO accession number GSE162170. We evaluated velocity on these transition directions: (RG/Astro → Cyc.), (Cyc. → nIPC/ExN), (nIPC/ExN → ExUp).

The human forebrain data(Dataset10) can be accessed at https://drive.google.com/file/d/1Bi5Ss7FtyDNV_gZOeoBPittFDf_EHsPw/view or the original work⁴ under Sequence Read Archive accession code SRP129388. We adopted the putative developmental orders provided by velocyto,⁴which are (Radial Glia → Neuroblast), (Neuroblast → Immature Neuron), (Immature Neuron → Neuron).

The human peripheral blood mononuclear cells data (Dataset11) is integrated in scVelo via scv.datasets.pbmc68k() or the original work⁴² under Sequence Read Archive accession code SRP073767. The transitions tested with CBDir are (CD4+/CD45RA+/CD25- Naive T → CD4+/CD45RO + Memory), (CD4+/CD45RA+/CD25- Naive T → CD4+/CD25 T Reg), (CD14⁺ Monocyte → Dendritic).

The human hematopoietic Stem/Progenitor Cell data(Dataset12) can be accessed at https://figshare.com/articles/dataset/Human_HSPC_RNA_Data/22575358 or the original work¹⁰ under GEO accession number GSE209878. The transitions tested with CBDir are (HSC →MPP), (MPP → LMPP), (MEP → Erythrocyte), (GMP → Granulocyte).

The mouse cortical neuron data (Dataset13) is integrated in Dynamo via dyn.sample_data.scNT_seq_neuron_splicing() or the original work³² under GEO accession number GSE141851. The time labels in this scNT-seq dataset record the duration of stimulation for each cell, thereby determining the transition sequence of the cells. The transitions tested with CBDir are the time points (0→15), (15→30), (30→60), (60→120).

The mouse Oligodendrocyte differentiation (Dataset14) data can be accessed at http://pklab.med.harvard.edu/ruslan/velocity/oligos/ or the original work⁴³ under Sequence Read Archive accession code SRP135960. The transitions tested with CBDir are (COPs → NFOLs), (NFOLs → MFOLs).

The mouse bone marrow data (Dataset15) can be accessed at https://cell2fate.cog.sanger.ac.uk/browser.html or the original work⁷ under GEO accession number GSE109989. We adopted the putative developmental orders of neutrophil maturation provided by velocyto,⁴ which are (dividing → progenitors), (progenitors → activating).

The mouse embryonic brain data(Dataset16) can be accessed at https://doi.org/10.6084/10X_multiome_mouse_brain.loom or the original work at https://www.10xgenomics.com/datasets. We evaluated velocity on these transition directions: (RG, Astro, OPC → IPC), (Deeper layer → Upper Layer).

The mouse hematopoiesis data(Dataset17) can be accessed at https://zenodo.org/records/6110279 or the original work⁴⁴ under GEO accession number GSE81682. The transitions tested with CBDir are (LTHSC → MPP), (MPP → LMPP), (MPP → CMP), (CMP → GMP), (CMP → MEP).

Down-sampling

We used 4 benchmark datasets from 17 datasets (that is, Dataset1: pancreatic endocrinogenesis, Dataset2: dentate gyrus nerve, Dataset3: erythroid maturation, Dataset4: human bone marrow) to test the impact of down-sampling of the splicing matrix on each RNA velocity method. Specifically, cells in the spliced matrix of each dataset were sampled stratified by cell cluster (sample rates = 0.4, 0.5, 0.6, 0.7 and 0.8). We downsampled each dataset 5 times at different rates to avoid errors caused by random selection.

Simulation scRNA-seq datasets

We generated synthetic scRNA-seq datasets with dyngen.²⁹ We used the three backbones provided by dyngen, bifurcating, linear, and cycle, and set to simulate 1000 cells to generate three datasets with other parameters set to default values. These datasets included simulated spliced and unspliced mRNA abundances. For the CBDir metric, we used the milestones defined by dyngen as cell transitions.

Brief introduction and parameter setting of methods

We evaluated the performance of 15 RNA velocity methods. The parameters of each method were set as described below for each program.

scVelo and velocyto

We followed the guidelines on the scVelo website: https://scvelo.readthedocs.io/en/stable. There are three approaches for estimating RNA velocity in scVelo, the steady-state model (using steady-state residuals), the stochastic model (using second-order moments), and the kinetic model (using a likelihood-based framework). For scVelo-sto and scVelo-dyn, we ran scVelo.tl.velocity with mode = ‘stochastic’ and mode = ‘dynamical’ with default parameters. The steady state model used in velocyto estimates the velocity. Thus we ran this model with scVelo.tl.velocity mode = ‘deterministic’ using default parameters.

MultiVelo

We followed the guidelines on the GitHub repository of MultiVelo: https://github.com/welch-lab/MultiVelo. Our dataset is conventional single-cell RNA sequencing data with no ATAC-sequencing (scATAC-seq) data, so we set the parameter rna_only = True.

veloAE

We followed the guidelines on the veloAE GitHub repository: https://github.com/qiaochen/VeloAE/blob/main/notebooks. We ran the model following the default settings.

veloVI

We followed the guidelines on the GitHub repository of veloVI: https://github.com/YosefLab/velovi_reproducibility. We ran the model following the default settings.

veloVAE

We followed the guidelines on the veloVAE GitHub repository: https://github.com/welch-lab/VeloVAE. We set to train a continuous veloVAE model.

UniTVelo

We followed the guidelines on the GitHub repository of UniTVelo: https://github.com/StatBiomed/uniTvelo/blob/main/notebooks/. we ran the model in “unified” mode by setting velo.FIT_OPTION = ‘1'.

Pyro-Velocity

We followed the guidelines on the Pyro-Velocity GitHub repository: https://github.com/pinellolab/pyrovelocity. We ran the model following the default settings.

LatentVelo

We followed the guidelines on the GitHub repository of LatentVelo: https://github.com/Spencerfar/LatentVelo/blob/main/paper_notebooks. We ran the model following the default settings.

Dynamo

We followed the guidelines on the Dynamo GitHub repository: https://github.com/aristoteleo/dynamo-tutorials. We ran the model following the default settings. Dynamo incorporates “deterministic,” “stochastic,” and “kinetic” modes. For conventional scRNA-seq data (splicing-based), Dynamo’s default and most commonly used implementation relies on the stochastic mode. The benchmarking dataset we employed consists of conventional scRNA-seq datasets, as we utilize Dynamo’s stochastic mode to simulate the most realistic user scenarios. And we used dyn.tl.cell_velocities with its kernel settings (cosine) to estimate the cell transition matrix.

cell2fate

We followed the guidelines on the GitHub repository of cell2fate: https://github.com/AlexanderAivazidis/cell2fate_notebooks. For datasets with more than 10000 cells, we used the compute_and_plot_total_velocity_scVelo() function to compute velocity.

cellDancer

We followed the guidelines on the cellDancer GitHub repository: https://github.com/GuangyuWangLab2021/cellDancer. When projecting RNA velocity vectors into a low-dimensional embedding, use the cosine kernel function in Dynamo’s dyn.tl.cell_velocities() function to estimate the cell transition matrix, with all other settings set to default.

DeepVelo

We followed the guidelines on the GitHub repository of DeepVelo: https://github.com/bowang-lab/DeepVelo. We ran the model following the default settings.

CellRank

We followed the guidelines on the GitHub repository of CellRank: https://github.com/theislab/cellrank. We ran the model following the default settings.

Computer platform

We ran CPU tests of the 5 RNA velocity methods (velocyto, scVelo-sto, scVelo-dyn, Dynamo-sto, and CellRank) on a computer cluster with Intel(R) Xeon(R) Silver 4210 CPU (2.2 GHz, 14.08 MB L3 cache, 40 CPU cores in total) and 125GB memory. Among these 10 methods, three of them (LatentVelo,Pyro-Velocity, and cell2fate) only support GPU processing, while the other 7 methods (veloAE, veloVI, veloVAE, cellDancer, DeepVelo, MultiVelo, and UniTVelo) support both GPU and CPU processing. The GPU tests for the 10 methods were performed on a computer with Intel(R) Xeon(R) Gold 6230 CPU (2.1 GHz, 55 MB L3 cache, 80 CPU cores in total), 1TB memory, and NVIDIA A800 GPU (80 GB of memory).

Quantification and statistical analysis

Benchmark metrics

We established a common pipeline to comprehensively assess the performance of RNA velocity methods across the 17 datasets. In the pipeline, we used the following four metrics to evaluate each method.

• 1.Cross-boundary direction correctness (CBDir).¹⁷

CBDir is used to assess the consistency of the velocity direction of a cell with the expected direction of development within a brief time frame. Specifically, given the ground truth development direction e.g., A → B, the correctness of the transition from the source cluster to the target cluster is measured by considering the boundary cells that reflect the development of the cell over a short period of time. The boundary cells represent the set as

$$C_{A\to B}=\{\mathrm{c}\in C_A|\exists c^{\prime }\in C_B\cap N\left(\mathrm{c}\right)\}$$ (Equation 1)

where C_A is sets of cells in source cluster A and C_B is sets of cells in target cluster B, N(c) stands for the neighboring cells of specified cell c.

The CBDir value was calculated using the following equation:

$$CBD\text{ir}\left(c\right)=\frac{1}{\left|\left\{c^{\prime }\in C_B\cap N\left(c\right)\right\}\right|}\sum _{c^{\prime }\in C_B\cap N\left(c\right)}\frac{{\mathrm{v}}_{\mathrm{c}}·\left({\mathrm{x}}_{{\mathrm{c}}^{\prime }}-{\mathrm{x}}_c\right)}{\left|{\mathrm{v}}_{\mathrm{c}}\right|·\left|{\mathrm{x}}_{{\mathrm{c}}^{\prime }}-{\mathrm{x}}_c\right|}$$ (Equation 2)

where Vc, $X_{c^{\prime }}$ and X_c are vectors representing computed velocity and positions of cell c and c’ in a low-dimensional space. $X_{{\mathrm{c}}^{\prime }}-X_c$ is the cell displacement in this space during the short time.

• 2.In-cluster coherence (ICCoh).¹⁷

ICCoh is computed by assessing the cosine similarity score of the velocity among cells within the same cluster, which evaluates the consistency of the inferred direction of the same cluster regardless of the correctness of the direction. This metric reflects the smoothing of velocities within the cluster. The formula for computing ICCoh score is

$$ICCoh\left(c\right)=\frac{1}{\left|\left\{c^{\prime }\in C_A\cap N\left(c\right)\right\}\right|}\sum _{c^{\prime }\in C_A\cap N\left(c\right)}\frac{{\mathrm{v}}_{\mathrm{c}}·{\mathrm{v}}_{{\mathrm{c}}^{\prime }}}{\left|{\mathrm{v}}_{\mathrm{c}}\right|·\left|{\mathrm{v}}_{{\mathrm{c}}^{\prime }}\right|}$$ (Equation 3)

• 3.Velocity consistency.⁵

The score is defined as the average cosine similarity of the velocity vectors to their neighbors. For each cell i,

$$C_i=\frac{1}{\left|N_i\right|}\sum _{\mathrm{j}\in Ni}S{\cos }_{\left(V_i,V_j\right)}$$ (Equation 4)

where N_i is the 30 nearest neighbor cells of cell i, S_cos represents the cosine similarity operation. V_i, V_j are the estimated velocities.

• 4.Method agreement A1 and A2.²⁵

This metric is used to assess the consistency of vector predictions for different RNA velocity methods. This metric produces two scores. One is the method agreement score A1, which calculates the cosine similarity of the cell transition vectors created by two different RNA velocity methods. For each pair of methods and each cell, this yields a method agreement score A1.

$$A_1=S_{\cos }\left(V_{\mathrm{i},M_1},V_{\mathrm{i},M_2}\right)$$ (Equation 5)

Where S_cos represents the cosine similarity operation. V_{i, M1} is the state transition vector of method M1 for cell i and V _{i, M2} is the state transition vector of method M2 for cell i.

The other is method agreement score A2, that is to calculate the cosine similarity between the transition vector of cell i in method M1 and the median vector of cell i.

$$A_2=S_{\cos }\left(V_{\mathrm{i},M_1},V_{i,M\text{ed}}\right)$$ (Equation 6)

Where V_{i, Med} is the median vector for the cell i. We calculate the central vector for each cell as the median transition vector across all methods.

Published: March 30, 2026