Systematic comparison of single-cell and single-nucleus RNA-sequencing methods

The scale and capabilities of single-cell RNA sequencing methods have expanded rapidly in recent years, enabling major discoveries and large-scale cell mapping efforts. However, these methods have not been systematically and comprehensively benchmarked. Here, we directly compare seven methods for single-cell and/or single-nucleus profiling – selecting representative methods based on their usage and our expertise and resources to prepare libraries – including 2 low-throughput and 5 high-throughput methods. We tested the methods on three types of samples: cell lines, peripheral blood mononuclear cells, and brain tissue, generating 36 libraries in six separate experiments in a single center. To directly compare the methods and avoid processing differences introduced by the existing pipelines, we developed scumi, a flexible computational pipeline that can be used with any scRNA-seq method. We evaluated the methods for both basic performance, such as the structure and alignment of reads, sensitivity, and extent of multiplets, as well as for their ability to recover known biological information in the samples.

Single-cell RNA sequencing (scRNA-seq) has emerged as a central tool for identifying and characterizing cell types, states, lineages, and circuitry^1-3. The rapid growth in the scale and robustness of lab protocols and associated computational tools has opened the way to substantial scientific discoveries and to an international initiative, the Human Cell Atlas (HCA), to build comprehensive reference maps of all human cells⁴. Methods for scRNA-seq differ in how they tag transcripts for their cell-of-origin and generate libraries for sequencing. Low-throughput, plate-based methods^{5, 6} sort a cell into a well of a multi-well plate. High-throughput, bead-based methods distribute a cell suspension into tiny droplets^7-9 or wells^{10, 11} containing reagents and barcoded beads to produce a single droplet or well with one cell and one bead that is used to mark all the cDNA generated from that cell. Scalable, combinatorial indexing methods reverse transcribe and barcode mRNAs in situ inside each cell or nucleus, without physically isolating single cells^12-14 (Extended Data Fig. 1).

ScRNA-seq remains a rapidly evolving field¹⁵, with continued development of new methods and improvement of existing ones. There is thus a need to provide benchmarking information to help users make informed choices based on each method’s capabilities and limitations, compare new methods to existing ones, identify shared weaknesses as targets for experimental improvement, and allow computational method developers to create new data processing software packages. Prior comparisons of scRNA-seq methods^16-21, though useful, have several shortcomings. Many are outdated, incomplete, inapplicable (e.g., not actually performed with single cells), or insufficiently controlled (e.g., performed using different biological samples for comparisons); others limit their assessment to basic technical factors, but do not assess the key benchmark of the ability to recover meaningful biological information, such as population heterogeneity and structure. In particular, comparisons often focused on cultured cell lines, even though in practice most scRNA-seq studies seek insights from tissues and primary cells.

Here, we systematically and directly compared seven methods (Fig. 1, Extended Data Fig. 1), including two low-throughput plate-based methods (Smart-seq2⁵ and CEL-Seq2⁶) and five high-throughput methods (10x Chromium⁹, Drop-seq⁸, Seq-Well¹⁰, inDrops⁷, and sci-RNA-seq¹²), producing expression profiles from ~92,000 cells overall. We selected representative methods that are more widely used and for which we had the expertise and resources to prepare libraries. We analyzed three sample types – a mixture of human and mouse cell lines, human peripheral blood mononuclear cells (PBMCs), and mouse cortex nuclei, each sample with two replicates – to generate a total of 36 different scRNA-seq libraries. For mouse cortex we tested four single-nucleus RNA-seq methods^{9, 12, 22, 23}. For each sample type, we characterized performance with basic metrics, and for PBMC and cortex libraries, we examined how well methods capture biological information, a critical part of most scRNA-seq studies and one that has not been evaluated in other benchmarking exercises that used relatively homogeneous cell lines^{16, 20}. Our study provides both immediate guidance on each method’s relative performance, and an experimental and computational framework to assess future techniques. For the low-throughput methods, Smart-seq2 and CEL-Seq2 performed similarly, though the latter may be affected more by contaminating reads from other cells. Among the high-throughput methods, 10x Chromium was the top performer.

Figure 1. Study Overview.

(a) samples, (b) scRNA-seq methods, (c) computational pipeline summary. Cell line mixtures tested with all methods. PBMCs tested with all methods except sci-RNA-seq. Cortex nuclei tested with Smart-seq2, 10x Chromium, Drop-seq (aka DroNc-seq for nuclei), and sci-RNA-seq. Additional details can be found in Extended Data Figs. 1 and 2.

RESULTS

A comparison of scRNA-seq methods

We selected seven scRNA-seq methods for comparison and tested each with up to three sample types: a mixture of mouse and human cell lines, human PBMCs, and mouse cortex nuclei (Fig. 1, Extended Data Fig. 1 and 2). We chose to profile a cell line mixture with 50% human HEK293 and 50% mouse NIH3T3 cells (“mixture”) because (1) these cells are a common test^{8, 9, 12, 14} for samples with relatively high amounts of RNA per cell and (2) multiplets, two or more cells being sequenced together and assigned one cell barcode, can be detected when cell barcodes have a substantial fraction of reads from both species. We profiled frozen human PBMCs because (1) they are a heterogeneous mixture of cells, particularly with respect to their amount of RNA per cell, yet they do not require dissociation (a separate technical challenge), and (2) their cell types and associated expression patterns are well-studied. (We do not include data for sci-RNA-seq with PBMCs because we detected very few genes (e.g., <10 genes) per cell in several experiments (data not shown).) We extended our study of single cells to single nuclei, as such samples have distinct properties including lower RNA input amounts. Using four methods that had previously been applied to nuclei^{12, 22, 23}, we profiled the mouse cortex because brain tissue is a major example of a tissue type commonly analyzed through single nucleus RNA-seq. Each sample type was tested in two experiments (Mixture1, Mixture2, PBMC1, PBMC2, Cortex1, and Cortex2) run on different days to assess reproducibility.

In each experiment, we aimed to collect data from ~384 cells for the low-throughput methods, ~3,000 cells for the high-throughput methods, and a bulk RNA-seq library as a control. We sequenced libraries together to similar coverage based on library type, except as noted (Methods).

scumi computational pipeline allows unified analysis across any scRNA-seq method

Because each method had its own standard computational pipeline, we developed and used a new “universal” pipeline, to permit direct comparison of all the experimental methods, and remove processing differences introduced by these existing pipelines (Extended Data Fig. 2). First, we developed the scumi software package (single-cell RNA-sequencing with UMI, Extended Data Fig. 2a), which starts from FASTQ files as input and generates gene-cell expression count matrices for downstream analyses.

Second, we addressed the major pre-processing challenge of filtering out low-quality cells prior to downstream analysis (Extended Data Fig. 2b). This is particularly important when comparing methods, to ensure that our approach is fair to all methods and less subjective. When selecting the cell barcodes with the largest number of reads or UMIs assigned to them, the challenge is to decide what threshold to choose for excluding lower quality cells or barcodes likely reflecting ambient RNA rather than real cells (or nuclei). For the mixture experiments we removed cells with low quantities of UMIs or reads per cell. For the more complex PBMCs and cortex samples, consisting of different cells with different characteristics, such a simple approach could bias against the recovery of cell types with relatively small amounts of RNA. Instead, we first looked at more cell barcodes than we expected to truly recover from experiments, did an initial clustering, identified differentially expressed genes to find cluster specific marker genes, and removed cells in clusters likely to be low quality (Extended Data Fig. 2b, Methods).

Third, prior to calculating metrics that potentially show improvements with greater sequencing depths, such as the number of genes per cell or ability to detect known cell types, we sampled the same number of reads per cell for all the methods of the same type, either low- or high-throughput, in a given experiment (Methods, Extended Data Fig. 2c). This leads to better relative performance for methods that have a higher fraction of informative reads, i.e. those reads aligning to genes and present in cells used for analysis. Note that because for most experiments we sequenced the poly(T) sequences that follow the cell barcode and UMI sequences in all methods except Smart-seq2, we tracked and removed reads without poly(T) at the expected positions because reads lacking the expected read structure are unlikely to be informative for further analysis.

Finally, we assessed the methods by several key metrics spanning (1) the structure and alignment of reads to the nuclear and mitochondrial genomes; (2) sensitivity in capturing RNA molecules; (3) extent of multiplets (assessed in mixture experiments); (4) their technical precision/reproducibility with respect to expression estimates; and (5) the ability to recover meaningful biological distinctions in cell types (for PBMC and cortex experiments).

Read structure and alignment reveal efficiency differences among methods

First, we characterized the methods by the distribution of reads from each library with respect to their structure and alignment with the genome (Extended Data Fig. 3). These metrics inform about the “efficiency” of methods in generating useful reads for downstream analysis. In the mixture experiments, we considered only uniquely mapped reads to minimize the effects of multi-mapped reads on calculating cell multiplet rates and other metrics. Methods differed with respect to reads without poly(T) at the expected positions (Supplementary Note 1).We next considered the distributions of reads across these categories: exonic, intronic, intergenic, overlapping different genes (ambiguous), multi-mapped, and unmapped. Exonic reads are typically the only reads used in scRNA-seq studies of cells, whereas intronic reads are also used for studies with nuclei^{24, 25}. In the mixture experiment, both replicates of Smart-seq2 and one replicate of inDrops had the highest fraction of exonic reads (51.0%, 53.7%, and 56.9%, respectively), with sci-RNA-seq performing worst (28.7% and 29.4%, Extended Data Fig. 3a). Overall, the PBMC samples had a lower fraction of reads aligned to exons than the mixture samples (Extended Data Fig. 3a,b), with one replicate of inDrops having the highest fraction of exonic reads (46%), and Seq-Well having the lowest (20%, Extended Data Fig. 3b). To explore the origin of unmapped reads, we further analyzed the PBMC datasets and found that most of these reads are low quality, align to adapter sequences added during library construction, or contain stretches of poly(A) (Supplementary Fig. 1a). Using Trimmomatic²⁶ to remove low quality and adapter sequences from the unmapped reads (Methods), we recovered <5% of the unmapped reads in each case, except for inDrops PBMC1 and Seq-Well PBMC2, for which we recovered ~8% and ~18%, respectively, of the initially unmapped reads (Supplementary Fig. 1b).

To determine the extent to which existing annotation limits recovery of reads aligning to genes²⁷, we used the PBMC1 and PBMC2 bulk RNA-seq to create a matched transcriptome and new annotation (Methods). The new transcriptome annotations led to very few (<2%) additional reads aligning (Supplementary Table 1).

While the relative performance of each method was generally similar between the cortex nuclei and the other experiments, there was a higher ratio of intron-aligning reads to exon-aligning reads in nuclei than in cells (Extended Data Fig. 3), as expected because nuclei contain a higher proportion of unspliced transcripts than whole cells²⁸. We assessed whether reads aligned in the sense or antisense orientation for each method, except Smart-seq2, which is not strand-specific, and investigated antisense reads (Supplementary Note 2).

Method performance relative to mitochondrially-aligned reads was also analyzed (Supplementary Note 3).

Similar relative ranking of method sensitivity across experiments

As scRNA-seq methods start with limited RNA inputs, a key quality metric is the sensitivity, or the ability to capture RNA molecules. We assessed the sensitivity of each method by measuring the number of detected UMIs or genes per cell in datasets sampled to the same number of reads per cell (Methods; Supplementary Table 2). The only exception was Seq-Well PBMC1 with ~46,000 reads per cell compared to ~69,000 reads per cell for the other high-throughput methods in PBMC1 (Supplementary Table 2). For the mixture experiments, we report the results for mouse and human cells separately as the number of UMIs and genes per cell in the two cell types differs, such that differences in the ratio of human to mouse cells among the libraries (Supplementary Table 3) could skew the results, but the overall ranking of the methods is the same for both human and mouse cells (Fig. 2a,b, Extended Data Fig. 4a).

Figure 2. Performance metrics for mixture experiments.

(a-b) Distribution of the number of UMIs (a) or genes (b) in human (top) or mouse (bottom) cells in the two Mixture experiments (n=1 biologically independent sample per experiment). For (a) and (b), median and box plots were based on all the cells, but a few outlier cells were omitted in drawing the violin plots. Box plots denote the medians (labeled on the right) and the interquartile ranges (IQRs). The whiskers of each boxplot are the lowest datum still within 1.5 IQR of the lower quartile and the highest datum still within 1.5 IQR of the upper quartile. Violin plot width is based on a Gaussian kernel density estimate of the data (estimated by the density function with standard parameters), scaled to have maximum width = 1. (c) Multiplet frequency. We ordered cells based on the number of detected UMIs (or reads for Smart-seq2), from highest (left) to lowest (right). For a given number of cells (x-axis value), the plot shows the percent of cells that are multiplets. The dotted lines for sci-RNA-seq Mixture1 and inDrops Mixture1 and Mixture2 show the multiplet rate including low-quality cells that were not included in subsequent analysis.

Overall, the low-throughput methods Smart-seq2 and CEL-Seq2 had the highest sensitivities, as expected¹⁶, whereas among high-throughput methods, 10x Chromium detected the most UMIs and genes per cell. In the mixture experiments, inDrops had the lowest sensitivity and Seq-Well detected fewer genes per cell compared to 10x Chromium (v2) and sci-RNA-seq, but more genes per cell compared to Drop-seq and inDrops. The relative ranking of the methods was generally consistent when comparing the median number of detected UMIs per cell (Fig. 2a), detected genes per cell (Fig. 2b), or mean detected reads per cell (Extended Data Fig. 4a). Similarly, in PBMCs low-throughput methods detected more UMIs and genes per cell than the high-throughput methods (Fig. 3, Extended Data Fig. 4b), with similar performance of Smart-seq2 (2406 and 2632 median number of genes detected) and CEL-Seq2 (2717 and 2545; Fig. 3b). Among the high-throughput methods, 10x Chromium (v3) had the highest median number of UMIs (4494) and genes (1482) per cell (Fig. 3), and inDrops (366 and 1118 UMIs; 256 and 568 genes) and Seq-Well (844 and 577 UMIs; 513 and 372 genes) had the lowest (Fig. 3). In cortex nuclei, Smart-seq2 was the only low-throughput method tested and we sequenced to a slightly higher depth than for the other samples (Supplementary Tables 3-5) and used all the reads. As expected¹⁶, Smart-seq2 detected more genes per cell than the high-throughput methods (Fig. 4, Extended Data Fig. 4c). Among the high-throughput methods, 10x Chromium (v2) had the highest median number of UMIs (5126 and 3127) and genes (2462 and 1744) per cell (Fig. 4, Extended Data Fig. 4c).

Figure 3. PBMCs sensitivity.

Distribution of the number of UMIs (a) or genes (b) per cell for each method in the two experiments (n=1 biologically independent sample per experiment). Violin and box plot elements are defined as in Fig. 2.

Figure 4. Cortex nuclei sensitivity.

Distribution of the number of UMIs (a) or genes (b) per cell for each method in the two experiments (n=1 biologically independent sample per experiment). Violin and box plot elements are defined as in Fig. 2.

To explore how sensitivity varied with sequencing depth, we sampled fewer reads per cell from each method in the PBMC datasets (based on the molecular information matrices, Methods). For each dataset, the relative ranking of the methods with respect to the median number of genes per cell (Extended Data Fig. 5a,b) or UMIs per cell (Extended Data Fig. 5c,d) detected remained the same at all sequencing depths tested. In addition, the number of genes detected may not have saturated at these sequencing depths, except for Seq-Well PBMC2, though it is possible that low levels of reads from other cells increasing the apparent number of genes detected may confound these analyses. Comparing this analysis sampling from the molecular information matrices vs. from the raw reads for five of the PBMC1 libraries yielded indistinguishable results (Supplementary Fig. 2). Notably, the number of genes and UMIs per cell are highly correlated (Extended Data Fig. 5e).

We also performed comparisons with published datasets (Supplementary Note 4).

Mixture experiments enable detection of multiplets and reads from other cells

In the mixture experiment, we were able to assess the frequency of multiplets, because we started with a mixture of human and mouse cells. The observed multiplet rates were <3.5% for all seven tested methods (Fig. 2c), except for the first inDrops experiment, which also had a high fraction of reads without poly(T) (Supplementary Table 3). The multiplet rate depends on the number of cells used in each experiment⁹ and the ratio of mouse to human cells, but it was not possible to sequence the same number of cells nor the same ratio of mouse and human cells with each method (Supplemental Table 3). The multiplet rates of low-throughput methods were the lowest (<1%), as expected as FACS was used to place a single cell in each well of a plate (Fig. 2c).

We also examined how the estimated multiplet rate varied with the number of detected UMIs per cell. Generally, multiplet rates were higher in cells with the largest number of UMIs (Fig. 2c), as expected because multiplets are expected to have more RNA input. While most cells with intermediate quantities of UMIs were not multiplets, cells with the lowest number of UMIs in some cases had higher rates suggesting that these cells might be low-quality or have more contributions from cell-free ambient RNA (Fig. 2c). In sci-RNA-seq experiment 2, the rate of multiplets decreased more gradually than for other methods for unknown reasons (Fig. 2c).

We also used the mixture experiments to ask whether the genes detected in a cell were actually from that cell instead of “contamination” from other cells. As sequencing depth increased, more genes were detected from the “wrong” species (Extended Data Fig. 6a,b), as reflected by the slope of a regression line along the cell barcodes adjacent to each axis (Methods), such that the best performing methods have the lowest slope. For the low-throughput methods, Smart-seq2 performed much better than CEL-Seq2. Among the high-throughput methods, inDrops had the best (lowest) slope and Seq-Well had the highest slope.

Technical precision, reproducibility, and accuracy in gene expression quantification

To assess technical precision in the mixture experiment, which consisted of two homogeneous cell lines grown in controlled conditions in culture, we also compared the variation in scRNA-seq data, which we expect to be primarily driven in this case by technical variation¹⁶, although some intercellular heterogeneity may still have been present in our cell cultures. Previous studies have demonstrated that such technical variation generally follows Poisson distributions^{16, 29, 30}. CEL-Seq2, inDrops, and Drop-seq consistently had relatively low extra Poisson coefficients of variation (CVs) (Extended Data Fig. 7). Consistent with previous findings, Smart-seq2 data had the highest extra Poisson CV, most likely because no UMIs were used (Extended Data Fig. 7).

We also compared reproducibility between replicates and accuracy using bulk and pseudo-bulk data (Supplementary Note 5).

Methods vary in their ability to distinguish and recover cell types

A key consideration in choosing a scRNA-seq method is its ability to uncover the underlying biology of interest. Among the many biological features studied by scRNA-seq, one of the most prominent use cases is the identification of distinct cell types by clustering scRNA-seq profiles. Both the PBMC and mouse cortex datasets consist of diverse cell types, and were chosen to allow us to compare methods for this use case.

To this end, we processed the data with the goal of a fair and optimal assessment of each method. Not only did we sample the same number of reads per cell for each low- and high-throughput method in each experiment as above for the sensitivity metrics, we also performed another round of sampling to use the same number of cells from each low- and high-throughput method in each experiment (Methods). The only exceptions were Seq-Well PBMC2, which had fewer cells (Supplementary Table 2 and 4) because we used only one microwell array for that experiment, while we used two arrays for Seq-Well PBMC1, and DroNc-seq Cortex2, which had fewer cells for unknown reasons (Supplementary Table 2 and 5).

For each dataset, we clustered the cells or nuclei based on their gene expression profiles to assess how well they detected the known cell types and their associated transcriptional profiles. For each dataset, we searched a range of parameters to select the optimal clustering to recover each of the expected cell types (Methods). We assigned each cluster a cell type identity based on known marker genes (Methods). To quantify the quality of the clusters at separating cell types, we scored the expression of each cell for each cell type signature generated from known marker genes and calculated the area under the curve (AUC) for each cluster to estimate how well the cells in a cluster score for each cell type (Methods). The AUC summarizes the performance of the gene signature scores in separating a cluster of cells from the rest of the cells, with AUC =1 for all cell types as the ideal outcome.

For PBMCs, methods varied in the ability to distinguish cell types, in the proportion of cell types recovered, and, in some cases, in the recovery of certain cell types altogether. As expected, methods had more difficulty in distinguishing transcriptionally related cell types, such as CD4⁺ T cells, CD8⁺ cytotoxic T cells, and natural killer (NK) cells (Fig. 5a,b, Extended Data Fig. 8). From the t-SNE plots for PBMC2, we observed that 10x Chromium and inDrops performed well (Fig. 5a, Extended Data Fig. 8b). As all the libraries for each experiment were generated from the same sample, we assessed the consistency across methods in the fraction of cells assigned to each cell type within an experiment (Fig. 5b). Generally, most methods successfully recovered the abundant cell types in PBMCs, but varied in the relative abundance of cell types. Methods also varied in whether cell types were detected, particularly for the rarer cell types, such as plasmacytoid dendritic cells and contaminating platelets that were captured in various proportions across methods (Fig. 5b). Although platelets are not PBMCs, their presence in this dataset reflects incomplete purification of PBMCs from whole blood, as has been observed previously⁹. For the low-throughput methods, we did not profile a sufficient number of cells to recover the rarer cell types (Fig. 5b) and performed similarly for the AUC measurements (Fig. 5c). In PBMC1 among the high-throughput methods, 10x Chromium (v2) showed the best quality for both the number of cell types identified and the average AUCs across cell types, followed by Drop-seq and 10x Chromium (v3), with Seq-Well and inDrops not identifying two cell types (Fig. 5c). In PBMC2, 10x Chromium (v2) and inDrops performed well – identifying all the cell types (Fig. 5c). For Seq-Well PBMC2, the poor performance was strongly influenced by the low number of cells recovered in the experiment (Fig. 5c).

Figure 5. Cell type identification and assignment in PBMCs.

(a) t-stochastic neighborhood embeddings (t-SNEs) of single cell profiles (dots) from representative PBMC2 libraries colored by cell type. (b) Proportion of cells of each cell type (y-axis) detected with different methods (x-axis). Those not labeled with a number rounded to one or less. Sum does not always add to 100 due to this and rounding. (c) The AUC (dot size, color and value) of each cluster from classifying the cell type to the cluster it was assigned for PBMC1 and PBMC2. See Supplementary Table 2 for the numbers of cells used (n=1 biologically independent sample per experiment).

Similar to PBMCs, the mouse cortex has well-defined cell types, including excitatory and inhibitory neurons, astrocytes, oligodendrocytes, oligodendrocyte progenitor cells (OPCs), microglia, endothelial cells, and pericytes³¹. In both experiments for all the methods, apart from sci-RNA-seq, we identified all these cell types, except pericytes, a rare cell type only found in DroNc-seq Cortex1 (Fig. 6; Extended Data Fig. 9). In the sci-RNA-seq datasets, we also could not find OPCs and microglia (Fig. 6). In the AUC analysis, Smart-seq2, 10x Chromium (v2), and DroNc-seq all had high AUCs, though their relative ability to detect the expected cells varied by cell type (Fig. 6c). Notably, even the small number of cells in the Smart-seq2 datasets (295 and 349) sufficed to find these cell types, in contrast to the PBMC datasets (Fig. 5). In the sci-RNA-seq datasets, we could not confidently assign cell types to some clusters of cells (7% and 4% of cells; Fig. 6a,b).

Figure 6. Cell type identification and assignment in cortex nuclei.

(a) t-SNEs of single cell profiles (dots) from Cortex1 libraries colored by cell type. (b) Proportion of cells of each cell type (y-axis) detected with different methods (x-axis). (c) The AUC (dot size, color and value) of each cluster from classifying the cell type to the cluster it was assigned for Cortex1 and Cortex2. See Supplementary Table 2 for the numbers of cells used (n=1 biologically independent sample per experiment). We could not confidently assign cell types to some clusters of cells from sci-RNA-seq and these cells were not used in calculating the AUCs.

Pooled data analysis across methods enhances biological signal and consistency

Two general reasons may underlie the failure to detect certain cell types: (1) libraries did not contain cDNAs from these cell types due to experimental issues; or (2) data quality from these cells may not have been sufficient to identify them, given this depth of sequencing and number of cells. To distinguish between these possibilities, we combined for each PBMC experiment all the sampled data together using Harmony³², re-clustered the cells (Extended Data Fig. 10a), and repeated our analysis. Following this analysis, all cell-types were detected in each library, supporting the second possibility and showing the power of accruing data across methods (Extended Data Fig. 10b-d). Moreover, we determined in which cell type these missing cell types were originally (mis)assigned (Extended Data Fig. 10c,d). Although most of the combined and individual cell type assignments agree, some cell types seemed to be harder to distinguish. For example, in several libraries, such as Smart-seq2 and CEL-Seq2, the undetected dendritic cells were grouped with the CD14⁺ or CD16⁺ monocytes (Extended Data Fig. 10c,d). Overall, 10x Chromium (v2) was the most consistent between the combined and individual level clustering, followed closely by 10x Chromium (v3), and others having fairly high but variable levels of consistency. By contrast with the datasets with lower quality or numbers of cells, the AUC scores for cell clusters in the 10x Chromium datasets did not consistently improve when all the datasets were used with the Harmony algorithm (Supplementary Fig. 3).

To check the cell types assigned by the combined analysis, we examined the cells assigned to cell types missing in our original analysis of each library separately. In most cases (20/25), we found that these cells could be assigned the same identity using our original AUC method (Methods), with some exceptions for rare cell types with only ~1-2% of cells in a cluster (Supplementary Table 6). Thus, the failure to identify all the relevant cell types was due, as least in part, to data quality issues, such as reads that could not be used in the analysis (Supplementary Table 4), with the possible exception of two rarer cell types in our datasets, platelets and plasmacytoid dendritic cells, which may not have been present in some datasets. Another possible explanation is that low quality cells were included that prevented identification of distinct cell types – this points to the difficulty in finding an optimal filtering threshold as well.

Similar analyses were done for mouse cortex nuclei (Supplementary Note 6).

Comparison of scumi with standard computational pipelines

Although we used the scumi computational pipeline in this study to analyze each method’s datasets in as similar a manner as possible, we also processed each of the datasets with its original method-specific pipeline for comparative purposes and found generally similar results (Supplementary Note 7).

DISCUSSION

In this study, we systematically benchmarked seven methods across three major categories: plate-based, bead-based, and combinatorial index-based methods and summarized their relative merits (Supplementary Table 7). Our results were generally consistent in their ranking of the methods for sensitivity (Figs. 2-4), reproducibility (Supplementary Fig. 4), technical precision (Extended Data Fig. 7), and capturing biological information about cell types (Figs. 5, 6, Extended Data Fig. 10, Supplementary Fig. 5). Having a lower fraction of reads aligned to exons (Extended Data Fig. 3) could explain lower performance metrics in our comparisons, e.g., in the Seq-Well PBMC datasets (Figs. 3, 5). One limitation in our study is that our samples were not appropriate for pseudotime analysis³³.

All of the methods were able to generate useful data, but overall we found that 10x Chromium had the strongest consistent performance – similar to a more limited comparison of high-throughput methods²⁰. In our limited testing of 10x Chromium (v3), it had higher sensitivity (Fig. 3), but we did not detect improved cell type identification (Fig. 5) and had a higher fraction of reads aligned to mitochondrial genes (Supplementary Fig. 6). sci-RNA-seq, which has the ability to scale to much larger numbers of single cells¹³, may require optimization for use with some samples, such as PBMCs. We used the original version with two rounds of indexing¹². Moreover, its performance with cortex nuclei was not ideal as it could not assign an identity to some cells and did not detect all the cell types present (Fig. 6, Supplementary Fig. 5). For the low-throughput methods, Smart-seq2 and CEL-Seq2 performed similarly without a consistent pattern for which was better (Figs. 2-5). For studies that require the highest sensitivity, these two methods are clearly better than the high-throughput methods (Figs. 2-4) as shown previously¹⁶. Smart-seq2 has inherent advantages for genetic variant detection and studying RNA splicing isoforms because its sequencing is not limited to the 3’ end of genes – along with the disadvantage of lacking UMIs. Note, however, that in CEL-Seq2 we cannot rule out the issue of contaminating reads from other cells (Extended Data Fig. 6)³⁴ and speculate there is a step after cDNAs are pooled from different cells in which cell barcodes from one cell are switched with those from a different cell.

Looking beyond performance, we compared the time and reagent costs for each method as performed in this study (Supplementary Table 8). Drop-seq, Seq-Well, and inDrops had the lowest costs and Smart-seq2 was the most expensive, primarily because there is no pooling during library preparation. Many of the methods, particularly sci-RNA-seq, would be more cost effective with larger numbers of single cells or nuclei¹³. The 10x Chromium method required the least time and Smart-seq2, CEL-Seq2, and inDrops took the most time. We did not utilize automation, but it could decrease hands-on time and affect cost.

Analyzing single nuclei rather than single cells is an important strategy, which addresses tissues that cannot be readily dissociated into a single cell suspension (such as brain, skeletal muscle or adipose) and frozen samples, as well minimizes the alteration of gene expression which may be caused by dissociation^{35, 36}. As in previous studies^{24, 37}, we found that single nucleus RNA-seq generally performed well for sensitivity (Fig. 4) and classification of cell types (Fig. 6). Even with the inclusion of intron-aligning reads in our analysis, a higher fraction reads for 10x Chromium, and to a lesser extent for DroNc-seq, could not be analyzed because of the absence of a poly(T) sequence or aligning in an antisense orientation (Extended Data Fig. 3).

Our study, the scumi pipeline (for its relative advantages, see Supplementary Note 8), data and approaches will be a resource for future research in many fields where scRNA-seq methods are applied, and provides important guidance. First, using a coherently and reproducibly collected set of data, spanning three sample types, it provides direct guidance on key methods by a rich set of parameters and considerations – from technical to biological. It spans key and popular methods, including the first comparison of single nucleus RNA-seq methods. We chose two very different tissue types to make our framework and conclusions more generalizable, and show this by analysis of published datasets for three other tissues (Supplementary Fig. 7). Our study would allow future studies to reduce the number of protocols that need to be tested for new tissues. Second, the results presented here for each method could be used to further optimize and improve existing scRNA-seq methods. Third, our use of representative and easily accessible sample types should allow future studies, particularly those introducing new or improved methods, to make direct comparisons to this benchmark study. Indeed, all datasets were collected in a manner that allows open sharing, including the human PBMC data. Finally, we expect our datasets will be valuable for computational method developers to benchmark algorithms and build pipelines for efforts such as HCA, the BRAIN Initiative, the Cancer Moonshot Human Tumor Atlas Network (HTAN), and other efforts to map cells in disease.

METHODS

Single cell or nucleus experimental design

We performed two experiments with each single cell method for the mixed cell lines and PBMCs, except as noted. To generate data for Seq-Well, we performed a second PBMC1 experiment on a different day with an aliquot identical to the one used in the main PBMC1 experiment to obtain a Seq-Well dataset with sufficient cells profiled for PBMCs. Similarly, we performed a third PBMC1 experiment with 10x Chromium (v2; designated as “B”) and (v3) on a different day. In addition, we performed two experiments with four methods for the mouse whole cortex nuclei. In all cases, each lab method was started at the same time by different researchers, so that the results would be directly comparable without any confounding due to the time cells or nuclei waited to start the experiment.

Additional experimental details are in Supplemental Methods.

Computational Methods

We present key steps in the analysis with the scumi pipeline here and other analysis can be found in the Supplemental Methods.

Annotating each cDNA read with its cell barcode and UMI

Different scRNA-seq protocols produce reads with different structures, especially the reads consisting of cell barcodes and/or UMI sequences. To address this issue, we used our scumi pipeline, which uses regular expressions (text strings defining search patterns) to extract the cell barcodes and UMI sequences from different FASTQ files and put them in the header of their corresponding cDNA reads. We started with the FASTQ files generated after de-multiplexing the BCL files from Illumina sequencers. For a typical 3’-tag based scRNA-seq experiment, a cDNA sequence fragment is in one read of a FASTQ file, and its corresponding cell barcode sequences and UMI sequences are in a paired read of a separate FASTQ file. For example, for the reads generated from the Drop-seq platform⁸, the cDNA reads are in read 2 and the cell barcodes (base 1 to base 12) and UMIs (base 13 to base 20) are in read 1. Details about the location of cell barcodes and UMIs can be found in Supplementary Table 9.

The scumi pipeline also corrects for sequence errors in the indices used by bead-based methods. We have implemented code similar to the standard Drop-seq pipeline that overcomes the problems observed for some batches of Drop-seq beads, in which up to 20% of the cell barcodes have errors in the last base (base 12), mostly because the beads (encoding the cell barcodes) only synthesized 11 (or fewer) bases⁸. In such cases, base 12 of the cell barcode is actually the first base of the UMI sequence, and the last base of the UMI sequence is from the poly(T) sequence. This bead synthesizing error can be detected by calculating the frequencies of T bases in the UMI sequences. The scumi pipeline first detects possible erroneous cell barcodes and then merges these cell barcodes that are the same in their first 11 bases but differ in the 12th base. If more than one base of the UMI sequences (with the same cell barcode) had a high-frequency of T bases (more than 80%), these cell barcodes were removed from further analyses.

Mapping reads to a reference genome

We aligned the merged FASTQ files (each cDNA read with its cell barcode and UMI annotations) to a reference genome using STAR³⁸ v 2.6.1a, except for Smart-seq2 and CEL-Seq2. For those libraries, we used HISAT2³⁹ v2.0.5 as it is better suited than STAR to paired-end read data such as Smart-seq2 because of the way it handles read pairs that do not both align to the same region of the genome – leading to more aligned reads and more detected genes per cell. Notably, HCA has adopted this aligner for its Smart-seq2 pipeline (https://staging.data.humancellatlas.org/learn/userguides/data-processing-pipelines/smart-seq2-workflow/). We also used it for CEL-Seq2 to facilitate better performance comparisons between the two low throughput methods. For mixture data, we used the STAR reference available in the hg19 and mm10 v2.1.0 Cell Ranger reference. For PBMC data, we used the STAR reference available in the GRCh38 v1.2.0 Cell Ranger reference. For cortex data, we used the STAR reference available in the mm10 v1.2.0 Cell Ranger reference. We downloaded Cell Ranger reference data from https://support.10xgenomics.com/genome-exome/software/pipelines/latest/advanced/references. For each sample type, we also generated a HISAT2 reference with the associated GTF and FASTA files.

Annotating each alignment with a gene tag

We use featureCounts⁴⁰ from the Subread package, v1.6.2, to add a gene tag to each alignment. To count reads overlapping with introns for single nucleus RNA-seq data, we used a two-step approach to first count the reads overlapping with exons. In the second step, the reads not overlapping with exons were recounted if they overlapped with introns. We only included reads aligning in the sense orientation with the genome annotation, except for Smart-seq2, which does not generate strand-specific data.

Counting transcripts of each gene in each cell

For the UMI-based methods, we used scumi to generate a cell x gene UMI count matrix. We included a multi-mapped read if all its alignments overlapped with a single gene, similar to the Cell Ranger pipeline⁹. We collapsed UMIs in reads from the same gene from the same cell based on a Hamming distance of one. To prevent over-collapsing UMIs⁴¹, we did not collapse two UMIs – in the same gene in the same cell – if they each had more than five reads support. For Smart-seq2, we used a similar procedure to generate the count matrix used for the sensitivity and technical precision metrics, except we created a cell x gene read count matrix. For Mixture data, we did not include multi-mapped reads in subsequent analyses, and instead used featureCounts to count uniquely mapped reads. For Smart-seq2, this resulted in each read of a paired-end read being counted separately in the Mixture data (though not in the other experiments), enabling us to count reads in pairs in which each aligned to a different gene. For clustering Smart-seq2 data and downstream analysis, we used RSEM⁴² v1.3.0 to generate a cell x gene transcripts per million (TPM) matrix, which was used instead of the UMI count matrix. We generated the RSEM reference using the FASTA and GTF files used for creating the STAR and HISAT2 references (see Mapping reads to a reference genome). When generating the RSEM reference for cortex data, we modified the GTF to include one unspliced transcript per gene that included all introns and exons in that gene. This allowed us to count reads that mapped to introns.

Selecting the number of cells

For a scRNA-seq experiment, we have a rough estimate of the number of cells N that can be recovered. A simple yet robust empirical method (used by Cell Ranger of the 10x Chromium pipeline) for cell barcode selection is to first estimate the library size m (in reads or UMIs) by the 99th percentile of the top N cell barcodes in terms of the number of reads (or UMIs). The cell barcodes with reads (or UMIs) greater than 0.1m are considered as ‘cells’.

For the cell line mixture experiments, we used different filtering approaches depending on the dataset. For 10x Chromium, Drop-seq, Seq-Well, inDrops, and sci-RNA-seq, we used this empirical rule for cell barcode selection. For Smart-seq2 and CEL-Seq2, we had a better estimation of the number of cells as we sorted individual cells into wells. We used a mixture of two Student’s t distribution model⁴³ to model the read or UMI (log₁₀ transformed) count distributions of each cell, and removed the cells that were likely from the mixture component with fewer reads or UMIs (posterior probability ≥0.5). The parameters of the Student’s t mixture model were estimated by maximizing the posterior distribution using the Expectation-Maximization algorithm. For sci-RNA-seq and inDrops, the empirical rule tended to select low-quality cell barcodes. We therefore used this mixture model on the cell barcodes selected by the empirical rule to further filter out likely low-quality cells.

For all the high-throughput PBMC datasets, we extracted two times the number of expected cell barcodes for each method, choosing the cells with the most reads. We removed cells with a high fraction of reads aligning to mitochondrial genes (names starting with ‘mt-’ for mouse and ‘MT-’ for human) – greater than 75th percentile + 3 * IQR of the mitochondrial ratios across the top returned cell barcodes, where IQR stands for interquartile range. For each cell, its UMI counts were divided by the total number of UMIs from that cell and then scaled by multiplying 10,000 to get transcripts per 10,000 (TP10K). We then added 1 to these TP10K and log transformed by the natural log. We then performed PCA using all genes, did clustering analysis (Louvain clustering^{44, 45} of the k-nearest neighbor (k-NN) graph built from the first 50 principal components of each single-cell dataset with parameter k=30 and a resolution parameter used for Louvain optimization of 1.0, implemented in the Seurat package⁴⁶ v2.3.4 (see Parameter selection for clustering analysis section for more details), followed by differential gene expression analysis with the FindAllMarkers command in Seurat to find cluster specific (up-regulated) marker genes. To filter out clusters of cells likely derived from low quality cells or empty droplets, we removed clusters with insufficient markers genes, as follows. First, we identified marker genes for each cluster as genes expressed in ≥25% of cells in that cluster and with FDR <0.01 (significantly highly expressed in the cluster compared to the cells not in that cluster). Second, we excluded ribosomal protein coding genes, MALAT1, and genes starting with MTRNR, as they could be erroneously identified as marker genes after normalization and scaling because for cells with a small number of UMIs, the UMIs of highly expressed genes will be weighted more than those from cells with a large number of UMIs based on the scaling formula x_j/sum_j x_j, where x_j is the expression of gene j of a cell. Third, we only kept the clusters in which > 70% of the top 15 marker genes (or 10 out of all markers genes in a cluster that had <15 marker genes) were not mitochondrial protein coding genes as high expression of mitochondrial genes can indicate stressed cells or empty droplets. This process was repeated twice.

We used a modified strategy for Smart-seq2 and CEL-Seq2 because there were fewer cells to cluster, which potentially could have led to the low-quality cells not forming distinct clusters. The assumptions for these cell selections were that (1) there were enough low quality cells to form distinct clusters and (2) the clustering algorithm did not split high-quality cells of the same cell type into many distinct clusters because this could have led to some sub-clusters having too few marker genes. We therefore set k=5 (the number of neighbors in building the k-NN graph) to detect small clusters with a low resolution parameter of 0.5 to prevent splitting cells of the same type into many clusters. We also only used the top 25 principal components as we did not expect to identify as many cell types from a smaller number of cells.

For each cortex dataset, the number of UMIs per cell barcode across all cell barcodes returned by scumi followed a bimodal distribution, with some cell barcodes having few UMIs and others having many. We therefore first used a mixture of two Student’s t distribution models to fit the UMI count distribution across all the returned cell barcodes. We considered the mixture component with a larger mean as the high-quality cell barcode component. We removed from further analyses the cell barcodes with posterior probabilities <0.5 from the high-quality component. The cell barcodes passing this mixture model filtering (with posterior probabilities from the high-quality cell barcode component ≥0.5) were then further filtered using the approach previously described for the PBMC data. We used the top 25 principal components and set the number of nearest neighbors k to 10 in clustering analyses (the choice of parameters was to help recover rarer cell types with lower numbers of UMI per cell, such as the oligodendrocyte precursors). To prevent splitting up big clusters due to the small k, we lowered the resolution to 0.8. For these nuclei libraries, the mitochondrial ratios were very low compared to those from cells, so that it was less likely to see low quality nuclei with mitochondrial protein coding genes as the top marker genes. Therefore, in addition to using mitochondrial protein coding genes to remove poor quality nuclei, we removed cluster-specific marker genes that were also expressed in >70% of the nuclei from the other clusters in a given library. We also used the top 20 marker genes in each cluster for filtering. For Smart-seq2 data, we only used the mixture model for cell selection as the remaining clusters could be assigned known mouse cortex cell types and applying the cluster-based filtering removed many cells that could be easily identified as known cell types.

Sampling reads

In order to correct for differences in sequencing depth between methods, we used seqtk v1.0 (https://github.com/lh3/seqtk) to sample the sequencing data, so that for each method we could analyze nearly the same average number of reads per cell. Low- and high-throughput methods were sampled separately. For a given experiment, we first decided on the average number of reads per cell to sample. We usually set this equal to the lowest average number of reads per cell for a method in that experiment, except for Seq-Well PBMC1, which had a lower average number of reads per cell, so that we chose the library with the next lowest average number of reads per cell. For each library in a given experiment, we then derived a sampling ratio by dividing this sampling target by the original average number of reads per cell in that library. We sampled the FASTQ file for each library with the ‘seqtk sample’ command, using the sampling ratio calculated above and the random seed set to 100 (after combining FASTQ files from different sequencing runs). Although we aimed to have the exact same average number of reads per cell for each library, there were some small deviations from this in practice because the number of cells we identified in each library was not always the same before and after sampling, as well as due to the random nature of the sampling. Details of the sampling for each experiment are in Supplementary Table 2.

Automatically assigning cell types to clusters

We followed common practices for scRNA-seq data clustering. Specifically, cells were divided into non-overlapping clusters by using the Louvain community detection algorithm^{44, 45}. For each cell from a dataset, we computed its k-nearest neighbors in that dataset, and then built a directed k-NN graph using all the cells from that dataset. This directed k-NN graph was further converted to an undirected weighed graph by using shared neighbors. The Louvain algorithm was used to partition the undirected weighted k-NN graph into non-overlapping clusters.

We used marker genes for each cell type to compute a score for each cell and automatically assign cell types to clusters. Both human PBMC and mouse cortex have well-annotated cell types and marker genes for each cell type^{31, 47, 48}. We generated lists of marker genes for each tissue with manual curation (Supplementary Tables 10 and 11). The score of cell i for cell type m is a normalized version of the percentage of total counts from marker genes from cell type m. Assuming that there were N_m marker genes for cell type m, we considered these N_m genes combined as a ‘meta-gene’ with counts $f_i^m=\sum _j^{N_m}{x}_{i,j}$ in cell i, where x_i,j was the expression (UMI count) of marker gene j for cell type m. The meta-gene relative expression in cell i was its count divided by the total count C_i in cell i. We obtained the score of cell i for cell type m as $s_i^m=\log (f_i^m∕C_i\ast {10}^4+1)$.

Based on the scores, we assigned cell types to clusters using the area under the receiver operating characteristic curves (AUCs). Stated another way, for a given cluster and a given cell type c, a cell i in that cluster is a true positive if the score $s_i^c$ is above a given threshold and a false negative otherwise. On the other hand, a cell not in that cluster is a false positive if it has a score above the threshold and a true negative otherwise. A receiver operating characteristic (ROC) curve plots the true positive rate against the false positive rate at different score thresholds. The AUC is 1.0 for perfectly assigning a cell type to a cluster (we can find a threshed score perfectly separating a cluster from the rest), and around 0.5 for randomly assigning a score to a cell. Specifically, for each cluster, the cell type with the maximum AUC was assigned to that cluster. As the same type of cells can be split into several clusters, after initial assignment of cell types to clusters, we recomputed the AUC of a cluster for a cell type by excluding other clusters of cells that were assigned to that cell type. This process was repeated until no changes in the cluster assignment. We then calculated the AUC for a cell type by merging the cluster of cells that were assigned to that cell type.

Data and software availability

RNA-seq data generated in this project are available from Gene Expression Omnibus with accession number GSE132044 and the Single Cell Portal (https://portals.broadinstitute.org/single_cell). The scumi Python package is available freely from bitbucket repository https://bitbucket.org/jerry00/scumi-dev/src/master/ and as Supplementary Code. The R scripts (used to assign cell types to clusters based on a set of marker genes, for parameter selecting for clustering analysis, and for filtering low-quality cells) are available from bitbucket repository https://bitbucket.org/jerry00/scumi-dev/src/master/.

Extended Data

Extended Data Figure 1.

Extended Data Figure 2.

Extended Data Figure 3.

Extended Data Figure 4.

Extended Data Figure 5.

Extended Data Figure 6.

Extended Data Figure 7.

Extended Data Figure 8.

Extended Data Figure 9.

Extended Data Figure 10.