Single cell RNA-sequencing: replicability of cell types

Abstract

Recent technical advances have enabled transcriptomics experiments at an unprecedented scale, and single-cell profiles from neural tissue are accumulating rapidly There has been considerable effort to use these profiles to understand cell diversity, primarily through unsupervised clustering and differential expression analysis. However, current practices to validate these findings vary. In this review, we describe recent efforts to evaluate clusters from single-cell RNA-sequencing data, and provide a framework for considering current evidence and practices in terms of their capacity to establish principles of cell biology. Single-cell RNA-sequencing has already transformed neuroscience. By facilitating detailed comparative and genetic perturbation analyses, it may provide the tools to uncover fundamental mechanisms of neural diversity throughout the tree of life.

Introduction

Single-cell RNA-seq (scRNA-seq) experiments use transcriptional profiling to characterize cells, sometimes grouping them into inferred cell types based on profile similarity [1]. Recent technical advances have increased the scale of scRNA-seq, making it feasible to profile thousands of cells in a single assay [2]. Already, studies of the nervous system span multiple species, anatomical regions, and developmental stages (e.g., [3-8]), and large scale efforts from consortia such as the Human Cell Atlas [9], the Brain Initiative Cell Census Network [10], and the CeNGEN project [11] only promise to increase the amount of available data. Many of these studies are focused on the same task, namely the classification of cells into types. However, because of the high degree of noise [12], the use of ad hoc methodology [13], and the lack of any unifying theory about what constitutes a cell type [14] it is important to assess how well scRNA-seq expression profiles replicate as an initial step toward their validation.

Indeed, the validation of single-cell clusters remains in its infancy, and in this review we aim to provide a framework for evaluating scRNA-seq clusters based on current practices in the field (see schematic, Figure 1). We note that this framework operates at the level of clusters, and we do not explicitly comment on the existence of specific cell types or subtypes, as definitions have yet to be established. In our view, evidence for scRNA-seq clusters can be divided into three levels, each of increasing consequence for establishing principles of cell biology: (i) replicability of expression levels, (ii) generalization to orthogonal data, and (iii) mechanistic validity. Assessments at each of these levels confirm robustness to particular classes of variation, discussed in further detail in the following.

Figure 1 -. ScRNA-seq replicability from transcriptional profile to external validity to mechanism and beyond.

Expression profiles can be compared (A-top left) to characterize replicability or clustered (A-top middle) to find groups of similar expression profiles, also visible as groupings of cells (A-top right) when the data is summarized via dimension reduction. New datasets should show the same clusters (A-bottom left), which can then be assessed for similarity (A-bottom right). External validity may be partially established by examining spatial localization of markers (B-left), connectivity (B-middle) or epigenomic profiles (B-right). Studies of genetic perturbation (C-left) and conservation (C-right) provide insight into the molecular and evolutionary mechanisms that drive expression diversity.

i. Replicability of expression levels

ScRNA-sequencing has become a convenient method for characterizing heterogeneity in biological systems, yet there are many sources of both technical and biological variation that can influence whether an experimental result is replicable. Technical variation may result from differences in library preparation protocols (e.g., 5’, 3’ or full-length mRNA capture, the inclusion of unique molecular identifiers, linear or PCR-based amplification); differences in sequencing depth; or differences in sample handling (e.g., dissociation procedures may affect cell viability, the probability of doublets, or the expression of immediate early genes). Biological variation is expected between cells, between individuals, and as a result of experimental selection procedures, such as sorting cells based on marker expression, or isolating RNA from nuclei. When comparing across datasets, further interpretive variation exists from the use of different clustering methods to group cells within each dataset Importantly, because scRNA-seq requires destructive sampling of cells, standard experimental design approaches for disentangling technical and biological sources of variability are not applicable. In the following we discuss the evaluation of scRNA-seq results across sources of technical and biological variation, and how these evaluations can provide insight into the replicability of inferred cell groupings.

i.i. Comparing expression profiles

The simplest way to compare two expression profiles is to plot one against another in a sample-sample scatterplot, and to calculate the Pearson or Spearman rank correlation coefficient In these plots, each point represents a gene and its position is determined by the gene’s expression level in each of the samples. In general, correlations between RNA-seq samples are high even when the samples are quite distinct biologically [15], because genes have fairly consistent expression levels. These correlations are typically very significant (e.g., p<10⁻¹⁰) because there are many data points (genes). Correlation is a fairly crude measure of sample replicability, and in bulk RNA-seq there have been efforts to encourage the use of visualizations and quality control (QC) measures that are more sensitive to distributional changes, such as Bland-Altman plots [16]. However, as long as both positive and negative cases are considered, sample-sample scatterplots can be useful for exploratory QC.

One thoughtful implementation of sample-sample comparison looks to recapitulate bulk expression profiles from merged single cells (e.g., [17,18]). Here, the biological signal is held constant (cell identity) to assess different technical approaches (bulk or single cell RNA-seq). Average expression profiles of groups of ten or more cells are often observed to be quite consistent with bulk profiles, even when their individual signals are much more variable (e.g., [18,19]). The observed cell-to-cell variability in scRNA-seq data is sometimes used as evidence for the existence of cell subtypes, and it is exploited by clustering algorithms to group cells into putatively biologically meaningful classes, even though it is also consistent with technical noise [20]. In spite of this limitation, clustering analyses have proven irresistible in the single-cell field, with studies routinely reporting that biologically relevant tissue heterogeneity has been ‘revealed’ by scRNA-seq [5,17-19,21]. We discuss the evaluation of these clusters below.

i.ii. Evaluating clusters

The assessment of scRNA-seq clusters follows two main strategies. The first strategy is to use multiple clustering pipelines to analyze a given data set, or to re-cluster data subsets, and determine how well the new clusters align with the original results (e.g., [19,22]). Here, almost all technical and biological sources of variation within the data are held constant In general, replicability analyses can only provide insight into the type of variation that is being sampled from. Because this strategy tests for in silico replication, it can only speak to the robustness of the results with respect to the parameter choices used for analysis. Consequently, results of these analyses are often good, likely because methods are only lightly calibrated to the data to begin with [23,24].

The second strategy is to perform additional experimental replicates and see whether clusters contain samples from all replicates. More formally, this is an assessment of whether the identified cell clusters replicate across batches, assuming the biological makeup of the replicate experiments can be held constant In scRNA-seq the most pernicious batch effects appear to arise during the preparation of sequencing libraries from cellular RNA [25,26], similar to what was previously observed for bulk RNA-seq [27]. In practice, this means that all cells prepared in a single 10x Chromium well, or on a single Fluidigm C1 chip, will tend to have higher within-batch correlations than across-batch, possibly due to ambient RNA [28] or other biochemical factors (reagents, temperature, etc.). This has prompted the invention of ‘cell hashing’ techniques that use barcoded antibodies to label samples so that they can be pooled in a given reaction (e.g., tumor and normal) [29]. While useful, increased sample multiplexing does not provide a measure of the robustness of results to sources of technical variation. To address this, one successful approach has been to generate multiple scRNA-sequencing libraries from biological replicates, and then validate clusters based on their reproducibility. This approach prevents the over-interpretation of clusters attributable to technical variation, such as differences in the fraction of the transcriptome that is assayed [30]. A prominent example comes from the landmark Drop-seq paper, where the authors aimed to transcriptionally characterize the retina [31]. Here, over forty thousand cells were profiled across seven batches, where each batch contained pooled retinal tissue from four to six mice. The authors identified thirty-nine clusters, and noted that all but one sampled proportionately from all batches. The cluster that failed to replicate more broadly appeared to contain non-retinal cell markers, suggesting a potential dissection artifact yielding cell type ascertainment bias. This experimental design and analysis strategy successfully separates replicable clusters from likely technical artifacts.

An extension of the second strategy is to assess cluster robustness across additional sources of biological or technical heterogeneity [32]. Comparisons between nuclear and whole-cell scRNA-seq, for example, have shown that cells of the same inferred type have broadly concordant expression profiles, with some explainable differences, such as the proportion of reads mapping to introns [33-35]. Comparisons may also be performed across published datasets, either to map cluster identity with respect to an external reference [36,37] or to determine the degree of evidence for putative subtypes, toward the goal of achieving field-wide consensus [38]. This approach has the advantage of sampling more broadly across all aspects of sample preparation and analysis, which are generally held constant within a laboratory. Cross-dataset analysis of expression clusters has a history in cancer, where independent groups aimed to verify cancer subtypes by comparing average profiles across studies [39]. Among single-cell studies, clusters derived from newly generated data are sometimes compared to previously reported clusters, treating prior results as post-hoc validation data (e.g., [40,41]). Cross-dataset comparisons of coarse-grained cell classes are often straightforward. As an example, Habib et al compared the clusters from their nuclear Drop-seq of mouse cortex and hippocampus to those characterized by the Allen Brain Atlas (ABA) using Smart-seq2 of the mouse cortex [19], finding positive correlations between cells of the same broad type (e.g., microglia) [42]. To map their results at finer resolution, the authors opted to take a machine learning approach, training a random forest classifier from their own GABAergic data to re-classify inhibitory clusters from the ABA, reporting the proportion of cells from each of the twenty-three ABA clusters to be associated with the eight Habib labels. While all of the clusters from the ABA were classified as one of the Habib labels, most Habib labels contained cells from multiple ABA clusters, suggesting differences in cluster resolution between the two datasets. Notably, three of the eight Habib clusters were not clearly associated with any particular ABA clusters. Although there are many thousands of cells in this comparison, the analysis here has an effective ‘n’ of two. In the absence of additional data, such as additional matched datasets (i.e., nuclear Drop-seq and Smart-seq2 of the same areas), or some other orthogonal validation to define ground-truth, it is impossible to say whether the observed failure of replicability is technical or biological. However, these results highlight the importance of critical evaluation of clustering results, and the difficulty of cross-dataset comparisons.

In general, three main issues plague cross-dataset analysis: heterogeneity of class structure between datasets (i.e., cell type proportions), including missing cell types; feature selection, typically of genes; and evaluation of dataset independence. Differences in dataset structure, including missing cell types, can have important consequences for cross-dataset comparison, particularly when model fitting (sometimes ‘manifold learning’) is performed with respect to within-dataset variability [43,44]. If two datasets are being merged, but they contain non-overlapping or only partially overlapping cell types, clusters that are quite distinct may incorrectly align to one another, generating clusters that contain a mixture of cell types. This is the primary failure mode of these approaches. Large class imbalances may also bear on the metrics used for evaluation [45]. Previous attempts to solve issues of feature selection and class structure between single-cell datasets have used machine learning approaches, as described above [22,42]. While these models are powerful and provide a deep summary of the data they are trained on, the primary challenges of machine learning – avoiding overfitting and improving interpretability [46] – are particularly problematic in the context of single-cell data, where artifacts may easily be confounded with biology. These issues also plague more recent approaches for data fusion in scRNA-seq [43,44,47-51]. In addition, establishing dataset independence is crucial for accurate interpretation of cross-study results, informing the degree to which a result may be considered validating. For example, we previously compared data from the ABA with that from Paul et al [52], and found almost perfect replicability of the Sst-Chodl interneuron profile [38]. However, we later learned that these two studies employed the same mouse line for their analyses [53], suggesting that this result is less surprising than it might appear. This is a clear example of non-independence, but there are likely to be cryptic dependencies between datasets due to their reliance on established marker genes, making this a difficult issue to solve.

Ultimately, while there are many complexities associated with cross-dataset analysis, we would emphasize that the outlook is bright Our work provided the first formal evaluation of single-cell RNA-seq data from the brain, indicating that approximately half of the inferred interneuron subtypes from three high profile publications were almost perfectly replicable [38]. This approach has since been adopted to characterize cell identity early in development [40,41], as well as across species [54], finding strong evidence of cluster replicability in each case. We anticipate that the availability of benchmark datasets from large consortia in combination with other comprehensive data resources [6,7], will soon allow for very detailed investigation of transcriptional variation within and across clusters, as well as the interaction of these profiles with other sources of biological and technical variability. We note that although approaches for joint analysis can often be quite complex, the reliance on mutual nearest neighbors is a recurrent theme [38,43,50]. Nearest neighbor or neighbor voting approaches have been shown to be useful across a wide range of machine learning tasks [55] and likely account for most of the performance of even very sophisticated approaches for joint modeling. Together, these data and analyses will enable the field to define consensus cluster profiles, which can then be assessed across additional modalities, and investigated for gene drivers, and evolutionary conservation, discussed in the following.

ii. Generalization to orthogonal data

As described above, scRNA-seq has been remarkably useful for identifying clusters that may represent cell types or subtypes. Yet historically, cell type definitions in neuroscience have been multimodal, requiring multiple strands of evidence to establish cell identity [56]. In light of this, it is common to assess the external validity of single-cell clustering results by performing experiments that can provide orthogonal information, such as imaging to assess spatial localization of marker genes [52], morphological reconstructions [19], retrograde or anterograde tracing [57], or electrophysiological recordings [58]. These data generally derive from different cells than those that were profiled with sequencing, though there are a number of examples of multimodal registration from the same cell (e.g., [59-61]). In the following, we discuss examples of three prominent types of orthogonal data used to provide evidence for neuronal clusters: spatial localization, connectivity, and epigenomic profiles.

ii.i. Spatial localization

A primary goal of scRNA-seq experiments is to discover novel marker genes that can distinguish between clusters of samples. Markers are useful because they can act as a “Rosetta stone”, enabling multimodal investigation of cell function through genetic targeting approaches, and they are typically identified through differential expression analysis. Often, the simplest follow-up experiment is to assess their spatial co-localization, particularly because established atlases can provide an initial filter on selected genes. While marker co-localization cannot definitively prove the existence of novel cell types, it provides validation that their co-expression is not a technical artifact of scRNA-seq. Certain spatial patterns known to reflect functional or developmental specification, such as layer-restricted distributions among neurons in the cortex, can also suggest shared lineages.

To avoid relying on a small number of markers or on profiles from dissociated cells, recent work has aimed to assess transcription in intact tissue, either through highly multiplexed RNA fluorescence in situ hybridization (FISH) [62,63] or with in situ sequencing [64]. Here there are different limitations from those of scRNA-seq, related to microscopy and spatial constraints. For example, determining optimal parameters regarding probe design and optical density may be quite specific to the tissue to be assayed, which lengthens start-up time for the application of these protocols across biological contexts. Thus, while highly promising, these techniques have yet to be adopted widely.

ii.ii. Connectivity

Understanding the wiring diagram of the brain has been the major goal of systems neuroscience, and there have already been a number of interesting developments to link neuronal connectivity to transcription. In mouse, a handful of studies have used retrograde tracers or viral injections, sometimes in combination with transgenic approaches, to selectively target cells for scRNA-seq based on their projections to a single location [19,21,57,65,66]. An approach that assays connectivity more broadly by using molecular barcodes, MAPseq [67], has recently been combined with in situ sequencing of barcodes and FISH (BARseq) [68]. In initial analyses of BARseq data, projection profiles have been clustered and compared to their laminar distributions, showing that cells from the same location can have highly divergent connectivity profiles, and, notably, the authors failed to detect a layer 2/3 projection class that was predicted from gene expression [19]. Yet, it is difficult to determine what to expect when comparing clusters from projection profiles and from gene expression: one-to-one relationships may provide evidence that expression is associated with functional distinctions between samples, but they need not occur. Higher degrees of multiplexing, or the evaluation of newly defined cluster markers, will facilitate more comprehensive comparisons of projection mapping techniques to scRNA-seq data, helping to formalize and test hypotheses about the link between molecular and projection identity.

ii.iii. Epigenomics

Patterning of gene expression in different cell types is attributed to the ‘epigenome’ – all of the modifications to DNA that do not alter its sequence, but may be associated with differences in activity. New methods have made it possible to examine chromatin accessibility [69] and DNA methylation [70-72] in single cells, or even to assay expression and epigenetic profiles at once [73]. In a recent study of the human brain, single-nucleus chromatin accessibility and expression profiles were successfully combined to identify putative regulatory regions [74], though the authors noted that the epigenomic data could not resolve finer cell subtypes. Because epigenomic techniques are genome-wide rather than transcriptome-wide they provide very sparse data, which present a challenge. Yet these early results are promising: if epigenomic techniques can be used to map enhancers, they may contribute toward a mechanistic understanding of cell diversity, discussed next.

iii. Mechanistic validity: Perturbation and evolutionary conservation

In the previous section we described current practices to gain additional evidence for clusters identified by singlecell RNA-seq. These experiments are valuable because they tend to have different sources of noise than scRNA-seq, so their alignment to single-cell clusters suggests robustness to technical variation. However, because they are purely observational, they are necessarily limited in their capacity to provide causal information about the mechanisms that drive variation within and across cell clusters. Ultimately, the goal of scRNA-seq should not be to simply define replicable profiles across more and more data modalities, but rather to understand why they are replicable. Requiring this level of understanding clarifies the limitations of external validation through orthogonal data, and specifies what remains to be done: perturbation experiments. This is the gold-standard for experimental biology, but it has yet to be consistently applied as a test of the validity of scRNA-seq clusters, likely because mechanistic experiments can be difficult to design and execute. To our knowledge, there have been at least two studies that knocked down cluster-specific genes and validated their impact on cell fate. Both were performed in the planarian Schmidtea Mediterranea, and the approach was likely feasible because of the organism’s inherent regenerative capacity [75,76]. Eventually, with enough basic understanding of transcription factor and regulatory information, causal modeling may allow for in silico experimentation [77]. A more immediate solution may be found with the implementation of technologies that integrate pooled genetic perturbation screens with single-cell sequencing [78-80].

Taking this further, we suggest that cross-species analysis in combination with genetic perturbation is likely to provide the most nuanced view of factors driving cell identity. Though few cross-species analyses of neural singlecell data have been performed, most have focused on mouse and human expression [54,81]. The examination of a wider variety of species, and particularly non-model systems where genetic and environmental diversity can be assayed, will provide greater insight into the pathways that underlie cell specialization. In this direction, recent work examined the evolution of mammalian neocortex through comparative study of reptilian pallium, finding evidence that GABAergic interneurons are ancestral to both reptiles and mammals, indicating conservation of interneuron diversity [5]. An impressive investigation into the cell type repertoire of the cnidarian Nematostella vectensis found that neurons were enriched for genes originating at multiple evolutionary times, and that a subset of gene orthologs conserved in mouse, nematode (C. elegans) and Nematostella all showed neuronal expression specificity, though gene-gene co-expression relationships were distinct [8]. Taking inspiration from these studies, we envision that future comparative single-cell analyses will incorporate phylogeny, gene age, and natural history to an ever greater degree, allowing us to answer questions like: Did similar cell-types evolve independently? Over what interval did a given type arise? What pressures led to the diversification of cell types? Do all cell types have adaptive significance? Decreasing sequencing costs, and efficient gene targeting with CRISPR should enable comparative study of a wide variety of species so that we can move beyond phenomenology toward a more holistic understanding of cell phenotypes.

Conclusions

Currently, the meaning of “cell type” is richly debated, and thresholds for cluster assignment within any pipeline or acting on any single data set feel arbitrary and premature. We have deliberately avoided this debate in this review by focusing on current practices in single cell transcriptomics, and providing a framework for considering reports of replicability in the context of their explanatory capacity. Moving forward, we believe that meta-analysis across laboratories and species will become increasingly important Moreover, explicitly considering degree of conservation of cell phenotypes opens up the possibility of formalizing definitions of cell type. The analogy to species can be useful to consider: it is sometimes practical to treat species as discrete entities, and sometimes it is not Likewise, at some scales, cell types will be meaningful as discrete entities, but because their existence over evolutionary time is the product of incremental modification, at other scales, it may be more suitable to consider them along a continuum [82]. As comparative genomics and functional genetics join forces with single-cell analysis, not only will we begin to see what replicates, but we will also see what doesn’t. More importantly, we will know how and why the difference occurs.

Highlights.

• Single-cell RNA-sequencing has been used to profile neural cells in many organisms.

• Cell expression clusters are often treated as potential cell types or subtypes.

• Cluster validation requires multiple strands of evidence.

• Mechanistic and evolutionary studies may reveal principles of cell diversity.