Single-cell manifold-preserving feature selection for detecting rare cell populations

Abstract

A key challenge in studying organisms and diseases is to detect rare molecular programs and rare cell populations (RCPs) that drive development, differentiation, and transformation. Molecular features such as genes and proteins defining RCPs are often unknown and difficult to detect from unenriched single-cell data, using conventional dimensionality reduction and clustering-based approaches. Here, we propose an unsupervised approach, SCMER (Single-Cell Manifold presERving feature selection), which selects a compact set of molecular features with definitive meanings that preserve the manifold of the data. We applied SCMER in the context of hematopoiesis, lymphogenesis, tumorigenesis, and drug resistance and response. We found that SCMER can identify non-redundant features that sensitively delineate both common cell lineages and rare cellular states. SCMER can be used for discovering molecular features in a high dimensional dataset, designing targeted, cost-effective assays for clinical applications, and facilitating multi-modality integration.

2. Introduction

A tissue in a living organism often consists of millions to billions of cells. While the terminally differentiated cells with relatively distinct molecular profiles can be readily distinguished via single-cell RNA sequencing (scRNA-seq) at current sampling depth, many cells involved in development, differentiation, and transformation remain difficult to detect^1,2. For example, a fraction of tumor cells in renal cell carcinomas can go through sarcomatoid transformation driven by epithelial to mesenchymal transformation (EMT)^3,4; tumor cells in pancreatic ductal adenocarcinomas can transiently express stemness features (e.g., SOX2) at its invasion fronts^5–7. These cells can be relatively rare in the sampled populations, transiently expressing certain molecular features and thereby may not form distinct clusters in high dimensional feature spaces^8,9.

To detect characteristic features (e.g., genes, proteins) in a single-cell dataset, studies^8,10–13 often employ unsupervised clustering followed by one-cluster-vs-all differential expression (DE) analysis, the optimal way for two-group hypothesis testing. These approaches can detect major cell types governed by lineage features that dominate data variance, but are insensitive to rare but unique features that have relatively small variance and manifest as level gradients within cell-type clusters (a.k.a. cell states)¹⁴. They are also clumsy at detecting features affecting multiple clusters, e.g., transcription factors (TFs) regulating multiple cell types¹⁵, as that involves comparison of an exponentially growing number of cluster combinations. To detect features associated with continuous developmental processes, many studies perform trajectory inference¹⁶ followed by correlation/regression analysis to identify correlated features (e.g., Monocle 2¹⁷). The selection of features depends on trajectories, which could be challenging to infer accurately for complex processes. A detailed comparison was performed by RankCorr¹² across various methods such as statistical tests, logistic regression, MAST¹⁰, scVI¹¹, and COMET¹³.

Most existing approaches regard features as independent variables without exploring their interactions¹⁸. As a result, they tend to identify redundant features (e.g., CD3D, CD3E, and CD3G for T cells). Some recent work such as scHOT¹⁸ and SCMarker¹⁹ started to exploit correlational patterns among co- or anti-expressing genes. However, they do not model complex interactions of more than two genes. SCMarker cannot characterize continuous cell states, and scHOT relies on the accuracy of trajectory inference.

To enhance sensitivity in detecting rare features and RCPs, many studies^20,21 had to slice and dice data spaces in empirical, multifaceted ways⁸ or perform iterative gating²² and re-clustering at variable resolutions, which may lead to biased, irreproducible results. For example, GiniClust2²³ selects a set of features to decide the major clusters and another set of features to discover RCPs (Supplementary Note 1). EDGE²⁴ slices feature space randomly to attempt to find RCPs. CellSIUS²⁵ refines clustering by examining gene sets upregulated in RCPs.

Increasing the number and variety of molecular features and improving the fidelity of the measurements can help discover RCPs²⁶. However, they unavoidably increase the already high cost of experiments. To make assays cost-effective towards clinical applications, it is important to select a compact actionable set of molecular features that unbiasedly represent molecular diversity in high dimensional data. This ability is important for designing and manufacturing customized assays, e.g., 10x targeted gene expression, MissionBio Tapestri and NanoString GeoMx, which perform multi-omics measurements of hundreds of selected DNA, RNA, and proteins.

To address these fundamental challenges, we developed SCMER (Single-Cell Manifold presERving feature selection), which selects an optimal set of features such as genes or proteins from a single-cell dataset. Similar to t-Distributed Stochastic Neighbor Embedding (t-SNE)²⁷ and Manifold Approximation and Projection (UMAP)²⁸, we hypothesize that a manifold defined by pairwise cell similarity scores sufficiently represents the complexity of the data, encoding both global relationship between cell groups and local relationship within cell groups²⁹. By preserving such a manifold while performing feature selection, the most salient features that unbiasedly represent the original molecular diversity will be selected.

SCMER does not require clusters or trajectories, and thereby circumvents the associated biases. It detects diverse features that delineate common and rare cell types, continuously changing cell states, and multicellular programs¹⁵ shared by multiple cell types. It reduces high dimensionality into a compact set of actionable features with definitive biological meanings. This distinguishes SCMER from PCA, t-SNE, UMAP, etc., which result in axes (meta-genes) with complex meanings. SCMER is efficiently implemented in Python using PyTorch³⁰, multithreading and GPU acceleration supported, with a user- friendly single-command interface.

3. Results

3.1. The SCMER Approach

In a nutshell, SCMER (Fig. 1a, Methods) examines a data matrix $\mathbf{\text{X}}$ (n cells x D features) and calculates a pairwise cell similarity matrix $\mathbf{\text{P}}$ representing the manifold in $\mathbf{\text{X}}$. It defines a weight vector w and let $\mathbf{\text{Y}}=\mathbf{\text{Xw}}$. It then calculates another pairwise cell similarity matrix $\mathbf{\text{Q}}$ from $\mathbf{\text{Y}}$ and quantifies the level of agreement between $\mathbf{\text{P}}$ and $\mathbf{\text{Q}}$ using Kullback-Leibler (KL) divergence. Finally, it uses elastic net to find a sparse and robust solution of w that minimizes the KL-divergence using the Orthant-Wise Limited Memory Quasi-Newton (OWL-QN) algorithm³¹. Features with nonzero weights in $\mathbf{\text{w}}$ are deemed chosen. $\mathbf{\text{Q}}$ can also be calculated from a different modality instead of $\mathbf{\text{X}}$, which enables a “supervised” and multi-omics mode of SCMER.

Fig. 1: The SCMER approach.

(a) Workflow of SCMER. SCMER selects the features that preserve the manifold from a single-cell omics dataset $\mathbf{\text{X}}$. Features can be selected from either $\mathbf{\text{X}}$ or another co-assayed omics ${\mathbf{\text{X}}}^{\prime }$. Vector $\mathbf{\text{w}}$ indicates the selection. $\mathbf{\text{Y}}$ is the dataset after feature selection. $\mathbf{\text{P}}$ and $\mathbf{\text{Q}}$ are cell-cell similarity matrices for $\mathbf{\text{X}}$ and $\mathbf{\text{Y}}$, respectively.

(b) Applications of SCMER. SCMER selects features that preserve the manifold and retain inter- and intra-cluster diversity, and thus can be applied to discover rich molecular pathways, integrate modalities, and design customized DNA/RNA/antibody panels of restricted sizes.

(c) Capabilities of SCMER compared with mainstream label/cluster-based differential expression (DE) analysis methods and correlation-based methods. The hypothetical branching trajectories contain common progenitors, precursors for A and B, and mature A and B.

A manifold encodes both clusters and continuums of cells. While clusters usually reflect distinct cell types, continuums reflect similar cell types and trajectories of transitioning/differentiating cell states³². SCMER selects optimal features that preserve the manifold and retain inter- and intra-cluster diversity (Fig. 1b). It can be applied to discover rich molecular pathways, identify prognostic genes, and design customized DNA/RNA/antibody panels of restricted sizes for clinical applications.

To elucidate the cell populations and features that SCMER identifies, we simulated a dataset containing a branching trajectory of 4,000 single cells from five major cell types, namely progenitor, precursors of A and B, and mature A and B (Fig. 1c). A total of 180 features are simulated from four categories (Supplementary Note 2), those (I) specific to one cell type/cluster, (II) shared by more than one cell type¹⁵, (III) gradually changing over cell states, and (IV) transiently activated (also known as checkpoints³³). In addition to major cell type labeling, the cells transitioning from precursor to mature are identified as “RCP-A” and “RCP-B”, which overexpress type-IV features. In as few as 45 selected features, SCMER recalled all types of features. In contrast, the top 45 features determined by a DE analysis are all from type I, while a pseudo-time-based correlation analysis missed type-IV features. As a result, SCMER significantly increased the precision and recall of detecting RCPs, while being comparable to other methods on major cell types (Table 1).

Table 1:

Precision and recall of detecting RCPs on simulated data.

	RCPs				Major cell types
Cell types	RCP-A		RCP-B		Progenitor		Precursor-A		Precursor B		Mature-A		Mature-B
Abundance	2.55%		2.68%		21.23%		22.43%		19.83%		16.73%		15.30%
Precision/recall	Pre.	Rec.	Pre.	Rec.	Pre.	Rec.	Pre.	Rec.	Pre.	Rec.	Pre.	Rec.	Pre.	Rec.
SCMER	0.82	0.68	0.87	0.67	0.97	0.96	0.95	0.96	0.94	0.94	0.95	0.96	0.94	0.93
DE analysis	0.61	0.34	0.73	0.40	0.94	0.95	0.94	0.93	0.95	0.94	0.94	0.95	0.95	0.96
Correlation	0.48	0.36	0.43	0.28	0.91	0.96	0.76	0.67	0.76	0.67	0.88	0.95	0.88	0.92

Listed are precision (pre.) and recall (rec.) using a K-NN classifier for one cell type at a time using feature selected by SCMER, DE analysis using known cell types, and correlation using pseudo-time (Supplementary Note 2). Higher is better for both metrics. Also shown are the abundances of the cell types.

To comprehensively assess SCMER, we ran it on eight datasets^34–41 (Supplementary Table 1) that involve a variety of biological and technological challenges, such as unresolved borderline cells that blur clustering, continuously changing cell states, multicellular and transient cellular programs. For comparison, we used supervised DE analysis and widely-used unsupervised feature selection methods, including highly expressed genes (HXG), highly variable genes (HVG), SCMarker¹⁹, Monocle 2¹⁷, RankCorr¹², GiniClust2²³, EDGE²⁴, and CellSIUS²⁵. SCMER robustly demonstrated the best performance in all the experiments.

3.2. Characterizing cell type and intratumoral heterogeneity

Single-cell datasets derived from cancer samples are often highly complex, containing heterogeneous cell types and states in not only tumor cells but also stromal and immune cells. Supervised analysis of cancer data is challenging as cancer cells are highly plastic⁴² and can express novel unknown features, which can heavily confound clustering and trajectory-based analysis. We applied SCMER on a scRNA-seq melanoma dataset containing 4,645 cells from 19 human melanoma samples³⁴. Most cells were annotated as malignant cells, B cells, T cells, macrophages, natural killer (NK) cells, endothelial cells, or cancer-associated fibroblasts (CAFs) by the authors based on clustering and DE analysis. However, there were unresolved borderline cells presenting between labeled clusters, which resemble multiple cell types and could be either doublets or RCPs (Fig. 2a). By selecting only 75 genes, SCMER clearly preserved the manifold: the resulting UMAP embedding is very similar to the original and the relations among cell types including the unresolved cells are preserved (Fig. 2b and Supplementary Figures 1–2).

Fig. 2: Results of the data of melanoma patients.

(a) UMAP embedding of the dataset without feature selection. Each dot represents a cell and the cell types are color-coded (Macro: macrophages, Endo: endothelial cells, CAF: cancer-associated fibroblasts, Unres: unresolved cells; labels and dots are colored synchronously by cell types).

(b) UMAP of the dataset using SCMER selected genes.

(c) Recall of gene sets for SCMER, SCMarker, Monocle 2, RankCorr, highly expressed genes (HXG), highly variable genes (HVG), principal component analysis (PCA), and differentially expressed genes (DEG, supervised). X-axis is the number of selected genes and Y-axis is the number of covered gene sets. A gene set is considered recalled when at least one gene in the set is selected. “Random” shows the expected number of gene sets for randomly selected markers. The area corresponds to 1.645 x standard deviation on each side. Results above the area has p < 0.05 based on one-sided z-test.

(d-f) RNA expression levels of genes showing intra-cluster gradients. Cells are in the same locations as in

(a) and overlaid with RNA expression levels (color bar).

(g-i) Overall Kaplan-Meier survival curve for selected markers in TCGA SKCM. High and low include patients in above and under 33% percentile, respectively. Each group includes n = 151 patients.

To understand the biological meanings of the selected genes, we compared them with the 11 gene sets described in the original publication that represent important cell types and pathways in the study. The selected genes compactly covered all the 11 gene sets (Supplementary Table 2). Interestingly, genes belonging to the known drug resistance AXL program and MITF program were also selected by SCMER. These genes do not preferentially express in a specific cluster (e.g., PMEL, TOB1, etc. in Fig. 2d and Supplementary Figure 1). Some genes such as PMEL, PDCD1, and OAS2 appeared predictive of survival outcome in TCGA SKCM patients⁴³ (Fig. 2g–i). The genes selected by SCMER which are not reported by the original publication (i.e., the 11 gene sets), are found to be enriched in EMT, inflammatory abnormality of the skin, T cell exhaustion, and other immune pathways (Supplementary Note 3, Supplementary Data 1–2).

To comprehensively assess the performance of SCMER, we varied the number of selected features and recorded the number of recalled gene sets. SCMER consistently recalled more gene sets than other methods for any given number of features (Fig. 2c). SCMER also showed high performance in recalling genes regardless of which sets they belong (Supplementary Figures 3–5) and end-to-end clustering (Supplementary Note 4, Supplementary Table 3).

We also applied SCMER to a large-scale pan-cancer single-cell transcriptomic study consisting of 198 cell lines and patient samples from 22 cancer types³⁵. SCMER showed high sensitivity in characterizing intra-cluster heterogeneity, identifying recurrent heterogeneous programs shared by most cell lines and by patient tumor samples (Supplementary Result 1, Supplementary Figure 6, Supplementary Data 3).

3.3. Defining cell subtypes and states of immunocytes

We further examined SCMER in a complex setting involving many cell subtypes, subtle intra-cluster structure, and shared pathways. The dataset contains 39,563 gastrointestinal immune cells collected from inflamed tissues from ten Crohn’s disease patients³⁶. As a cancer risk factor, chronic inflammation involves extensive interaction among various immune cell types such as helper T cells (T_H) and innate lymphoid cells (ILCs), which are regulated by both shared and cell-type specific TFs and cytokines and are difficult to delineate in high dimensional embeddings. The dataset appeared to include 27 cell types and subtypes/states in the original report. Four major cell types, T cells, B cells, phagocytes, and stromal cells each appeared as a cloud in the original embedding (Fig. 3a) but can be further dissected into subtypes (RCPs). For example, T cells were dissected into eight subtypes/states through further clustering.

Fig. 3: Results of the ileum lamina propria immunocytes data.

(a) UMAP embedding of the original dataset. Each dot represents a cell and the cell types are color coded (T (gd): gamma-delta T cell, Tregs: regulatory T cell, Endo: endothelial cell, TRM: tissue-resident memory T cell, DC: dendritic cell, ILC: innate lymphoid cell).

(b) UMAP embedding of the same dataset based on genes selected by SCMER.

(c-f) Examples of RNA expression levels of the genes selected by SCMER that (c) distinguish major cell types and (d) subtypes, (e) are transcription factors regulating different cell types, and (f) show gradual changes among cell states. Cells are in the same locations as in (a) and overlaid with RNA expression levels (color bar).

(g) The RNA expression level of DUSP1. See Supplementary Figure 7 for DUSP2 and DUSP4.

(h) Distributions of the RNA expression levels in major cell types of the genes above.

(i) Recalls of the gene sets selected by SCMER, SCMarker, Monocle 2, HXG, HVG, PCA, and DEG, similar to Fig. 2c.

Circumventing clustering, SCMER selected 250 features from 3,573 highly variable genes (Supplementary Table 4) with the manifold well preserved. The separability among cell types was comparable with the original embedding, and the manifold of subtypes in each major cell type was maintained (Fig. 3b).

SCMER identified features delineating both clusters and sub-clusters. For example, the well-known lineage features such as CD79A (B cells) and CD7 (T cells) and immune subtype markers such as FCER2 (naïve B cells) and ANKRD28 (TRM) were identified (Fig. 3c–d, Supplementary Figures 7a and 8). Less reported features such as SEPP1 for M2 macrophages were also among the list (Supplementary Figure 7b). The selected features also included genes that encode lysozyme (LYZ), complements (C1QA, C1QB, and C1QC), granulysin (GLNY), and granzymes (GZMA, GZMB, GZMK, and GZMH) (Fig. 3c, Supplementary Figure 7c).

NK and ILC1 cells were mixed together in one cluster and can hardly be further dissected based on unsupervised clustering and DE analysis. However, based on the genes selected by SCMER such as GNLY, CCL4, etc., which displayed dichotomizing levels within the cluster, we were able to further separate NK and ILC1 cells and estimate their abundances (Supplementary Figure 9).

SCMER also found TFs that regulate a wide range of cellular activities, including JUN and FOS (Fig. 3d), which are important for immune cell interactions. These features changed gradually among all the cell types, rather than expressing specifically in certain clusters. Other features such as CD69 (known T cell activation feature) and ODF2L (novel T cell subtype feature) also showed gradual change among subtypes instead of on-and-off patterns (Fig. 3f). Notably, among our selected features that were not reported in the original publication, DUSP1, DUSP2, and DUSP4 (Fig. 3g, Supplementary Figure 7d) were potential key regulators of both innate and adaptive immune responses that are highly relevant to Crohn’s disease (Supplementary Note 5).

SCMER again compared favorably to the other methods that selected various numbers of features (Fig. 3i, Supplementary Note 6). It was evident that the other methods tended to ignore features associated with intra-cluster heterogeneity and multicellular programs. The genes selected by SCMER which do not show in the original publication were also highly enriched in multiple immune pathways⁴⁴ (humoral immune response, leukocyte migration, complement activation, etc.; Supplementary Data 4). Overall, SCMER sensitively preserved different types and levels of heterogeneity in the original data. Besides continuums of cell subtypes, SCMER also achieved top performance on continuous hematopoietic trajectories (Supplementary Result 2, Supplementary Figures 10–11, Supplementary Tables 5–6, Supplementary Data 5–6).

3.4. Identifying molecular drivers in a cancer treatment

More and more studies using single-cell technologies to investigate heterogeneity of cells in response to a genetic or chemical perturbation⁴⁵. In these experiments, cell state may transition under complex kinetics.

To investigate the utility of SCMER in studying cellular responses, we applied it on single-cell data derived from dexamethasone (DEX) treated A549 lung adenocarcinoma cell line³⁸. As reported in the original publication, the 1,429 cells sampled at 0, 1, and 3 hours after the DEX treatment formed a continuum in the transcriptomic space (Fig. 4a), indicating heterogeneous responses of the cell population. After running SCMER on the sci-RNA-seq data, 80 genes were selected, with the manifold and treatment states largely preserved (Fig. 4b).

Fig. 4: Results of the A549 lung cancer cell line data.

(a-c) UMAP embedding of (a) the original sci-RNA-seq dataset, (b) the sci-RNA-seq dataset on SCMER selected markers, and (c) the sci-ATAC-seq peak dataset. Each dot represents a cell. Treatment time points are color-coded.

(d-i) RNA expression levels of selected genes show in (d-f) RNA space and (g-i) ATAC space. ATAC space only includes co-assayed cells.

(j) Heatmap of RNA expression levels of selected genes and motif-based activity of highly variable transcription factors (TFs). (Uncor: uncorrelated, Pos: positively correlated, Neg: negatively correlated, with regard to NR3C1 and NR3C2.) ETV3 and ETV4 are in the ETS transcription factor family.

We inferred TF activities based on motif enrichment⁴⁶ in the chromatin accessibility (sci-ATAC-seq) data co-assayed on the same set of cells³⁸ (Methods, Fig. 4c). Among the top 50 highly variable TFs (Supplementary Figure 12), NR3C1, the primary target of DEX³⁸, had the most prominently increasing activity level over treatment time. Other TFs such as FEV⁴⁷ and the ETS family⁴⁸, also targets of DEX, had decreasing activity levels.

We then correlated the expression levels of the genes selected by SCMER with the activity levels of the top TFs. We found that FKBP5, GALNT18, NRCAM, etc. were positively correlated with NR3C1, while CYP24A1, COL5A2, etc. were negatively correlated (Supplementary Table 7, Supplementary Figure 13). In particular, FKBP5, a factor in the negative feedback loop of glucocorticoid receptor response and regulator of immune processes^49,50, had the highest positive correlation (r = 0.355) in the whole transcriptome; while CYP24A1, which regulates multiple metabolism processes⁵¹, was the most negative (r = —0.365). Cells of high FKBP5 expression levels came mostly from 1 and 3 hours (Fig. 4d), with matched polarized distributions in the RNA and the ATAC embeddings (Fig. 4g). Similar patterns were observed between cells of high and those of low CYP24A1 expression levels (Fig. 4f,i). Compared with other feature selection⁵² and DE analysis methods⁹, SCMER performed one of the best in recalling DEX target genes (Supplementary Note 7 and Supplementary Figure 14).

Interestingly, SCMER also selected a group of genes uncorrelated with prominent TF activities (Fig. 4j, Supplementary Figure 13). Among them were MKI67 (e.g., r = —0.005 with NR3C1) (Fig. 4e,h), which encodes proliferation marker protein Ki-67, and other cell-cycle genes such as CENPF, TOP2A, RYBP, MLH3, etc. Pathway analysis confirmed that these genes are highly enriched in cell proliferation pathways (Supplementary Data 7), indicating that an appreciable fraction of cells continued proliferating despite the treatment. It is not surprising that the levels of these genes were uncorrelated with chromatin state changes, as it has been shown that cell cycling status has little direct effect on chromatin accessibility⁵³. Also among uncorrelated ones were several cancer cell stemness marker genes⁴⁴ such as ACTG1, TSC22D1, and FN1, which may indicate that a fraction of cancer cells maintained their stemness during the course of the treatment. These genes would have been missed by a DE analysis supervised by the treatment time.

Taken together, our results demonstrated the superior power of SCMER in discovering features associated with heterogeneous cellular state change in the context of perturbation experiments. It explores alternative explanations and reports the most salient features representing different facets of cells.

3.5. Mapping features across modalities

One challenge in applying scRNA-seq for cell-typing is that expression levels of mRNAs can differ substantially from those of homologous proteins, due to post-transcriptional modifications⁵⁴. Although performing multi-omics assays may be the ultimate solution, they are currently associated with higher cost and lower throughput. Thus, rather than simply selecting the homologous mRNAs, it is beneficial to identify the set of genes whose expression levels maximally represent cellular diversity at the protein level. This capability can be important for designing targeted, cost-effective assays for preclinical and clinical applications. SCMER is ideally suited for such a purpose, as it allows selecting features in one modality while preserving manifold in another modality.

We ran SCMER on a CITE-seq dataset containing 14,468 bone marrow mononuclear cells (BMNC)³⁹. The protein manifold based on 25 markers was utilized to “supervise” the selection of mRNAs (Methods). CITE-seq, which co-assays mRNA and protein markers from the same set of cells, is ideal for obtaining the optimal mapping between mRNAs and proteins (Fig. 5a,b, Supplementary Figure 15).

Fig. 5: Results of the CITE-seq bone marrow mononuclear cells data.

(a-c) UMAP embedding of the original dataset using (a) protein, (b) all genes, and (c) SCMER selected genes. CD4-like T cells (CD4 T_MEM and CD4 Naïve T), CD8-like T cells [Effector CD8 T, CD4 T_MEM, CD8 Naïve T, gamma-delta T (gdT) cells, and Mucosal-associated invariant T (MAIT) cells] are framed by dotted circles, respectively, for better visual identification. Three circles are present in the result of RNA because of two separate clusters for CD8-like T cells. Also highlighted are progenitor cells [hematopoietic stem cells (HSCs), lymphoid-primed multipotent progenitors (LMPPs), granulocyte-monocyte progenitor cells (GMPs), and Progenitor (Prog) of B cells, megakaryocytes (Mks), red blood cells (RBCs), and dendritic cell (DCs)]. Fully annotated cell types are shown in Supplementary Figure 10.

(d-f) Levels of (d) proteins, (e) genes encoding the proteins, and (f) genes selected by SCMER. Cells are in the same locations as in (a) and overlaid with RNA expression level (color bar).

As shown, the mRNA expression levels of genes that encode the protein markers, such as CD4 (CD4, a T_h cell marker) and NCAM1 (CD56, an NK cell marker), offered low power in delineating the corresponding cell types (Fig. 5d,e). Some markers, e.g., CD45RA (B cells and naïve T cells) and CD45RO (memory T cells) are isoforms of the same gene, PTPRC. Consequently, T cell subtypes were less distinguishable in the RNA space than in the protein space (Fig. 5b). The differences among CD8 T cell subtypes were even bigger than the differences between CD4 and CD8 T cells.

SCMER selected a set of genes that best preserved the diversity at the protein-level, notably the continuum among naïve CD8 T cells, memory CD8 T cells, and effector CD8 T cells (Fig. 5c) (SCMER adjusted Rand index (ARI) 0.544; RNA ARI 0.438; Supplementary Table 8). It identified genes that are non-homologous to the protein markers but better represent the protein level difference, for example, GPR183, KLRF1, CD79B, and S100A4 for CD4, CD56, CD45RA, and CD45RO, respectively (Fig. 5d,f). On the other hand, the SCMER result appeared to better delineate progenitor cells (ARI = 0.489) than the protein markers (ARI = 0.303), which demonstrates a strength of integrating complimentary modalities.

Similar conclusions were drawn when applying SCMER on another smaller PBMC CITE-seq dataset⁴⁰ with 10 protein markers (Supplementary Result 3, Supplementary Figures 16–19, Supplementary Tables 9–11).

Importantly, the genes selected by SCMER from one donor (14,468 cells) appeared to preserve the cell diversity in another donor (16,204 cells) (Supplementary Figure 15), which validated the applicability of SCMER in designing targeted panels for populational level testing.

4. Discussion

For datasets with multiple samples, SCMER stratifies the samples to find consensus features that prioritize biological but not technical variances (Methods). SCMER can also run in various supervised modes. For example, it can select features from a shortlist (Supplementary Result 2) and find the best “partner” features for preselected features (Supplementary Result 3). The framework appears effective on cell line and patient data generated by various technologies, including scRNA-seq and mass cytometry⁴¹ (Supplementary Result 4, Supplementary Figure 20), and can potentially be extended to other modality combinations such as scRNA with scATAC, or mRNA with miRNA.

There are some possible limitations in this study. The evaluations were partly based on the gene sets provided in the publications, which may have some biases. In manifold transferring, SCMER does not provide an explicit mapping from one modality to the other, and thus requires additional analysis to clarify the interaction of features in the two modalities.

SCMER is efficiently implemented. On a dataset with 10,000 cells and 2,000 candidate features, it typically converges in 20 to 40 iterations, which takes 5 to 10 minutes using a 3.20GHz 6-core Intel Core i7–8700 CPU. The time consumption is halved with a middle-end NVidia GTX 960M GPU.

Because SCMER detects informative features that represent wider and more complex biological processes, we expect it to be of interest in projects producing large numbers of unsorted cells, such as the Human Cell Atlas⁵⁵, the Human BioMolecular Atlas Program (HuBMAP)⁵⁶, the Precancer Atlas⁵⁷ and the Human Tumor Atlas Network⁵⁸. It will be beneficial in various scenarios including biomarker discovery and clinical assay designing. As a feature selection method tailored for biomedical data with complex manifolds, it can potentially be applied to non- single-cell data, for example, bulk RNA expression²⁹, copy number aberration, and genetic and drug screening data in large cohort studies such as TCGA and GTEx⁵⁹.

5. Methods

5.1. Cell-cell Similarity

SCMER is inspired by three methods: Stochastic Neighbor-Preserving Feature Selection (SNFS)⁶⁰, t-distributed stochastic neighbor embedding (t-SNE)²⁷ and Uniform Manifold Approximation and Projection (UMAP)²⁸.

t-SNE is one of the most widely used method for data embedding. For a dataset $\mathbf{\text{X}}\in {ℝ}^{n\times D}$ with n cells and D features, the similarity of a cell i to another cell j is defined as

$$p_{ij}=\frac{\exp \left(-{\Vert {\mathbf{\text{X}}}_i-{\mathbf{\text{X}}}_j\Vert }^2/2{\sigma }^2\right)}{{\sum }_{k\ne l}\exp \left(-{\Vert {\mathbf{\text{X}}}_k-{\mathbf{\text{X}}}_l\Vert }^2/2{\sigma }^2\right)},$$

which comprises a cell-cell similarity matrix $\mathbf{\text{P}}\in {ℝ}^{n\times n}$. σ is a scaling factor. It creates an d-dimensional embedding $\mathbf{\text{Y}}\in {ℝ}^{n\times d}$. It calculates another cell-cell similarity matrix $\mathbf{\text{Q}}\in {ℝ}^{n\times n}$ for $\mathbf{\text{Y}}$, whose entries are

$$q_{ij}=\frac{{\left(1+{\Vert {\mathbf{\text{Y}}}_i-{\mathbf{\text{Y}}}_j\Vert }^2\right)}^{-1}}{\sum _{k\ne l}{\left(1+{\Vert {\mathbf{\text{Y}}}_k-{\mathbf{\text{Y}}}_l\Vert }^2\right)}^{-1}}.$$

The cost function is defined as the Kullback-Leibler (KL) divergence of P and Q, formally

$$C=KL(\mathbf{\text{P}}\parallel \mathbf{\text{Q}})=\sum _i\sum _j{p}_{ij}\log \frac{p_{ij}}{q_{ij}}.$$

SNFS uses t-SNE formulation directly. Because emerging evidences show that UMAP is more sensitivity to both global relationship between cell groups and local relationship within cell groups²⁹, we borrowed a part of the UMAP formulation, i.e.,

$$p_{ij}=\frac{\exp \left(-\left(\Vert {\mathbf{\text{X}}}_i-{\mathbf{\text{X}}}_j\Vert -{\rho }_i\right)/{\sigma }_i\right)}{{\sum }_{k\ne l}\exp \left(-\left(\Vert {\mathbf{\text{X}}}_k-{\mathbf{\text{X}}}_l\Vert -{\rho }_i\right)/{\sigma }_i\right)},q_{ij}=\frac{{\left(1+\left(\Vert {\mathbf{\text{Y}}}_i-{\mathbf{\text{Y}}}_j\Vert -{\tau }_i\right)/{\sigma }_i\right)}^{-1}}{{\sum }_{k\ne l}{\left(1+\frac{\Vert {\mathbf{\text{Y}}}_k-{\mathbf{\text{Y}}}_l\Vert -{\tau }_i}{{\sigma }_i}\right)}^{-1}},$$

where ${\rho }_i=\min \Vert {\mathbf{\text{X}}}_i-{\mathbf{\text{X}}}_j\Vert$ and ${\tau }_i=\min \Vert {\mathbf{\text{Y}}}_i-{\mathbf{\text{Y}}}_j\Vert$. The scaling factor ${\sigma }_i$ is chosen such that ${\sum }_j\exp \left(-\left(\Vert {\mathbf{\text{X}}}_i-{\mathbf{\text{X}}}_j\Vert -{\rho }_i\right)/{\sigma }_i\right)={\log }_{2}k$, which may be viewed as constructing a soft nearest neighbor graph. We default it to 100 in our experiments. Similar to UMAP, setting it in the range 10 to 1,000 gives very similar results²⁸.

5.2. Marker Selection by Elastic Net

Different from t-SNE and UMAP, instead of allowing $\mathbf{\text{Y}}$ to be an arbitrary matrix, we require each column of $\mathbf{\text{Y}}$ to be directly taken from a column of $\mathbf{\text{X}}$, i.e., to select a feature. To formally model this procedure. We use a vector $\mathbf{\text{w}}\in {ℝ}^D$ (initialized as $w_i=1,\forall i$ in optimization) to indicate the selection of the features, where 0 means unselected, and set

$$\mathbf{\text{Y}}=\mathbf{\text{Xw}},$$

which set all unselected features to zero in $\mathbf{\text{Y}}$. In terms of calculating the distances, zeroing out the columns is effectively discarding them. Thus, the calculation of $\mathbf{\text{Q}}$ using $\mathbf{\text{Y}}$ is unchanged. Ideally, to select d features, we optimize

$$\underset{\mathbf{\text{w}}}{\min }C\text{ subject to }\parallel \mathbf{\text{w}}{\parallel }_0=d,$$

where $\parallel \mathbf{\text{w}}{\parallel }_0$ is the $l_0$-pseudo-norm, i.e., the number of nonzero entries. However, this question is known to be NP-hard, whose determination requires checking all the $\left(\begin{array}{l}D\hfill \\ d\hfill \end{array}\right)$ possibilities. Thus, we fall back to $l_1$-norm, the convex approximation of $l_0$-pseudo-norm, as in

$$\underset{\mathbf{\text{w}}}{\min }C+\lambda \parallel \mathbf{\text{w}}{\parallel }_1,$$

where $l_1$-norm $\parallel \mathbf{\text{w}}{\parallel }_1={\sum }_i\left|w_i\right|$ and $\lambda$ is the strength of the regularization. We denote the loss function as L. Because the number of chosen features decreases when $\lambda$ gets larger, for a given d, we use a binary search to find a $\lambda$. We used Orthant-wise limited memory quasi-Newton algorithm (OWL-QN, detailed below) to optimize $\mathbf{\text{w}}$. Due to limitations of precision, the specific d may not always be achievable. In that case, we allow for a few more features to be selected, and discard those that are assigned with the lowest weights (Supplementary Note 8). In the result, the features who have nonzero weights in $\mathbf{\text{w}}$ are considered selected. The specific weight is not used in downstream analysis.

The cost, $C=KL(\mathbf{\text{P}}\parallel \mathbf{\text{Q}})$, is a robust indicator of whether the manifold is successfully retained. A typical range of C is 2.0 – 4.0 when the manifold is reasonably retained. More features (i.e., smaller $l_1$-regularization) may be needed if the C is greater than 4.0.

Our model also allows an additional $l_2$-regularization (ridge) to form an elastic net model. It may improve the robustness of the panel by slightly increase the redundancy, so that noise or drop-out in one feature has less effects (Supplementary Note 9 and Supplementary Figure 21).

5.3. Batch Effect Correction by Stratification

Batch effect is a common problem in experiments including multiple samples. For SCMER, the samples are considered a stratum. In specific, a set of $\mathbf{\text{P}}$ and $\mathbf{\text{Q}}$ can be constructed for each sample, denoted as ${\mathbf{\text{P}}}^{(i)}$ and ${\mathbf{\text{Q}}}^{(i)}$, while $\mathbf{\text{w}}$ is shared by all samples. A cost $C^{(i)}$ can thus be calculated for each sample, and collectively form a new objective $C={\sum }_i{C}^{(i)}$. Thus, SCMER will ignore features that identify different samples and focuses on features that retain cell-cell similarities in all/most samples.

5.4. Supervised Multi-omics Mode

To transfer the manifold in one matrix ($\mathbf{\text{X}}$) to another (${\mathbf{\text{X}}}^{\prime }$), either between different modalities or subsets of features of the same modality, we simply modify the definition of $\mathbf{\text{Y}}$ to $\mathbf{\text{Y}}={\mathbf{\text{X}}}^{\prime }\mathbf{\text{w}}$. With all other procedures unchanged, the algorithm is now searching for features in ${\mathbf{\text{X}}}^{\prime }$ that gives a manifold similar to that of $\mathbf{\text{X}}$. This is also applicable to select features from a shortlist of the original ones.

5.5. Using Preselected features

In the case that a researcher wants to specify a few features that are known to be useful, we slightly modify the regularization to $\lambda \parallel \mathbf{\text{Vw}}{\parallel }_1$, where $\mathbf{\text{V}}=diag(\mathbf{\text{v}})$ is a diagonal matrix. If a feature is considered important a priori, the corresponding entry in $\mathbf{\text{v}}$ is set to 0 to avoid $l_1$-regularization. In this “softly-supervised” way, SCMER is more likely to select these features, but may still discard some of them if they are contradicting with the manifold. Thus, in addition, we provide a “hard-supervised” way where a set of features are guaranteed to be kept. Other features are selected to supplement them.

5.6. Orthant-Wise Limited Memory Quasi-Newton Algorithm

Limited-memory BFGS (L-BFGS) is an widely-used optimization algorithm in the quasi-Newton methods family⁶¹. It approximates the Broyden-Fletcher-Goldfarb-Shanno (BFGS) algorithm with 0(mD) memory, where m can be chosen based on computing resources.

Although L-BFGS usually converge very fast (<20 iterations) for most $l_2$-regularized regression problems, it will diverge for $l_1$-regularization, whose partial derivative is undefined at $\left\{\mathbf{\text{w}}\mid w_i=0\exists i\right\}$:

$$\frac{\partial \parallel \mathbf{\text{w}}{\parallel }_1}{\partial w_i}=\frac{\partial \sum _i\left|w_i\right|}{\partial w_i}=\frac{\partial \left|w_i\right|}{\partial w_i}=\{\begin{array}{cc}1& w_i>0\\ \text{ undefined }& w_i=0\\ -1& w_i<0\end{array}.$$

It should be noted that setting the undefined point to 0 (or any other value) at $w_i=0$ does not solve the problem as the discontinuity will also break L-BFGS. SNFS restricts $w_i\in [0,1]$ to avoid the discontinuity, but we find it having problems enforcing the sparsity. Instead, a modified version of L-BFGS called orthant-wise limited memory quasi-Newton (OWL-QN) algorithm³¹ is more suitable for this problem. A modified version of L-BFGS called orthant-wise limited memory quasi-Newton (OWL-QN) algorithm³¹ solves this problem by introducing pseudo-gradients and restrict the optimization to an orthant without discontinuities in the gradient.

In brief, we first derive the pseudo-gradient, where the pseudo-partial derivative at a discontinuity ${\mathbf{\text{w}}}_0$ of the loss function $L=C(\mathbf{\text{w}})+\lambda \parallel \mathbf{\text{w}}{\parallel }_1$ is defined as

$${\widehat{\partial }}_{i}L\left({\mathbf{\text{w}}}_0\right)=\{\begin{array}{cc}{\partial }_i^{-}L\left({\mathbf{\text{w}}}_0\right)& {\partial }_i^{-}L\left({\mathbf{\text{w}}}_0\right)>0\text{ and }{\partial }_i^{+}L\left({\mathbf{\text{w}}}_0\right)>0\\ {\partial }_i^{+}L\left({\mathbf{\text{w}}}_0\right)& {\partial }_i^{-}L\left({\mathbf{\text{w}}}_0\right)<0\text{ and }{\partial }_i^{+}L\left({\mathbf{\text{w}}}_0\right)<0\\ 0& \text{ otherwise }\end{array},$$

where ${\widehat{\partial }}_{i}L\left({\mathbf{\text{w}}}_0\right)$ is the pseudo partial derivative and ${\partial }_i^{-}L\left({\mathbf{\text{w}}}_0\right)$ is the short hand of ${\lim }_{w_i\to {\left({\mathbf{\text{w}}}_0\right)}_i}-{\frac{\partial L}{\partial w_i}|}_{w_k={\left({\mathbf{\text{w}}}_0\right)}_k}$, i.e, the left limit of the partial derivative. Similarly, ${\partial }_i^{+}L\left({\mathbf{\text{w}}}_0\right)$ is the right limit.

Note that the gradient of $C\left(\mathbf{\text{w}}\right)$ is continuous, i.e., ${\partial }_i^{-}C\left({\mathbf{\text{w}}}_0\right)={\partial }_i^{+}C\left({\mathbf{\text{w}}}_0\right)={\partial }_{i}C\left({\mathbf{\text{w}}}_0\right)$, and discontinuities of L are all at $\left\{\mathbf{\text{w}}\mid w_i=0\exists i\right\}$. Thus, the pseudo-gradient can be simplified to

$${\widehat{\partial }}_{i}L\left({\mathbf{\text{w}}}_0\right)=\{\begin{array}{cc}{\partial }_{i}C\left({\mathbf{\text{w}}}_0\right)-\lambda & {\partial }_{i}C\left({\mathbf{\text{w}}}_0\right)-\lambda >0\\ {\partial }_{i}C\left({\mathbf{\text{w}}}_0\right)+\lambda & {\partial }_{i}C\left({\mathbf{\text{w}}}_0\right)+\lambda <0\\ 0& -\lambda \le {\partial }_{i}C\left({\mathbf{\text{w}}}_0\right)\le \lambda \end{array}.$$

Then, we confine the search area in each quasi-Newton optimization step so that it does not cross any discontinuity. Specifically, for our problem where all discontinuities are at 0, when updating ${\mathbf{\text{w}}}^t$ to ${\mathbf{\text{w}}}^{t+1}$. we reset the value of $w_i^{t+1}$ to 0 if $sign\left(w_i^{t+1}\right)\ne sign\left(w_i^t\right)$. It constrains the optimization to be in the same “orthant” in each iteration.

L-BFGS optimizer is provided in PyTorch³⁰, in which SCMER is implemented. Based on it, we implemented a special case of OWL-QN algorithm for optimization of the model. Two modifications we made are as follows.

5.7. Data Preprocessing

For the melanoma data³⁴, which is TPM based, after removing ERCC spike-ins, we processed the data using the standard workflow of SCANPY⁶², including quality control (filtering out genes that are detected in less than 3 cells), normalization (10,000 reads per cell), log transformation, highly variable genes detection (with a loose threshold to filter out noisy genes; not to be confused with the DXG we compared with), and scaling.

For the Ileum Lamina Propria Immunocytes data³⁶, bone marrow data³⁷, and A549 data³⁸, which are UMI based, we used the standard workflow of SCANPY, including quality control (filtering out genes that are detected in less than 3 cells), normalization (10,000 reads per cell), log transformation, highly variable genes detection, and scaling. We used the stratified approach to suppress batch effect on the Ileum Lamina Propria Immunocytes data.

For protein data in CITE-Seq^39,40, we followed the preprocessing of protein data described in the original publication. For mRNA data in CITE-seq, we follow the standard workflow of SCANPY, as described above, except that we did not filter highly variable genes. We preprocessed protein data as mRNA data, without filtering highly variable genes.

5.8. Inference of TF Activities

Because TFs tend to bind at sites with cognate motifs, accessibility at peaks with the motifs reflects their activity. To estimate transcription factor activity from sci-ATAC-seq data, we use chromVAR⁴⁶ package with the default setting. It quantifies accessibility variation across single cells by aggregating accessible regions containing a specific TF motif. The observed accessibility of all peaks containing a TF motif is compared with a background set of peaks normalized for known technical confounders.

5.9. Comparison with Other Methods

To identify the highly expressed genes (HXG), we used the standard SCANPY⁶² workflow. HXG is defined by the total reads of a gene across all cells. To identify the highly variable genes, we followed the standard scoring method in SCANPY⁶².

SCMarker¹⁹ provides a gene list without ranks. It has two parameters, n and k, which affect the number of resulting features. Based on our observation, n has a minor effect on the result. Thus, we fixed n = 50 and tested k from 10 to 1,200 to create feature gene lists of various sizes.

We ran Monocle 2¹⁷ in unsupervised and supervised manners. For the supervised run, the labels were used directly. The trajectory was inferred using clusters/labels and pseudo-time is calculated. Genes were ranked by the degree they are explained by functions (which were fitted with cubic splines) of pseudo-time. For the unsupervised run, we clustered the cells and visually confirmed the clusters are concordance with the labels.

We ran RankCorr¹² in both supervised and unsupervised manner. For the supervised run, we used the label from the data directly. For the unsupervised run, we used the Leiden algorithm⁶³ for clustering which is the recommended method in SCANPY. Default parameters were used, and the clusters are visually checked that they are reasonable.

For random results, we randomly selected gene sets of given sizes. Reported are mean performance and the critical level of statistically significantly better (or worse) than random as defined by single-sample one-sided z-test at 5% significance level.

Data Availability

All original datasets are accessible through the original publications^34–41, including the melanoma data (GSE72056), pan-cancer cell line data (https://singlecell.broadinstitute.org/singlecell/study/SCP542 ), immune cell subtypes data (https://singlecell.broadinstitute.org/singlecell/study/SCP359 ), hematopoiesis data (GSE116256), A549 data (GSE128639), CITE-seq data (GSE128639 and GSE100866), and CyTOF data (https://cytobank.org/nolanlab/reports/Levine2015.html). Source Data for Figures 1–5 are available with this manuscript.