Directly selecting cell-type marker genes for single-cell clustering analyses

Summary

In single-cell RNA sequencing (scRNA-seq) studies, cell types and their marker genes are often identified by clustering and differentially expressed gene (DEG) analysis. A common practice is to select genes using surrogate criteria such as variance and deviance, then cluster them using selected genes and detect markers by DEG analysis assuming known cell types. The surrogate criteria can miss important genes or select unimportant genes, while DEG analysis has the selection-bias problem. We present Festem, a statistical method for the direct selection of cell-type markers for downstream clustering. Festem distinguishes marker genes with heterogeneous distribution across cells that are cluster informative. Simulation and scRNA-seq applications demonstrate that Festem can sensitively select markers with high precision and enables the identification of cell types often missed by other methods. In a large intrahepatic cholangiocarcinoma dataset, we identify diverse CD8⁺ T cell types and potential prognostic marker genes.

Graphical abstract

Highlights

• •Festem enables direct selection of cluster-informative marker genes from scRNA-seq
• •Festem effectively controls false discovery rates for marker identification
• •Festem improves clustering and identifies often-missed cell types
• •Festem is applied to analyze immune cell subsets in intrahepatic cholangiocarcinoma

Motivation

A fundamental problem in single-cell RNA sequencing (scRNA-seq) studies is identifying cell types and their associated marker genes using clustering and differential expression analysis between clusters. Many sequenced genes are cell-type irrelevant and significantly influence cell-type identification. Ideally, one should select marker genes for the best cell-type identification. However, because cell types are unknown, directly selecting marker genes seems impractical, and surrogate criteria, such as variance, deviance, and zero proportions, are used for gene selection. The surrogate criteria can miss important genes or select unimportant genes, leaving potentially relevant cell types unidentified. We aim to develop a method that can directly select marker genes with high accuracy and significantly improve cell-type identification.

Chen et al. develop Festem (feature selection by expectation maximization test), a method that enables direct cell-type marker gene selection for downstream clustering of scRNA-seq data. Festem outperforms existing methods in marker gene identification and improves clustering accuracy. Application to intrahepatic cholangiocarcinoma scRNA-seq data identifies potential prognostic biomarkers.

Introduction

Single-cell RNA sequencing (scRNA-seq) technologies have been widely used in many different areas to study the cellular states, transcriptomic dynamic changes, and intercellular communications in healthy or diseased tissues.^1,2,3,4 One of the fundamental problems in scRNA-seq analysis is to identify cell types and their associated marker genes. In a standard protocol, cell types are first identified by clustering single cells based on a set of pre-selected genes. Then, assuming that the cell types derived from clustering are known, marker genes are detected by performing differentially expressed gene (DEG) analysis between the cell types. Although widely used, this protocol has two important issues to be resolved.

The first issue involves the selection of genes for clustering analysis. Current scRNA-seq technologies can profile ∼10,000 or more genes for thousands of single cells in one experiment. However, not all profiled genes contain information about cell types. Many genes have the same or very similar expressions in different cell types. Inclusion of these uninformative genes in the clustering can seriously influence the clustering accuracy and unnecessarily increase the computational burden. Currently, genes used for clustering are often chosen as the highly variable genes (HVGs) selected by methods such as HVGvst.⁵ However, HVGs are not always clustering informative (Figure S1A), and vice versa, clustering-informative genes may not be HVGs (Figure S1B). Important cell types can thus be difficult distinguish by the HVGs. Other gene selection methods^6,7,8 have been developed. However, similar to HVGs, instead of directly selecting clustering-informative genes, they all use surrogate metrics and thus have the similar problem as HVGs.

The second issue is the so-called “double-dipping”^9,10 or “selection-bias”¹¹ problem. The expression data are first used for clustering and then reused for DEG detection to find marker genes of the obtained clusters using available methods such as DESeq2¹² and EdgeR.¹³ The uncertainty in the clustering is often ignored in the DEG detection step. However, single cells can be misclustered, especially for cell types with similar expression patterns. The noisy cluster labels can lead to decreased precision and increased false discoveries in marker gene detection.¹⁴ Recently, a few novel tools, such as the truncated normal (TN) test¹¹ and singleCellHaystack,¹⁴ have been developed to address this problem. However, partly because these methods are often ad hoc and lack statistical theories for their statistical significance assessment, they still tend to report many false discoveries.

To address these two computational issues, we propose Festem (feature selection by expectation maximization [EM] test) that can accurately select clustering-informative genes before the clustering analysis and identify marker genes. We observe that the gene selection and marker gene detection problems are closely related. Ideally, a gene selection procedure should select all marker genes for downstream clustering analyses because marker genes are cluster informative and non-marker genes are not, and selecting marker genes should give the most accurate clustering and cell-type identification. Marker genes follow different distributions in different cell types (heterogeneous distribution) and are clustering informative. Non-marker genes follow the same distribution among different cell types (homogeneous distribution) and are clustering irrelevant. Festem selects the clustering-informative marker genes by performing a statistical test to determine if a gene is homogenously or heterogeneously distributed. Mathematical analysis shows that the homogeneity test asymptotically follows a distribution that can be bounded by a chi-squared distribution,¹⁵ which thus allows Festem to effectively control the false discovery rate (FDR) for marker gene detections. Extensive simulation and real data analyses reveal that clustering can be significantly improved based on Festem-selected genes, and Festem can sensitively detect marker genes with high precision. Finally, in an application to a large scRNA-seq dataset (122,329 cells) of intrahepatic cholangiocarcinoma (iCCA), Festem identifies two subtypes of CD8 T cells and their associated marker genes that may serve as important prognostic biomarkers of patients with iCCA.

Results

The Festem model and algorithm

For each gene, Festem performs a statistical test to determine if its expression is homogenously distributed (not a marker gene) or heterogeneously distributed (a marker gene) and assigns a p value based on the chi-squared distribution ${\chi }_3^2$ (Figure 1). The p values are adjusted for multiple comparisons using the Benjamini-Hochberg procedure.¹⁶ Genes are ordered by the adjusted p values, and the top genes are selected for clustering analysis. Users can then use these genes to cluster the single cells with available methods such as kmeans¹⁷ or the Louvain algorithm.¹⁸ Genes with significant adjusted p values are reported as marker genes. Given cluster labels, Festem then assigns the reported genes to the clusters as their markers by the Scott-Knott test¹⁹ (Figure 1).

Figure 1.

Overview of Festem

Single-cells are first roughly clustered to obtain a prior for the cell type compositions (optional). Then, Festem performs a hypothesis test to determine if genes are clustering informative and assigns p-values to genes based on the chi-square distribution ${\chi }_3^2$ (bottom panel). Priors can be incorporated in the EM-test. Genes are ordered by the adjusted p-values and top genes are selected as DEGs and used for clustering analysis. After clustering, Festem assigns DEGs to clusters as their markers by the Scott-Knott test. See also Figure S1.

Since the cell-type labels are unknown, we cannot directly compare the distributions of gene expression in different cell types to test whether a gene is homogenously distributed. However, if a gene is homogenously distributed, then its expression distributions in different cell types are the same and identical to its marginal distribution. On the other hand, if a gene is heterogeneously distributed, then it will have different distributions in at least two cell types, and its marginal distribution will be a mixture distribution (Figure 1). Therefore, homogeneity can be tested by testing whether the distribution is a mixture distribution.

We model the expression of a gene in a cell type following a negative binomial distribution.²⁰ Thus, if a gene is homogenously distributed, then its marginal distribution is also a negative binomial distribution. If a gene is heterogeneously distributed, then its marginal distribution is a mixture of negative binomial distributions (Figure 1). Festem tests the homogeneity using an EM test for negative binomial distributions (STAR Methods), which can be viewed as a generalized version of the EM test for normal distributions.²¹ The EM test is very similar to log likelihood ratio tests, but it avoids the pathogenicity of log likelihood ratio tests for the mixture models.²¹ We develop a computational efficient algorithm to calculate the EM test statistic (STAR Methods). It can be proved mathematically that the EM test statistic is asymptotically less than a ${\chi }_3^2$ random variable¹⁵ (STAR Methods). Thus, the p value obtained from ${\chi }_3^2$ is conservative (i.e., the type I error can be effectively controlled), which is also validated by our simulation and real data analyses. Festem can also incorporate prior information, which is useful for down-weighting heterogeneously distributed genes due to factors other than cell types, such as cell cycles.²² The prior information can be very rough and may be chosen as the clustering based on the HVGs.

Festem enables accurate clustering and marker gene detection in simulation

We used simulation to evaluate the performance of Festem. In the simulation, we considered 4 different scenarios, including 2 different numbers of cell types (2 and 5 types) in the datasets and 2 different noise levels (low and high). For each scenario, we generated 20 simulation datasets, each of which had 3,000 cells and 20,000 genes (STAR Methods). We also varied the tuning parameter $G$ (the number of components in the mixture distribution) to study its influence on the performance of Festem.

We first compared the clustering accuracies based on genes selected by Festem and other widely used methods including HVGvst,⁵ HVGdisp,²³ DUBStepR,²⁴ devianceFS,⁸ and TrendVar.⁷ For all methods, we first selected their top 500 genes and computed the first 10 principle components, then used the Louvain algorithm¹⁸ to cluster the single cells. The adjusted Rand index and the silhouette index were calculated to evaluate the clustering results given by different algorithms. Festem performed very well compared with other methods (Figures 2A–2D and S2A–S2D), especially in the 5-cell-type and high-noise case (Figures 2A–2D). The reason for this was that the available methods used surrogate metrics such as variances or deviances to select clustering-informative genes. When the noise in the data is high, many genes selected by these methods just had high variances or deviances and did not contain clustering information. In comparison, Festem directly evaluated the clustering information contained in the genes, and the clustering-relevant genes were always more likely to be selected, regardless of the noise level.

Figure 2.

Performance of Festem on clustering and DEG analyses in simulation

Top and bottom images correspond to simulations with low noise (400 DEGs) and high noise (200 DEGs), respectively.

(A and B) Typical UMAP plots based on genes selected by feature selection method. Colors represent different cell types.

(C and D) Adjusted Rand indices (ARIs) and silhouette indices (Sis) of the clustering results based on genes selected by different feature selection methods.

(E and F) The precision and recall of DEG detection methods. The red lines are the nominal FDR level, 0.05. The gray dashed lines are the contour lines with constant F1-scores, and the numbers next to the gray lines in (E) are the corresponding F1-scores.

See also Figures S2 and S3.

Then, we assessed the performance of Festem on marker gene detection and compared it with several popular and top-performing DEG methods in previous benchmark studies,^25,26 including DESeq2,¹² EdgeR,¹³ MAST,²⁷ the Wilcoxon rank-sum test,²⁸ ROSeq,²⁹ singleCellHaystack,¹⁴ and the TN test.¹¹ Festem and singleCellHaystack could detect marker genes without pre-clustering of the single cells. For other methods, we first pre-clustered the single cells based on HVGs and provided the clustering results to these methods for DEG detection. Festem was able to effectively control the FDR (1 - precision) and achieved reasonably good sensitivity (Figures 2E, 2F, S2E, S2F, and S3A–S3H). Due to the double-dipping problem, most other methods could not control the FDR, and their FDRs were often much larger than the nominal level, 0.05 (the target FDR level). Although singleCellHaystack was clustering independent, it could not effectively control the FDR, possibly because of its ad hoc method of assessing statistical significance. ROSeq was largely able to control the FDR but had lower power than Festem. We also generated datasets based on scDesign2³⁰ and scDesign3,³¹ two popular scRNA-seq simulators, and found that Festem achieved consistently good performance, including scenarios with continuous trajectories (Figures S2G–S2M).

Finally, we investigated the robustness and computational efficiency of Festem (Figure S3). Overall, Festem was robust to the choice of the parameter $G$. Different choices of $G$ gave very similar performances for both clustering (Figures S3I, S3J, S3M, and S3N) and marker gene detection (Figures S3A, S3B, S3E, and S3F). In terms of computational efficiency, Festem was computationally less efficient than many gene selection methods (Figures S3K, S3L, S3O, and S3P) because these methods only need to calculate simple statistics such as variance and deviance, while Festem need to use the EM algorithm to calculate the EM test statistic. However, compared with the DEG methods, Festem was often computationally more efficient, especially in terms of memory usage (Figures S3C, S3D, S3G, and S3H). Considering that Festem could simultaneously perform gene selection and report marker genes, we concluded that Festem was comparable with other methods in computational efficiency.

Festem had a high precision and recall in marker gene detection

We evaluated the performance of Festem on marker gene detection using four scRNA-seq datasets, a human peripheral blood mononuclear cell (PBMC) dataset (the PBMC3K dataset, 2,622 cells), an immune cells dataset from eight patients with lupus³² (the Kang dataset, 6,410 cells) and two batches of PBMCs profiled using 3′ or 5′ 10× Genomics protocols³³ (the Zheng datasets, 8,381 and 7,726 cells, respectively). To compare different algorithms, we first constructed silver standards of marker genes and non-marker genes. Since housekeeping genes were usually expected to have similar expression levels in different cell types,^34,35 we defined true non-marker genes as housekeeping genes with small Moran’s indices in the 2-dimensional uniform maniform approximation and projection (UMAP) embedding and true marker genes as non-housekeeping genes with large Moran’s indices (STAR Methods).

With these silver standards, we assessed the precision and recall of different marker gene detection methods (Figure 3A). Compared with other methods, Festem could achieve high precision (low FDR) and high recall, thus having high overall F1-scores. Festem was also robust to the choice of the cluster number G (Figure S4A). In comparison, most other methods either had low precision (high FDRs) or low recall. Visual inspection showed that their reported false marker genes (housekeeping genes in the silver standard) indeed had no expression preference toward any cluster on the UMAP plot (Figures S4B and S4C). With its default settings, the popular DEG method EdgeR had high precision but low recall. If we turned off its ad hoc filters and only used corrected p values for DEG detection, then EdgeR’s recall achieved 1, but its precision became much smaller (Figure S4A). The other popular method DESeq2 tended to be conservative and failed due to a memory limit in the two batches of Zheng data. Because of the inaccurate clustering, the classical non-parametric Wilcoxon’s test also had low precision. Since singleCellHaystack did not depend on clustering, singleCellHaystack performed better than other available methods. Note that the way that we built our silver standards was similar to the way that singleCellHaystack detected marker genes. Both methods relied on analyzing gene expression patterns in the 2-dimensional UMAP embedding. Thus, the evaluation might be in favor of singleCellHaystack. Methods like ROSeq, TN test, and MAST had very high recall levels but low precision levels. Moreover, we also assessed precision and recall based on marker genes from CellMarker2.0.³⁶ Festem also achieves the largest average F1-score (Figure S5).

Figure 3.

Performance of Festem on clustering and DEG analyses in real data analyses

(A) Precision-recall plots of DEGs reported by each method under the nominal FDR level, 0.05 (red dashed lines). The gray dashed lines are the contour lines with constant F1-scores (the numbers in the top-left image). The inset bar plots are the numbers of DEGs reported by each method. DESeq2 failed on the two Zheng datasets due to being out of memory and thus is not shown. ROSeq reported no DEGs for the Zheng dataset (5′ seq) and thus was excluded from the plot.

(B) Calinski-Harabasz (CH) index and variation of information of feature selection methods. For each dataset, CH indices were computed based on the corresponding UMAP coordinates of each method. The variation of information was computed between the annotation and the clustering result based on the genes selected by each method.

See also Figures S4 and S5.

Festem-selected genes improve clustering

We then compared the clustering results based on Festem-selected genes with genes selected by other available methods. Due to the lack of a gold standard of cell types, we employed the commonly used Calinski-Harabasz (CH) indices³⁷ to compare different clustering results. Larger values of the CH index corresponded to better separated clusters. Besides, we also employed CellTypist³⁸ to annotate each single cell and computed the variation of information³⁹ between the annotation and different clustering results (STAR Methods). A smaller variation of information represents a higher level of consistency with the annotation. Among all methods, Festem gave the largest CH index in three of the four benchmark datasets and the second largest in the other dataset, as well as a relatively small variation of information (Figure 3B), indicating that Festem-selected genes could produce consistently good clustering results compared with other methods. Note that a few cell types presented in the four benchmark datasets were not recorded in CellTypist, so it may underestimate the performance of Festem in terms of the variation of information. DUBStepR was not considered in this comparison because it aimed at selecting the minimum number of non-redundant genes and thus reported much fewer genes than other methods.

We further drew the receiver operating characteristic (ROC) curve of each method based on the silver standards constructed in the previous section and calculated the area under the ROC curve (AUROC) to quantify the quality of the selected genes (Figure S4D). In the four datasets, Festem always had a high AUROC (mean = 0.97), indicating that Festem stably selected most of the markers or clustering-informative genes. In comparison, the AUROCs of other methods either varied greatly among different datasets or were consistently lower than that of Festem. For example, the AUROCs of HVGdisp were slightly larger than those of Festem in two of the four datasets (∼0.98) but much smaller in the other two datasets (<0.89). Similarly, one of devianceFS’s AUROCs was slightly larger than that of Festem, but the other three AUROCs of devianceFS were much smaller. These results suggested that the superior clustering results given by Festem probably resulted from its high-quality marker gene selection.

Festem enabled identification of often-missed fine cell types

We further investigated the cell types identified in the PBMC3K and Kang datasets based on genes selected by Festem and other available gene selection methods. We used the Louvain algorithm¹⁸ to cluster the single cells based on genes selected by different methods (STAR Methods).

Using the Festem-selected genes, we identified 10 clusters in the PBMC3K dataset and annotated them based on the expression of canonical markers. The 10 clusters included immune cells such as naive CD4 T cells, memory CD4 T cells, CD8 T cells, CD14 monocytes, FCGR3A monocytes, natural killer (NK) cells, B cells, and dendritic cells (DCs)^3,40,41,42 (Figures 4A and S6A). In addition to these common cell types, Festem identified a fine cell type, CD27⁻CD4⁺ memory T cells, which were often missed by other methods (Figure 4A).

Figure 4.

Festem identifies fine cell types in the PBMC3K and Kang datasets

(A) The UMAP plot of the PBMC3K dataset constructed from genes selected by Festem. Left: all cell types identified by Festem. Right: the C5 or C9 clusters by different methods (highlighted dots). These clusters are the ones identified by other methods that contain the most cells of the fine cell type (CD27⁻CD4⁺ memory T) identified by Festem.

(B) Violin plots of CD27⁻CD4⁺ memory T cell markers. The CD27⁻CD4⁺ memory T cells identified by Festem and the C9 clusters identified by devianceFS, trendVar, and DUBStepR are shown. The C5 clusters identified by HVGvst and HVGdisp are clearly mainly CD8 T cells, and thus their violin plots are not shown.

(C and D) Sankey plots for the clustering results between Festem and devianceFS or between Festem and HVGvst.

(E) The t-distributed stochastic neighbor embedding (tSNE) plot of the Kang dataset constructed from genes selected by Festem. The two fine cell types often missed by other methods are highlighted in red circles.

(F) Expressions of IFN-related genes in IFN^hi CD14⁺ monocytes and CD14 monocytes identified by Festem.

(G) Expressions of heat shock protein-related genes in different cell types. The annotation is based on the clustering results of Festem.

See also Figures S6–S9.

The CD27⁻CD4⁺ memory T cells identified by Festem expressed common marker genes (IL7R and S100A4) of memory T cells but did not express CD27. These cells also had downregulated expression of SELL, CCR7, MAL and LEF1 and upregulated expression of CCL5 (Figure 4B) and thus were the CD27⁻ CD4⁺ memory T cells in the literature.⁴³ The CD27⁻CD4⁺ memory T cells were known to be at a more differentiated state and have stronger antigen-recall responses than their CD27⁺ counterparts. These results were further validated using another PBMC5K dataset from 10× Genomics (Figure S7).

The other available methods could not correctly cluster the CD27⁻CD4⁺ cells. For example, most of the CD27⁻CD4⁺ cells of Festem were assigned to the C9 cluster of devianceFS (Figures 4A and 4C). However, many cells in the C9 cluster of devianceFS expressed CD27 and SELL or did not express CCL5 (Figure 4B), implying that these cells were more likely to be naive CD4 T cells or memory CD4 T cells instead of CD27⁻CD4⁺ memory T cells (Figures 4C and S6B). Similarly, in the clustering of HVGvst, the CD27⁻CD4⁺ memory T cells were not clustered together but rather separately assigned to other cell types such as naive CD4 T cells and CD8 T cells (Figure 4D). One possible reason that these methods did not correctly identify the CD27⁻CD4⁺ memory T cells is that some markers, such as CD27 and SELL, were not selected as clustering-informative genes.

For the Kang dataset,³² we obtained 16 cell types based on the Festem-selected genes (Figures 4E and S8A), including two fine cell types that were often missed by other methods. The two cell types were interferon (IFN)^hi CD14⁺ monocytes with high expression of type I IFN-related genes (Figures 4E and 4F) and heat shock protein (HSP)⁺ CD4 T cells with upregulated HSP-related genes (Figures 4E and 4G). The HSP⁺ CD4 T cells might result from the cryopreservation and thawing of T cells.⁴⁴ Except for devianceFS, the other available methods could not identify these HSP⁺ CD4 T cells (Figure S8B).

It was found that genes in IFN pathways were often upregulated in PBMCs of lupus and associated with the severity of lupus,⁴⁵ and their upregulation was mainly in monocytes and lymphocytes.⁴⁶ IFN^hi CD14⁺ monocytes identified by Festem were thus probably the subgroups of monocytes in patients with lupus with upregulated IFN genes. The Kang dataset (the control data) had paired IFN-stimulated single-cell data. In these IFN-stimulated cells, almost all CD14⁺ monocytes had a high IFN expression score (Figures S8C and S8D). Joint analysis showed that IFN^hi CD14⁺ monocytes from the control and IFN-stimulated groups mixed together (Figures S8E and S8F), indicating that they were the same type of monocytes that responded to type I IFN. DUBStepR and devianceFS could not distinguish the IFN^hi CD14⁺ monocytes, and they mixed these monocytes with other types of monocytes (Figure S8B) that did not express IFN-related genes (Figure S9).

Festem identified prognostic biomarkers for iCCA

We applied Festem to analyze 122,329 cells from 14 patients with iCCA.² We obtained 21 clusters including several subtypes of T cells, NK cells, macrophages, DCs, B cells, fibroblasts, and endothelial cells (Figures 5A and S10A). Among them, CD8 T cells had the most diverse subtypes, including terminally differentiated effector memory or effector T cells (T_emra cells), terminal exhausted T cells (terminal T_ex cells), granzyme K-positive (GZMK+) effector memory T cells (GZMK T_em cells), and IFN-stimulated gene (ISG)-positive T-like cells (ISG⁺ CD8 T-like cells)⁴⁷ (Figure 5B). In addition to known markers of T_emra cells and terminal T_ex cells (Figure 5B), Festem also identified several markers of terminal T_ex cells in iCCA, such as FAM3C and DUSP4, for which we could not find previous evidence in the literature. Except for devianceFS, all other available methods could not identify T_emra and terminal T_ex cells simultaneously (Figure S10B). devianceFS was able to identify the two CD8 cell types but also assigned many cells, possibly of other cell types, to the two CD8 cell types (Figure S10C), leading to a decreased fold change of marker genes of these two cell types (Figure S10D).

Figure 5.

Festem identifies clinically relevant markers in the iCCA dataset

(A) The UMAP plot constructed from genes selected by Festem. The Festem-identified cell types are shown in different colors.

(B) Violin plots of T_emra and terminal T_ex cell markers.

(C) The Kaplan-Meier curves of the four groups of patients defined by the expression of markers of T_emra and terminal T_ex cells in two independent datasets. p values given by the log-rank test are shown in the plots.

See also Figures S10 and S11.

We observed that the proportions of T_emra and terminal T_ex cells varied greatly among different patients (Figure S11A) and hypothesized that enrichment of T_emra and terminal T_ex cells might be predictive of a patient’s survival. We therefore investigated two public iCCA datasets^48,49 with bulk RNA-seq data and patients’ outcome data. Using markers of T_emra (CX3CR1) and terminal T_ex (FAM3C and DUSP4) cells, we grouped the patients into 4 groups, emra^highex^low, emra^highex^high, emra^lowex^low, and emra^lowex^high (STAR Methods), and found that the overall survival of patients in these groups was highly significantly different for both iCCA datasets (Figure 5C, log-rank test). As expected, the emra^highex^low group had the best survival and the emra^lowex^high group had the worst survival in the 4 groups. Furthermore, using spatial transcriptomics data of hepatocellular carcinoma,⁵⁰ we found that T_emra cells tended to be colocalized with fibroblasts in some patients (Figure S11B). Considering that cancer-associated fibroblasts were associated with poor prognosis,⁵¹ we sought to study whether the two emra^high groups could be further decomposed into subgroups with distinct survival. Using the marker gene POSTN of fibroblasts identified by Festem, patients enriched with fibroblasts (POSTN^high) indeed had significantly worse survival than other patients in both emra^highex^low and emra^highex^high iCCAs (Figure S11C).

Discussion

Gene selection and marker gene identification are the two most important and commonly used analyses in single-cell studies. The two problems are closely related, and selecting cell-type marker genes should give the most accurate clustering and cell-type identification. However, because cell types are unknown, genes are often selected using criteria that may not be directly related to clustering, and marker genes are identified by DEG analysis, assuming that the cell types are known. This standard protocol could miss important genes, leave important cell types unidentified, and give many false marker genes. In this paper, we present a unified framework, Festem, for gene selection and marker gene identification by investigating whether a gene has a homogeneous or heterogeneous distribution. Extensive simulation and real data analyses demonstrate that Fetem can improve clustering and sensitively detect DEGs with low false discoveries. Further theoretical analyses also guarantee that Festem can correctly select clustering-informative genes and control the FDR of marker gene detection.¹⁵

Festem has a tuning parameter, the number of components $G$ in the mixture distribution. In practice, the true number of cell types, $G^{\ast }$, is usually unknown, so this parameter $G$ serves as a “working” cell-type number. However, $G$ does not need to be equal to $G^{\ast }\text{.}$ Theoretically, p values given by Festem are always valid (that is, type I error can be effectively controlled) even if $G$ is different from $G^{\ast }\text{.}$¹⁵ Different choices of $G$ only influence the power of Festem. Choosing a $G$ smaller than $G^{\ast }$ could make Festem more conservative (smaller power), and choosing a $G$ larger than $G^{\ast }$ has little influence on the power. Further, our empirical studies show that Festem is largely robust to the choice of $G$ (Figures S3 and S4A). In practice, we often can have a rough estimate of $G^{\ast }$ and could choose $G$ to be slightly larger than the rough estimate.

In the benchmarking analyses using real scRNA-seq data, the FDRs of Festem are slightly larger than the target level (Figure 3). The reason is that Festem detected genes that are unrelated to cell types but to other factors such as cell cycles. For example, HMGB2, MAP1LC3B, and NDUFA2 were non-marker genes in our silver standard (Figure S6C) but were reported as marker genes by Festem. HMGB2 was associated with the G2M phase,⁵² and its expression was significantly different in different cell cycle stages (Figure S6C), and MAP1LC3B and NDUFA2 were identified as differentially expressed between different cell cycle stages by Seurat’s⁵³ function CellCycleScoring. Marginally, these genes were also heterogeneously distributed, but the latent clusters were not cell types but cell cycle stages. Although we used pre-clustering priors to down-weight heterogeneously distributed genes unrelated to cell types, some still had highly significant p values and were reported as marker genes. These false positives could be removed by comparing them with a prior gene list that contains known genes related to cell cycles.

In the real data benchmarking studies, singleCellHaystack performed very well and gave the highest F1-scores in some cases. This might be attributed to the similarity between our approach to constructing silver standards and singleCellHaystack’s method for marker gene detection. singleCellHaystack first conducts dimension reduction using genes selected by HVGvst and subsequently identifies marker genes by investigating the Kullback-Leibler divergence between the distributions of cells and gene expressions in the low-dimensional space. In the construction of silver standards, we also utilized the low-dimensional embedding based on HVGvst-selected genes as a filtering criterion. Thus, even though singleCellHaystack performed very well as measured using these silver standards, it could not identify the CD27⁻CD4⁺ memory T cells, even when using a larger resolution of the Louvain clustering algorithm (Figure S6D). On the other hand, singleCellHaystack reported almost twice as many marker genes as Festem in the four benchmarking datasets (Figure S4A), some of which were clearly false positives (Figures S4B and S4C). Its potential high false positive rate might make fine cell types such as the CD27⁻CD4⁺ memory T cells more difficult to detect. Our simulation studies also demonstrated that singleCellHaystack had high false positive rates. Moreover, singleCellHaystack highly relies on the input dimension reduction, while Festem is robust to different pre-clustering labels (Figure S4A).

In the analysis of the PBMC3K dataset, we found an often-missed fine cell type, CD27⁻CD4⁺ memory T cells. This finding was validated with another PBMC5K dataset from 10× Genomics (Figure S7). Besides, we identified two markers of the CD27⁻CD4⁺ memory T cells, NKG7 and S100A11, for which we could not find previous evidence in the literature. They show significant upregulated and downregulated patterns in the CD27⁻CD4⁺ memory T cells in both PBMC3K and PBMC5K datasets. Recent studies suggest that NKG7 is essential for the activation and proinflammatory responses of CD4⁺ T cells,⁵⁴ and S100A11 was highly expressed in older CD4 T cells than their younger counterparts.⁵⁵ Hence, our newly found marker may help further characterize the CD27⁻CD4⁺ memory T cells.

By analyzing a large iCCA scRNA-seq dataset, we identified two groups of important T cells, T_emra and terminal T_ex cells. The former exhibits high killing activity, while the latter has reduced cytotoxicity.⁵⁶ We found that their marker genes, CX3CR1, FAM3C, and DUSP4, were predictive of survival in patients with iCCA. These genes have been previously reported as biomarkers in several other cancer types,^57,58,59 further supporting their prognostic values. On the other hand, it is worth noting that these markers may not only express in T_emra and terminal T_ex cells. For example, CX3CR1 and DUSP4 are reported to be expressed in some NK cells.^60,61 Hence, the prognostic values of these genes might be an aggregated effect of several immune cell types involved in cancer responses. In addition, this iCCA dataset contains over 120,000 cells. Festem was able to finish the analyses in 27 min with 20 cores, demonstrating that Festem scales well to very large datasets.

Limitations of the study

One limitation of Festem is that although it can efficiently find out whether a gene is a marker gene, it cannot attribute the detected markers to specific cell types. So, after clustering with the selected marker genes, we have to assign these markers to different clusters as cluster markers. This step does not involve testing whether a gene is a marker but just assigns the cluster markers. In this step, by default, we used the Scott-Knott test, a clustering algorithm originally proposed as a post hoc procedure in ANOVA for grouping the means in ANOVA. Here, the data reuse could inflate the p values of the statistical tests in the Scott-Knott test, resulting in homogeneous clusters not being grouped together. This might lead to false negative assignments of the markers (a marker should have been assigned to a cluster but was not assigned) but does not affect the statistical validity that the gene is a marker. Thus, in terms of overall maker gene discovery, Festem avoids double-dipping or selection bias. On the other hand, the traditional cluster and DEG detection methods cannot guarantee FDR control in marker gene discovery due to double-dipping. In addition, the Scott-Knott test is not the only way to assign markers. For example, we may simply assign markers to the clusters exhibiting the highest expression.

Another limitation is that Festem costs more computational time compared with many gene selection methods due to the use of the EM iterations. However, on the other hand, Festem is comparable or faster than many popular DEG identification methods, such as DESeq2, edgeR, MAST, and the Wilcoxon rank-sum test. Since Festem can simultaneously perform gene selection and report cell-type markers, we believe that the computational time Festem needs is largely acceptable.

STAR★Methods

Key resources table

REAGENT or RESOURCE	SOURCE	IDENTIFIER
Deposited data
3k PBMCs from a Healthy Donor (Single Cell Gene Expression Dataset by Cell Ranger 1.1.0)	10X Genomics	https://support.10xgenomics.com/single-cell-gene-expression/datasets/1.1.0/pbmc3k/
PBMCs profiled using 3′ and 5′ 10x Genomics protocols	Zheng et al.33	https://hub.docker.com/r/jinmiaochenlab/batch-effect-removal-benchmarking
Immune cells gene expressions from eight lupus patients	Kang et al.32	GEO: GSE96583
Single-cell gene expression from 14 intrahepatic cholangiocarcinoma (iCCA) patients	Song et al.2	GSA: HRA000863
Spatial transcriptomes from hepatocellular carcinoma patients	Wu et al.50	GSA: HRA000437
Housekeeping gene list	HRT Atlas v1.0 database35	http://www.housekeeping.unicamp.br/.
Software and algorithms
Festem v1.0.0	This paper	https://doi.org/10.5281/zenodo.11331165
R v4.0.5	R Core Team	https://www.r-project.org/
Python v3.9.10	Van Rossum and Drake, 2009	https://www.python.org
Seurat v4.0.1	Hao et al.53	https://github.com/satijalab/seurat/releases
scran v1.18.7	Lun et al.7	https://bioconductor.org/packages/release/bioc/html/scran.html
scry v1.2.0	Townes et al.8	https://www.bioconductor.org/packages/release/bioc/html/scry.html
DUBStepR v1.2.0	Ranjan et al.24	https://github.com/prabhakarlab/DUBStepR
DESeq2 v1.30.1	Love et al.12	https://bioconductor.org/packages/release/bioc/html/DESeq2.html
EdgeR v3.32.1	Robinson et al.13	https://bioconductor.org/packages/release/bioc/html/edgeR.html
MAST v1.16.0	Finak et al.27	https://bioconductor.org/packages/release/bioc/html/MAST.html
ROSeq v1.2.10	Gupta et al.29	https://bioconductor.org/packages/release/bioc/html/ROSeq.html
singleCellHaystack v0.3.4	Vandenbon et al.14	https://github.com/alexisvdb/singleCellHaystack/tree/binary?tab=readme-ov-file
truncated_normal v0.4	Zhang et al.11	https://github.com/jessemzhang/tn_test
scDesign2	Sun et al.30	https://github.com/JSB-UCLA/scDesign2
scDesign3	Song et al.31	https://github.com/SONGDONGYUAN1994/scDesign3
ScottKnott v1.3-1	Jelihovschi et al.62	https://cran.r-project.org/web/packages/ScottKnott/index.html
CellTypist v1.6.2	Domínguez Conde et al.38	https://pypi.org/project/celltypist/
harmony v1.2.0	Korsunsky et al.63	https://cran.r-project.org/web/packages/harmony/index.html
survival v3.5-3	Therneau et al.	https://cran.r-project.org/web/packages/survival/index.html
survminer v0.4.9	Kassambara et al.	https://cran.r-project.org/web/packages/survminer/index.html
Monocle2 v2.26.0	Qiu et al.64	https://bioconductor.org/packages/release/bioc/html/monocle.html
slingshot v2.6.0	Street et al.65	https://new.bioconductor.org/packages/release/bioc/html/slingshot.html

Resource availability

Lead contact

Further information and requests for resources and reagents should be directed to and will be fulfilled by the lead contact, Ruibin Xi (ruibinxi@math.pku.edu.cn).

Materials availability

This study did not generate new materials.

Data and code availability

• •This paper analyzes existing, publicly available data. The housekeeping gene list is downloaded from http://www.housekeeping.unicamp.br/. The PBMC3K dataset is downloaded from https://support.10xgenomics.com/single-cell-gene-expression/datasets/1.1.0/pbmc3k/. The Zheng datasets are downloaded from https://hub.docker.com/r/jinmiaochenlab/batch-effect-removal-benchmarking. The Kang dataset is downloaded from Gene Expression Omnibus (GEO) under accession number GSE96583. The iCCA dataset is available in Genome Sequence Archive in National Genomics Data Center under the accession number HRA000863. The spatial transcriptomics data of hepatocellular carcinoma is downloaded from http://lifeome.net/supp/livercancer-st/data.htm. All datasets used in this study are also available at https://doi.org/10.5281/zenodo.8185811.
• •The R package is available at Github: https://github.com/XiDsLab/Festem and at Zenodo: https://doi.org/10.5281/zenodo.11331165. Codes to reproduce the analyses in this manuscript are available at GitHub: https://github.com/XiDsLab/Festem_paper and at Zenodo: https://doi.org/10.5281/zenodo.11331165.
• •Any additional information required to reanalyze the data reported in this paper is available from the lead contact upon request.

Method details

Festem Algorithm

Statistical model

Let$x_{ij}$ be the observed expression of gene $j$ in cell $i$ ($i=1,\dots ,N,j=1,\dots ,M$). We assume that the expression has been normalized to account for the library size by the TMM factor.⁶⁶ If gene $j$ has different expression patterns in different cell-types, it is clustering informative; Otherwise, it is clustering irrelevant. Given that cell $i$ belong to cell-type $g$, assume that expression of gene $j$ follows the distribution $f_{jg}\left(x\right)$. Since the cell-type $g$ is unknown, the expression $x_{ij}$ of gene $j$ should follow the mixture distribution ${\varphi }_j\left(x\right)={\sum }_{g=1}^G{\alpha }_g{f}_{jg}\left(x\right)$, where ${\alpha }_g>0$ is the proportion of the cell-type $g$ in the sequenced cells ($\sum _{g=1}^G{\alpha }_g=1$). With this mixture model, if gene $j$ is clustering-informative, there are at least two $g_1$ and $g_2$ ($g_1\ne g_2$) such that $f_{j{g}_1}\left(x\right)\ne f_{j{g}_2}\left(x\right)$. If gene $j$ is clustering-irrelevant, we have $f_{j1}\left(x\right)=f_{j2}\left(x\right)=\cdots =f_{jG}\left(x\right)$.

Studies have shown that the negative binomial distribution is a suitable model for most scRNA-seq count data, especially for the mostly widely used unique molecular identifier (UMI) data.^20,67,68,69 Hence, we assume that $f_{jg}\left(x\right)$ is a negative binomial distribution. Then, the probability mass function (p.m.f.) of $x_{ij}$ is

$$\varphi \left(x;{\mathbf{\mu }}_{\mathit{j}},{\mathbf{r}}_{\mathit{j}},\mathbf{\alpha }\right)=\sum _{g=1}^G{\alpha }_{g}f\left(x;{\mu }_{jg},r_{jg}\right)\text{,}$$ (Equation 1)

where $f\left(x;\mu ,r\right)$ is the p.m.f. of the negative binomial distribution with mean $\mu$ and size $r$, ${\mathbf{\mu }}_{\mathit{j}}={\left({\mu }_{j1},\cdots ,{\mu }_{jG}\right)}^T,{\mathit{r}}_{\mathit{j}}={\left(r_{j1},\cdots ,r_{jG}\right)}^T$ and $\alpha ={\left({\alpha }_1,\cdots {\alpha }_G\right)}^T$. Previous study has shown that the mixture of negative binomial distributions in 1 is identifiable.⁷⁰ Under model (1), if the gene $j$ is clustering-irrelevant, we have $\left({\mu }_{j1},r_{j1}\right)=\cdots =({\mu }_{jG},r_{jG})$; Otherwise, if gene $j$ is clustering-informative, we have $\left({\mu }_{j{g}_1},r_{j{g}_1}\right)\ne \left({\mu }_{j{g}_2},r_{j{g}_2}\right)$ for some $g_1\ne g_2$. Hence, determining whether the gene is clustering informative, we can consider the following hypothesis testing problem

$$H_{0j}:\left({\mu }_{j1},r_{j1}\right)=\cdots =\left({\mu }_{jG},r_{jG}\right)\leftrightarrow H_{1j}:\left({\mu }_{j{g}_1},r_{j{g}_1}\right)\ne \left({\mu }_{j{g}_2},r_{j{g}_2}\right)\text{for}\text{some}{g}_1\ne g_2\text{.}$$ (Equation 2)

We call the null hypothesis $H_{0j}$ the homogeneity model and the alternative hypothesis $H_{1j}$ the heterogeneity model.

Testing the heterogeneity via the EM-test

We introduce the statistical test for the hypothesis testing problem Equation 2. Since we perform the test for each gene individually, for notation simplicity, we leave out the subscript $j$ in this section. Thus, $x_i$ stands for the expression of a gene in cell $i$. To quantify the evidence of heterogeneity, it is natural to use the log likelihood ratio between the alternative and the null hypothesis as the testing statistic. However, when one of ${\alpha }_g$ equals to zero, the Fisher’s information matrix of model Equation 1 is not finite, leading to the failure of the log likelihood ratio test.²¹ To avoid this, following Li and Chen (2009),²¹ we consider the following penalized likelihood:

$$\mathrm{p}{\mathrm{l}}_{\mathrm{n}}\left(\mathbf{\mu },\mathbf{r},\mathbf{\alpha }\right)=\sum _{i=1}^N\log \varphi \left(x_i;\mathbf{\mu },\mathbf{r},\mathbf{\alpha }\right)+p\left(\mathbf{\alpha }\right)\text{,}$$ (Equation 3)

where

$$p\left(\mathbf{\alpha }\right)=C\left(\sum _{g=1}^G\log {\alpha }_g+G\log G\right)\text{,}$$ (Equation 4)

and $C$ is a parameter. The penalty function $p\left(\mathbf{\alpha }\right)$ is to encourage the proportions ${\alpha }_g$ to stay away from zero. We always set $C=0.001$ in all simulation and real data analyses. We used such a small value due to concerns that a large penalty may over-penalize α, resulting in missing some features relevant to small clusters. We have demonstrated that the choice of C had little influence on the overall performance.¹⁵

Largely speaking, the EM-test statistic is the difference of the maximum penalized log likelihoods (3) under the heterogeneity model $H_1$ and the homogeneity model $H_0$. We denote ${\mathbf{\alpha }}_{\mathbf{0}}={\left(1/G,\cdots ,1/G\right)}^T$ and define $\left({\hat{\mathbf{\mu }}}_0,{\hat{\mathbf{r}}}_0\right)={\left({\hat{\mu }}_0,\dots ,{\hat{\mu }}_0,{\hat{r}}_0,\dots ,{\hat{r}}_0\right)}^T$ be the maximum likelihood estimation of $(\mathbf{\mu },\mathbf{r})$ under $H_0$, where

$$\left({\hat{\mu }}_0,{\hat{r}}_0\right)=\mathrm{Arg}\underset{\left(\mathbf{\mu },\mathbf{r}\right)}{\max }\sum _{i=1}^n\log f\left(x_i;\mathbf{\mu },\mathbf{r}\right)\text{.}$$

Noting that under the homogeneity model $H_0$, the likelihood (1) does not depend on the proportion parameter $\mathbf{\alpha }$, then, $\left({\hat{\mathbf{\mu }}}_0,{\hat{\mathbf{r}}}_0,{\mathbf{\alpha }}_{\mathbf{0}}\right)$ maximizes the penalized log likelihood (3) under the homogeneity model $H_0$.

Under the heterogeneity model $H_1$, it is difficult to directly maximize the penalized likelihood (3), and we use the expectation maximization (EM) algorithm. We choose the initial value of the EM algorithm for $(\mathbf{\mu },\mathbf{r})$ as $\left({\mathbf{\mu }}^{\left(\mathbf{0}\right)},{\mathbf{r}}^{\left(\mathbf{0}\right)}\right)=\left({\hat{\mathbf{\mu }}}_0,{\hat{\mathbf{r}}}_0\right)$. Given an initial value for the proportion parameter ${\mathbf{\alpha }}^{\left(0\right)}={\left({\alpha }_1^{\left(0\right)},\dots ,{\alpha }_G^{\left(0\right)}\right)}^T$, suppose that after $K$ EM updates, we obtain the estimates $({\mathbf{\mu }}^{\left(K\right)},{\mathbf{r}}^{\left(K\right)},{\mathbf{\alpha }}^{\left(K\right)})$ and define the difference of the penalized log likelihood with the initial value ${\mathbf{\alpha }}^{\left(0\right)}$ as

$$M_n^{\left(K\right)}\left({\mathbf{\alpha }}^{\left(0\right)}\right)=2\left[\mathrm{p}{\mathrm{l}}_{\mathrm{n}}\left({\mathbf{\mu }}^{\left(K\right)},{\mathbf{r}}^{\left(K\right)},{\mathbf{\alpha }}^{\left(K\right)}\right)-\mathrm{p}{\mathrm{l}}_{\mathrm{n}}({\hat{\mathbf{\mu }}}_0,{\hat{\mathbf{r}}}_0,{\mathbf{\alpha }}_{\mathbf{0}})\right]\text{.}$$

Given a set of initial values for the mixture proportions $\{{\mathbf{\alpha }}_t:t=1,\dots ,T\}$, the EM-test statistic is defined as

$$\mathrm{E}{\mathrm{M}}_n^{\left(K\right)}=\underset{t=1,...,T}{\max }{M}_n^{\left(K\right)}\left({\mathbf{\alpha }}_t\right)\text{.}$$ (Equation 5)

Here, we always assume that one of ${\mathbf{\alpha }}_{\mathrm{t}}$’s is ${\mathbf{\alpha }}_0$. It can be shown that the limiting distribution of this EM-test statistic under the homogeneity model $H_0$ is stochastically smaller than the chi-square distribution with 3° of freedom ${\chi }_3^2$ (Methods). Hence, we assign p-values using the chi-square distribution ${\chi }_3^2$. By default, Festem uses $K=100$. We always set ${\mathbf{\alpha }}_1={\mathbf{\alpha }}_0$. Other ${\mathbf{\alpha }}_{\mathrm{t}}$’s are randomly sampled from the $\mathrm{G}$-dimensional simplex. If a pre-clustering of single-cells is available, we use ${\mathbf{\alpha }}_2$ as the proportion of different cell-types in the pre-clustering.

The EM algorithm

We use the EM algorithm to optimize the penalized log likelihood (3). Suppose we already have the EM estimation after $\mathrm{k}$ steps, ${\mathbf{\alpha }}^{\left(\mathrm{k}\right)}$ and $\left({\mathbf{\mu }}^{\left(\mathrm{k}\right)},{\mathbf{r}}^{\left(\mathrm{k}\right)}\right)$. We use the following E-step and M-step to update the parameter estimates.

• •E step: calculate the weights $w_{ig}^{\left(k+1\right)}$ ($i=1,\cdots ,N;g=1,\cdots ,G$)

$$w_{ig}^{\left(k+1\right)}=\frac{{\alpha }_g^{\left(k\right)}f\left(x_i;{\mu }_g^{\left(k\right)},r_g^{\left(k\right)}\right)}{\varphi \left(x_i;{\mathbf{\mu }}^{\left(k\right)},{\mathbf{r}}^{\left(k\right)},{\mathbf{\alpha }}^{\left(\mathit{k}\right)}\right)}$$ (Equation 6)

• •M step:

$$\left({\mathbf{\mu }}^{\left(\mathit{k}+\mathbf{1}\right)},{\mathbf{r}}^{\left(\mathit{k}+\mathbf{1}\right)}\right)=\mathrm{Arg}\underset{\left(\mathit{\mu },\mathit{r}\right)}{\max }\sum _{i=1}^n\sum _{g=1}^G{w}_{ig}^{\left(k+1\right)}\log f\left(x_i;{\mu }_g,r_g\right)\text{;}$$ (Equation 7)

$${\mathbf{\alpha }}^{\left(k+1\right)}=\mathrm{Arg}\underset{\mathbf{\alpha }}{\max }\sum _{i=1}^n\sum _{g=1}^G{w}_{ig}^{\left(k+1\right)}\log {\alpha }_g+p\left(\mathbf{\alpha }\right)={\left(\frac{\sum _{i=1}^n{w}_{ig}^{\left(k+1\right)}+C}{n+GC}\right)}_{g=1,\dots ,G}\text{.}$$ (Equation 8)

In the M-step, we optimize Equation 7 with the Newton-Raphson method.

We observe that genes selected based on the EM-test statistic Equation 5 could include genes that are related with other factors such that cell cycle or protein synthesis but unlikely to be related with cell-types. Notice that $w_{ig}^{\left(k+1\right)}$ in Equation 6 can be viewed as the probability of cell $i$ belonging to group $g$. Therefore, to prioritize the genes related with cell-types, we use a prior to reweight $w_{ig}^{\left(k+1\right)}$. The prior is obtained by pre-clustering single-cells using HVGs. Based on the prior, we replace $w_{ig}^{\left(k+1\right)}$ by ${\tilde{w}}_{ig}^{\left(k+1\right)}=\left(1-\gamma \right)w_{ig}^{\left(k+1\right)}+\gamma {\beta }_{ig}$, where

$${\beta }_{ig}=\{\begin{array}{c}1-0.001\left(G-1\right),\\ 0.001,\end{array}\begin{array}{c}\text{ if}\text{cell}i\text{belongs}\text{to}\text{group}g\text{in}\text{the}\text{pre}-\text{clustering;}\\ \text{otherwise.}\end{array}$$ (Equation 9)

and $\gamma$ is called the weight of the prior (by default = 0.05). Then, we can similarly define an EM-test statistic ${\widehat{\text{EM}}}_n^{\left(K\right)}$. Mathematical analysis (Methods) shows that we can still use ${\chi }_3^2$ to get a valid p-value (that is, we can still effectively control the type I error).

It is worth noticing that although the weights $w_{ig}^{\left(k+1\right)}$ can be viewed as the probability of cell $i$ belonging to group $g$, we do not use these weights $w_{ig}^{\left(k+1\right)}$ to define a cell’s cell type. Instead, after selecting the marker genes, we use Seurat to cluster the single-cells based on the selected marker genes and then manually annotate the cell types of the obtained clusters by comparing their markers with markers of known cell types. We do not use $w_{ig}^{\left(k+1\right)}$ to define cell types because these weights are marginally estimated and they are not accurate estimate of the probability that cell $i$ belong to group $g$.

DEG reporting and gene ranking

DEGs are reported based on the p-values and the EM statistics of the genes. More specifically, we first perform the EM-test with a very weak prior (by default $\gamma =0.05$) and obtain a p-value for each gene. The prior is obtained by pre-clustering of single-cells using HVGs. Then, we adjust the p-values using the Benjamini-Hochberg Procedure. Denote the adjusted p-value for gene $j$ as $p_j$. Since the prior at this step is weak, the genes with small adjusted p-values may still contain genes related with other factors but not with cell-types. We further perform the EM-test with a strong prior ($\gamma =0.9$). Let ${\widehat{\text{EM}}}_j^{\mathrm{s}}$ be the EM-test statistic of gene $j$ with the strong prior. Given an FDR level $\beta$, genes with $p_j<\beta$ and ${\widehat{\text{EM}}}_j^{\mathrm{s}}>0$ are called high confidence markers and are reported as DEGs. Genes are first ranked by the confidence of genes, and within high or low confidence gene categories, genes are ranked by the p-values $p_j$. Top genes are selected for downstream clustering and dimension reduction analyses.

Data preprocessing

Before applying the EM-test, we preprocess the data to remove outliers and adjust for the library size. To remove the outliers, we applied the following winsorization⁷¹ procedure.

Let$y_{ij}$ be the observed UMI or read count of gene $j$ in cell $i$ ($i=1,\dots ,N,j=1,\dots ,M$). For each gene $j$, we first select cells with the largest top 5% expression $y_{ij}$. Let $q_1$ and $q_3$ be the 0.25^th and 0.75^th quantile of $y_{ij}$ over these selected single-cells. Denote $\text{IQR}=\min \left\{q_3-q_1,1\right\}$ as the interquartile range. If any $y_{ij}$ ($i=1,\dots ,N$) is larger than $q_3+3\text{IQR}$, we consider it as an outlier. We replace the expression of the outliers with the mean of $y_{ij}$’s over the selected single-cells whose expressions are smaller than $q_3+3\text{IQR}$.

After removing the outliers, we normalize the library size with the TMM factor.⁶⁶ For each cell, we calculate its TMM factor by the calcNormFactors function in package edgeR. Then, we divide the count of each gene in this cell by its TMM factor and using the result as the mean of the Poisson distribution to sample a new count. Then, we apply the outlier removal procedure again to the new counts and using this count matrix to detect DEGs.

Assignment of DEGs to cell clusters

Note that testing hypothesis (2) does not indicate in which clusters a DEG highly expresses. After reporting the DEG list, we further use the Scott-Knott test¹⁹ to assign each DEG to a cluster or clusters as its or their marker based on the mean expression within each cluster. The Scott-Knott test is originally proposed as a post-hoc procedure in ANOVA for grouping the means in ANOVA. In our scenario, given a marker gene, it groups cell clusters into several homogeneous groups with similar mean expressions of the marker. Specifically, for a given clustering result, which can be derived from the DEGs reported by Festem, we first estimate the mean expression of a DEG in each cluster. Then, we use the SK function in R package ScottKnott⁶² with significant level 0.05 to classify these means into several expression levels, from the highest to the lowest. Finally, we call this gene the marker of a cluster if its mean is in the highest two expression levels.

Derivation of EM iterations

In this section, we will give details of the derivation of Equations 6, 7, and 8 in the previous section. The assumptions and notations used here were the same as in the previous section. Remember we assume that the probability mass function (p.m.f.) of the observed expression $x_{ij}$ of gene $j$ in cell $i$ is

$$\varphi \left(x;{\mathbf{\mu }}_{\mathit{j}},{\mathbf{r}}_{\mathit{j}},\mathbf{\alpha }\right)=\sum _{g=1}^G{\alpha }_{g}f\left(x;{\mu }_{jg},r_{jg}\right)\text{.}$$

Since we perform the test for each gene individually, for notation simplicity, we leave out the subscript $j$ in this section. Thus, $x_i$ stands for the expression of a gene in cell $i$ with p.m.f.

$$\varphi \left(x;\mathit{\mu },\mathbf{r},\mathbf{\alpha }\right)=\sum _{g=1}^G{\alpha }_{g}f\left(x;{\mu }_g,r_g\right)\text{,}$$

where $\mathit{\mu }={\left({\mu }_1,\dots ,{\mu }_G\right)}^T$ and $\mathit{r}={\left(r_1,\dots ,r_G\right)}^T\text{.}$ As explained in the Methods section, we aim to use the standard EM algorithm⁷² to optimize the penalized log likelihood (3), that is

$$\mathrm{p}{\mathrm{l}}_{\mathrm{n}}\left(\mathbf{\mu },\mathbf{r},\mathbf{\alpha }\right)=\sum _{i=1}^N\log \varphi \left(x_i;\mathbf{\mu },\mathbf{r},\mathbf{\alpha }\right)+p\left(\mathbf{\alpha }\right)\text{,}$$ (Equation 10)

where $p\left(\mathbf{\alpha }\right)=C\left(\sum _{g=1}^G\log {\alpha }_g+G\log G\right)\text{.}$ Now, we introduce hidden variables $z_{ig}={\mathbf{1}}_{\left\{x_i\text{belongs}\text{to}\text{cell}\text{type}\mathrm{g}\right\}}$ and define ${\mathit{z}}_{\mathit{i}}={\left(z_{i1},\dots ,z_{iG}\right)}^T,\mathit{Z}={\left({\mathit{z}}_1^T,\dots ,{\mathit{z}}_N^T\right)}^T$. Then, ${\mathit{z}}_{\mathit{i}}\sim \text{Multinomial}\left({\alpha }_1,\dots ,{\alpha }_G\right)$ with p.m.f.

$$p\left({\mathit{z}}_{\mathit{i}}\right)=\prod _{g=1}^G{\alpha }_g^{z_{ig}}·{\mathbf{1}}_{\left\{\sum _{g=1}^G{z}_{ig}=1\right\}}$$

and $x_i|{\mathit{z}}_i$ has p.m.f.

$$p\left(x_i|{\mathit{z}}_i\right)=\prod _{g=1}^{G}f{\left(x_i;{\mu }_g,r_g\right)}^{z_{ig}}\text{.}$$

So, the first term in the penalized log likelihood Equation 10 is actually the marginal distribution of $x_i$. Hence, Equation 10 can be viewed as the penalized marginal log likelihood, whereas the penalized full log likelihood is

$${\widetilde{\text{pl}}}_{\mathrm{n}}\left(\mathbf{\mu },\mathbf{r},\mathbf{\alpha }\right)=\sum _{i=1}^N\sum _{g=1}^G{z}_{ig}\log f\left(x_i;{\mu }_g,r_g\right)+p\left(\mathbf{\alpha }\right)$$

Using the Bayes formula, we know that the distribution of $\mathit{Z}$ conditioning on $x_1,\dots ,x_N$ is

$$p\left(\mathit{Z}|x_1,\dots ,x_N\right)\propto \prod _{i=1}^N\prod _{g=1}^{G}f{\left(x_i;{\mu }_g,r_g\right)}^{z_{ig}}·\prod _{i=1}^N\prod _{g=1}^G{\alpha }_g^{z_{ig}}=\prod _{i=1}^N\prod _{g=1}^G{\left\{{\alpha }_{g}f\left(x_i;{\mu }_g,r_g\right)\right\}}^{z_{ig}}\text{.}$$

Therefore, we know that ${\mathit{z}}_{\mathit{i}}|x_1,\dots ,x_N\sim \text{Multinomial}\left(w_{i1},\dots ,w_{iG}\right)$ with

$$w_{ig}=\frac{{\alpha }_{g}f\left(x_i;{\mu }_g,r_g\right)}{\varphi \left(x_i;\mathit{\mu },\mathbf{r},\mathbf{\alpha }\right)}$$

and ${\mathit{z}}_{\mathbf{1}},...,{\mathit{z}}_N$ are independent.

Hence, suppose we already have the EM estimation after $\mathrm{k}$ iterations, ${\mathbf{\alpha }}^{\left(\mathrm{k}\right)}$ and $\left({\mathbf{\mu }}^{\left(\mathrm{k}\right)},{\mathbf{r}}^{\left(\mathrm{k}\right)}\right)$. Then, in the next E-step, we need to compute

$${\mathbb{E}}_{Z\sim p\left(·|x_1,\dots ,x_N,{\mathbf{\alpha }}^{\left(\mathrm{k}\right)},{\mathbf{\mu }}^{\left(\mathrm{k}\right)},{\mathbf{r}}^{\left(\mathrm{k}\right)}\right)}\left[\log {\widetilde{\text{pl}}}_{\mathrm{n}}\left(\mathbf{\mu },\mathbf{r},\mathbf{\alpha }\right)\right]=\sum _{i=1}^N\sum _{g=1}^G{\mathbb{E}}_{Z\sim p\left(·|x_1,\dots ,x_N,{\mathbf{\alpha }}^{\left(\mathrm{k}\right)},{\mathbf{\mu }}^{\left(\mathrm{k}\right)},{\mathbf{r}}^{\left(\mathrm{k}\right)}\right)}\left[z_{ig}\right]\log f\left(x_i;{\mu }_g,r_g\right)+p\left(\mathbf{\alpha }\right)=\sum _{i=1}^N\sum _{g=1}^G{w}_{ig}^{\left(k+1\right)}\log f\left(x_i;{\mu }_g,r_g\right)+p\left(\mathbf{\alpha }\right)\text{,}$$ (Equation 11)

where

$$w_{ig}^{\left(k+1\right)}=\frac{{\alpha }_g^{\left(k\right)}f\left(x_i;{\mu }_g^{\left(k\right)},r_g^{\left(k\right)}\right)}{\varphi \left(x_i;{\mathbf{\mu }}^{\left(k\right)},{\mathbf{r}}^{\left(k\right)},{\mathbf{\alpha }}^{\left(\mathit{k}\right)}\right)}\text{.}$$

Hence, the E-step is equivalent to computing the weights $w_{ig}^{\left(k+1\right)}$ ($i=1,\cdots ,N;g=1,\cdots ,G)$. Then, in the M-step, we need to maximize Equation 11 with respect to $\mathit{\mu },\mathit{r},\mathit{\alpha }$. We note that in (S2), $(\mathit{\mu },\mathit{r})$ and $\mathit{\alpha }$ are decoupled, so we can optimize them separately:

$$\begin{array}{c}\left({\mathbf{\mu }}^{\left(\mathit{k}+\mathbf{1}\right)},{\mathbf{r}}^{\left(\mathit{k}+\mathbf{1}\right)}\right)=\mathrm{Arg}\underset{\left(\mathit{\mu },\mathit{r}\right)}{\max }\sum _{i=1}^n\sum _{g=1}^G{w}_{ig}^{\left(k+1\right)}\log f\left(x_i;{\mu }_g,r_g\right)\text{;}\\ {\mathbf{\alpha }}^{\left(k+1\right)}=\mathrm{Arg}\underset{\mathbf{\alpha }}{\max }\sum _{i=1}^n\sum _{g=1}^G{w}_{ig}^{\left(k+1\right)}\log {\alpha }_g+C\left(\sum _{g=1}^G\log {\alpha }_g+G\log G\right)={\left(\frac{\sum _{i=1}^n{w}_{ig}^{\left(k+1\right)}+C}{n+GC}\right)}_{g=1,\dots ,G}\text{.}\end{array}$$

To conclude, the EM update runs as follows:

E step: calculate the weights $w_{ig}^{\left(k+1\right)}$ ($i=1,\cdots ,N;g=1,\cdots ,G$)

$$w_{ig}^{\left(k+1\right)}=\frac{{\alpha }_g^{\left(k\right)}f\left(x_i;{\mu }_g^{\left(k\right)},r_g^{\left(k\right)}\right)}{\varphi \left(x_i;{\mathbf{\mu }}^{\left(k\right)},{\mathbf{r}}^{\left(k\right)},{\mathbf{\alpha }}^{\left(\mathit{k}\right)}\right)}$$

M step:

$$\left({\mathbf{\mu }}^{\left(\mathit{k}+\mathbf{1}\right)},{\mathbf{r}}^{\left(\mathit{k}+\mathbf{1}\right)}\right)=\mathrm{Arg}\underset{\left(\mathit{\mu },\mathit{r}\right)}{\max }\sum _{i=1}^n\sum _{g=1}^G{w}_{ig}^{\left(k+1\right)}\log f\left(x_i;{\mu }_g,r_g\right)$$ (Equation 12)

$${\mathbf{\alpha }}^{\left(k+1\right)}={\left(\frac{\sum _{i=1}^n{w}_{ig}^{\left(k+1\right)}+C}{n+GC}\right)}_{g=1,\dots ,G}$$ (Equation 13)

In the M-step, we optimize Equation 12 with the Newton-Raphson method.

Proof for the bound of limiting distribution of the EM-test statistic under null hypothesis

Let ${\widehat{\text{EM}}}_n^{\left(K\right)}$ be the shrunk version of EM-test statistic, and $\mathrm{E}{\mathrm{M}}_n^{\left(K\right)}$ represent the non-shrunk version. Recall that the difference of the penalized log likelihood with the initial value ${\mathbf{\alpha }}^{\left(0\right)}$ is defined as

$$M_n^{\left(K\right)}\left({\mathit{\alpha }}^{\left(0\right)}\right)=2\left[\mathrm{p}{\mathrm{l}}_{\mathrm{n}}\left({\mathit{\mu }}^{\left(K\right)},{\mathit{r}}^{\left(K\right)},{\mathit{\alpha }}^{\left(K\right)}\right)-\mathrm{p}{\mathrm{l}}_{\mathrm{n}}({\hat{\mathit{\mu }}}_0,{\hat{\mathit{r}}}_0,{\mathit{\alpha }}_{\mathbf{0}})\right]\text{,}$$

where $\left({\mathit{\mu }}^{\left(K\right)},{\mathit{r}}^{\left(K\right)},{\mathit{\alpha }}^{\left(K\right)}\right)$ is the non-shrunk version parameter estimates, ${\mathbf{\alpha }}_{\mathbf{0}}={\left(1/G,\cdots ,1/G\right)}^T\text{,}$ and $\left({\hat{\mathbf{\mu }}}_0,{\hat{\mathbf{r}}}_0\right)={\left({\hat{\mu }}_0,\dots ,{\hat{\mu }}_0,{\hat{r}}_0,\dots ,{\hat{r}}_0\right)}^T$ is the maximum likelihood estimation of $(\mathbf{\mu },\mathbf{r})$ under $H_0$ with

$$\left({\hat{\mu }}_0,{\hat{r}}_0\right)=\mathrm{Arg}\underset{\left(\mathbf{\mu },\mathbf{r}\right)}{\max }\sum _{i=1}^n\log f\left(x_i;\mathbf{\mu },\mathbf{r}\right)\text{.}$$

Given a set of initial values for the mixture proportions $\{{\mathit{\alpha }}_t:\mathrm{t}=1,\dots ,\mathrm{T}\}$, the non-shrunk version EM-test statistic is defined as

$$\mathrm{E}{\mathrm{M}}_{\mathrm{n}}^{\left(\mathrm{K}\right)}=\underset{t=1,...,T}{\max }{\mathrm{M}}_{\mathrm{n}}^{\left(\mathrm{K}\right)}\left({\mathbf{\alpha }}_{\mathrm{t}}\right)$$

Without loss of generality, we assume that $T=1$. Then, the non-shrunk version EM-test statistic can be simplified as

$${\text{EM}}_{\mathrm{n}}^{\left(\mathrm{K}\right)}=2\left[\mathrm{p}{\mathrm{l}}_{\mathrm{n}}\left({\mathit{\mu }}^{\left(K\right)},{\mathit{r}}^{\left(K\right)},{\mathit{\alpha }}^{\left(K\right)}\right)-\mathrm{p}{\mathrm{l}}_{\mathrm{n}}({\hat{\mathit{\mu }}}_0,{\hat{\mathit{r}}}_0,{\mathit{\alpha }}_0)\right]\text{.}$$

Likewise, the shrunk version EM-test statistic can be expressed as

$${\widehat{\text{EM}}}_n^{\left(K\right)}=2\left[\mathrm{p}{\mathrm{l}}_{\mathrm{n}}\left({\hat{\mathit{\mu }}}^{\left(K\right)},{\hat{\mathit{r}}}^{\left(K\right)},{\hat{\mathit{\alpha }}}^{\left(K\right)}\right)-\mathrm{p}{\mathrm{l}}_{\mathrm{n}}\left({\hat{\mathit{\mu }}}_0,{\hat{\mathit{r}}}_0,{\mathit{\alpha }}_0\right)\right]\text{,}$$

where $\left({\hat{\mathit{\mu }}}^{\left(K\right)},{\hat{\mathit{r}}}^{\left(K\right)},{\hat{\mathit{\alpha }}}^{\left(K\right)}\right)$ is the shrunk version parameter estimates. Then, we define the mixing version parameter estimates as follows:

$$\{\begin{array}{c}\left({\tilde{\mathit{\mu }}}^{\left(K\right)},{\tilde{\mathit{r}}}^{\left(K\right)},{\tilde{\mathit{\alpha }}}^{\left(K\right)}\right)=\left({\mathit{\mu }}^{\left(K\right)},{\mathit{r}}^{\left(K\right)},{\mathit{\alpha }}^{\left(K\right)}\right),{\text{EM}}_{\mathrm{n}}^{\left(\mathrm{K}\right)}\ge {\widehat{\text{EM}}}_n^{\left(K\right)};\\ \left({\tilde{\mathit{\mu }}}^{\left(K\right)},{\tilde{\mathit{r}}}^{\left(K\right)},{\tilde{\mathit{\alpha }}}^{\left(K\right)}\right)=\left({\hat{\mathit{\mu }}}^{\left(K\right)},{\hat{\mathit{r}}}^{\left(K\right)},{\hat{\mathit{\alpha }}}^{\left(K\right)}\right),{\text{EM}}_{\mathrm{n}}^{\left(\mathrm{K}\right)}<{\widehat{\text{EM}}}_n^{\left(K\right)}.\end{array}$$

Next, we define the mixing version EM-test statistic as

$${\widehat{\text{EM}}}_n^{\left(K\right)}=2\left[{\text{pl}}_{\mathrm{n}}\left({\tilde{\mathit{\mu }}}^{\left(K\right)},{\tilde{\mathit{r}}}^{\left(K\right)},{\tilde{\mathit{\alpha }}}^{\left(K\right)}\right)-{\text{pl}}_{\mathrm{n}}\left({\hat{\mathit{\mu }}}_0,{\hat{\mathit{r}}}_0,{\mathit{\alpha }}_0\right)\right]\text{.}$$

Clearly, we have

$${\widehat{\text{EM}}}_n^{\left(K\right)}=\max \left\{{\widehat{\text{EM}}}_n^{\left(K\right)},{\text{EM}}_n^{\left(K\right)}\right\}\ge {\widehat{\text{EM}}}_n^{\left(K\right)}\text{.}$$

Using Lemma 19, 20 and the proof of Theorem 4 in Wang et al. (2023),¹⁵ we can conclude that ${\widehat{\text{EM}}}_n^{\left(K\right)}$ is stochastically less than ${\chi }^2\left(3\right)$. Consequently, ${\widehat{\text{EM}}}_n^{\left(K\right)}$ is also stochastically less than ${\chi }^2\left(3\right)$.

Benchmarking on simulated datasets

Generation of simulated datasets

In the simulation, we generated data with 3000 cells and 20,000 genes from four different settings, with varying number of cell-types and marker genes. Specifically, the four settings are: (1) 2 cell-types with 400 marker genes (low-noise), (2) 2 cell-types with 200 marker genes (high-noise), (3) 5 cell-types with 400 marker genes (low-noise) and (4) 5 cell-types with 200 marker genes (high-noise). The marker genes were set as the first $K$ (200 or 400) genes. We set the marker genes only upregulated in one cell-type and had the same expression distributions in other cell-types. For example, in the simulation with 2 cell-types and 400 marker genes, the first 200 marker genes were set as upregulated in the first cell-type and had the same distribution in the other cell-type; the second 200 marker genes were set as upregulated in the second cell-type and had had the same distribution in the other cell-type; all other genes had the same distribution in all cells.

To make simulation data more similar to the real ones, for each gene in each cell-type of the PBMC3K dataset, we first estimated the mean and size parameter of a negative binomial distribution from the count data of the gene in the cells of the cell-type. The cell-type annotation was chosen as the annotation of the PBMC3K dataset given in the Seurat tutorial (https://satijalab.org/seurat/articles/pbmc3k_tutorial.html#assigning-cell-type-identity-to-clusters-1). Then, we used these estimated mean and size parameters to generate simulation data. For each gene, we first randomly sampled a pair of mean $\mu$ and size parameters $r$ from the estimated parameters. If a gene was a non-marker gene, we just independently generated count data for each cell from the negative binomial distribution with the sampled parameters $(\mu ,r)$. If a gene was a marker gene, we further sampled a log₂-foldchange $\delta$ from the uniform distribution in [1.5, 2.5]. For cells with the gene upregulated, we generated their expressions of the gene from the negative binomial distribution with mean $\mu ·2^{\delta }$ and $r$. The expressions of the gene in other cells were generated from the negative binomial distribution with mean $\mu$ and $r$.

Benchmarking on marker gene detection

We benchmarked Festem on marker genes detection and compared with DEG detection algorithms DESeq2,¹² EdgeR,¹³ MAST,²⁷ Wilcoxon rank-sum test,²⁸ ROSeq,²⁹ singleCellHaystack¹⁴ and TN test.¹¹ For methods requiring a clustering result, we used the following procedure to detect DEGs. In the simulation study, we first selected 1000 HVGs using HVGvst and then performed principal component analysis (PCA) dimension reduction. We then used the first 10 principle components (PCs) to construct a nearest-neighbor graph by Seurat’s⁵³ FindNeighbors function, and clustered the cells with the Louvain algorithm¹⁸ by the FindClusters function. In order to get the desired number of clusters, we applied the Louvain algorithm starting with the resolution 0.01 and gradually increased the resolution (step size 0.01). When Louvain first reported a cluster number that were the same as the true number of clusters, we stopped this process and used the corresponding cluster result. For real data analyses, we used 2000 HVGs and selected PC numbers based on the elbow plot. Clustering was performed similarly and the resolution was chosen such that the obtained cluster number was the same as the original study of the data.

For Festem, we used the prior as the pre-clustering results based on HVGs (1000 HVGs in simulations and 8000 HVGs in real analyses). singleCellHaystack does not require a clustering result but requires a dimensional reduction result. For singleCellHaystack, we first performed UMAP dimensional reduction using Seurat’s RunUMAP function on the 10 PC representation of the data. Then, singleCellHaystack was used to detect marker genes on both the 2D UMAP and top PC representations of the data. In simulations, we used the top 10 PCs, and in real data analyses, we selected the PC numbers based on elbow plots.

For DESeq2, we ran with default parameters and recorded adjusted p-values with and without the independent filtering step. For EdgeR, we used its quasi-likelihood test setting with default parameters. p-values both with and without the filtering step were recorded. MAST and ROSeq were also run with their default settings. The Wilcoxon rank-sum test was performed using wilcox.test function from the R language. For TN test, we first normalized the counts and split the dataset into two parts. According to TN test, one dataset was used for clustering and training of the separating hyperplanes between clusters. The clustering was performed using the same procedure described above. The other dataset was used for marker gene detection based on the obtained hyperplanes.

Wilcoxon rank-sum test and ROSeq can only compare two groups of cells. When there were multiple groups, we first used these methods to do “one-versus-the-rest” comparison for each cluster and then combine the derived p-values with the Bonferroni method.⁷³ For all methods, we adjust the derived p-values with the Benjamini-Hochberg method¹⁶ and reported genes with adjusted p-values smaller than 0.05 as marker genes.

Benchmarking feature selection

We compared Festem with 5 feature selection methods HVGvst,⁵ HVGdisp,²³ DUBStepR,²⁴ devianceFS⁸ and TrendVar.⁷ We used the FindVariableFeatures function in Seurat for HVGvst and HVGdisp. DUBStepR, devianceFS and TrendVar were run with the DUBStepR, scry and scran packages, respectively. For simulations, we selected the top 500 genes reported by each method for downstream analyses. For PBMC3K, Kang and Zheng datasets, we used the top 1000, 2500, and 3000 genes, respectively. Then, we used the first 10 PCs of the selected genes for clustering. Louvain was used for clustering and the resolution was selected as in the previous section. ARIs were calculated between the clustering results and the known cell-type labels (for simulation study). UMAP dimension reduction was further performed using the 10 PC data, and SIs of known cell-type labels were calculated on the UMAP coordinates by the silhouette function in the cluster package.

Construction of marker genes silver standards

For each dataset, we selected its top 2000 HVGs, performed PCA analyses and obtained UMAP embeddings of the single-cells using the top 10 PCs. Then, we calculated each gene’s Moran index in the 2-dimensional UMAP-embedding space. The non-marker genes were taken as the housekeeping genes with absolute values of Moran’s indices smaller than a threshold $\mathrm{T}$ and with p-values larger than 0.05. For PBMC3K and Kang dataset, $\mathrm{T}$ was set to be 0.02, and for Zheng dataset, $\mathrm{T}$ was set to be 0.005. $\mathrm{T}$ was set to ensure that the expressions of genes in the non-marker gene list were not visually aggregated on the UMAP plot. The marker genes were taken as non-housekeeping genes that had Moran’s indices larger than 0.1 and had non-zero expression in at least 5% cells. We did not consider genes with negative Moran’s indices as markers because Moran’s indices of all genes are larger than −0.02 and those with negative Moran’s indices do not have any expression preference on the UMAP plots. We also visually checked that these genes do show expression preference toward certain cell-types on the UMAP plots.

Comparison of feature selection methods

To annotate single-cells, we applied CellTypist (v1.6.2) under the “probabilistic match” mode based on the available “Immune_All_Low” model (v2, containing 98 immune cell-types). Then, we compute that variation of information between the annotated cells and their clustering results. Formally, given two sets of clustering results $X=(x_1,\dots ,x_N)$ and $Y=(y_1,\dots ,y_N)$, we denote the label categories in them as $\mathcal{X}$ and $\mathcal{Y}$, and define $p_x=\left|\right\{i:x_i=x\left\}\right|$ and $p_{xy}=\left|\right\{\left(i,j\right):x_i=x,y_i=y\left\}\right|$ for all $x\in \mathcal{X},y\in \mathcal{Y}\text{.}$ Then, the variation of information is defined as the sum of two conditional entropies:

$$V\left(X,Y\right)=H\left(X|Y\right)+H\left(Y|X\right)\text{,}$$

where $H\left(X\right|Y)$ is defined as:

$$H\left(X|Y\right)=-\sum _{x\in \mathcal{X},y\in \mathcal{Y}}{p}_{xy}\log \frac{p_{xy}}{p_x}\text{.}$$

We computed the variation of information using the vi.dist function in R package mclust (v6.0.0). The Calinski–Harabasz (CH) index characterizes the extent of between-cluster separation compared with the within-cluster dispersion. We computed the CH index of each method based on their top principle components with the calinhara function in R package fpc (v2.2-9).

Survival analysis of public iCCA datasets

We grouped samples into emra^high and emra^low groups based on the average expression of CX3CR1. If the expression of CX3CR1 of a sample is higher than the average expression, then this sample belongs to the emra^high group; otherwise, it belongs to the emra^low group. Similarly, we further divided these two groups based on the average total expression of FAM3C and DUSP4. As a result, we got four groups of samples, emra^highex^low, emra^highex^high, emra^lowex^low and emra^lowex^high. Kaplan-Meier curves and log rank tests were carried out with R functions survfit and ggsurvplot in packages survival and survminer. Batch removal were performed using harmony.⁶³

Quantification and statistical analysis

All details regarding the method are fully explained in the section Method Details. Here we provide a brief summary of certain analyses, parameters, and software that were used. The section of Method Details entitled “Festem Algorithm” uses base R v.4.0.5. To assign DEGs to cell clusters, we use ScottKnott v.1.3–1 with significant level 0.05. In Figures 2 and 3, DEGs are identified using nomial FDR level 0.05. In Figure 5C, Kaplan-Meier curves and log rank tests were carried out with R packages survival and survminer.^64,65

Published: July 8, 2024