SigEMD: A powerful method for differential gene expression analysis in single-cell RNA sequencing data

Abstract

Differential gene expression analysis is one of the significant efforts in single cell RNA sequencing (scRNAseq) analysis to discover the specific changes in expression levels of individual cell types. Since scRNAseq exhibits multimodality, large amounts of zero counts, and sparsity, it is different from the traditional bulk RNA sequencing (RNAseq) data. The new challenges of scRNAseq data promote the development of new methods for identifying differentially expressed (DE) genes. In this study, we proposed a new method, SigEMD, that combines a data imputation approach, a logistic regression model and a nonparametric method based on the Earth Mover’s Distance, to precisely and efficiently identify DE genes in scRNAseq data. The regression model and data imputation are used to reduce the impact of large amounts of zero counts, and the nonparametric method is used to improve the sensitivity of detecting DE genes from multimodal scRNAseq data. By additionally employing gene interaction network information to adjust the final states of DE genes, we further reduce the false positives of calling DE genes. We used simulated datasets and real datasets to evaluate the detection accuracy of the proposed method and to compare its performance with those of other differential expression analysis methods. Results indicate that the proposed method has an overall powerful performance in terms of precision in detection, sensitivity, and specificity.

1. Introduction

The next generation sequencing (NGS) [1] of mRNAs in single cells is a major advancement over sequencing of mRNAs in bulk cells. Sequencing mRNAs at a single-cell level enables discovering the cell-specific changes caused by intrinsic or extrinsic stimuli in transcriptome [2–4], which cannot be detected at a bulk-cell level. Differential gene expression analysis uses single-cell RNA sequencing (scRNAseq) data to detect differentially expressed genes between different conditions, such as cell types or cell development stages.

In RNAseq experiments, mRNAs are from a bulk of cells and a gene expression value is the average of thousands of cells’ expression values. However in single cell sequencing data, expression values provide the distribution of expression levels for each gene across a population of cells. This distribution exhibits multimodal and heterogeneous patterns [5,6]. In study [7], it has been shown that scRNAseq data tend to exhibit complex distribution and have more than two modes, while the RNAseq data do not. Single-cell gene expression is a stochastic process, so that high variability exists in gene expression values among cells [8]. In other words, the expression level is related to the cell subtype and the state of the cell in the whole cell cycle. Therefore, the biological differences between cells, such as different cell types, different mRNA contents, and different cell states, result in the multimodality and heterogeneity in gene expression values. For example, cells from the same brain tumor pose huge heterogeneity [6].

Another characteristic of scRNAseq data is the large numbers of zero counts. However, not all the detected zero counts from the sample cells are truly zero expressed. That means some truly expressed genes may not be detected in the process of sequencing. This is partly due to the tiny amount of starting RNAs and low capture efficiency of RNAs. The small quantity of starting RNAs results in many transcripts being below the detection threshold. Additionally, the low capture efficiency can miss plenty of transcripts during the reverse transcription. Thus, we can observe a “drop-out” phenomenon [9] where some transcripts are strongly expressed in some cells but are unexpressed in others, even though these cells are under the same condition. Since the drop-out event is inevitable and has an influence on the downstream analysis of the expression data, such as differential gene expression analysis and cell type identification, several studies [10–14] have used data imputing approaches to reconstruct the missing values. The missing value of a gene in a single cell can be estimated directly using the expression values from other genes and cells via regression model [10–12]. Also, the missing values can be imputed by projecting the expression values to a low dimension space. In the low-dimensional latent space, expectation-maximization algorithm is used to estimate the latent expression levels of the observed zero values through the information from non-zero values in the expectation step [14]. Heterogeneity and multimodality, large amounts of zero counts, and sparsity of scRNAseq data pose new challenges to the differential gene expression analysis. To address these challenges, new strategies and models [9,15–18] have been proposed specifically for scRNAseq data. SCDE (single-cell differential expression) [9] uses a two-part joint model for detecting differentially expressed (DE) genes to accommodate multimodal expression values and drop-out events; one part corresponds to the normal observed genes, the other corresponds to the drop-out events. Another approach, MAST (model-based analysis of single-cell transcriptomics) [15], uses a hurdle model to represent the zero counts and positive expression values; then it uses logistic regression and linear regression to respectively identify DE genes for each part. Monocle [16] also uses a linear model – generalized additive models (GAMs) – to identify DE genes. A recent approach named scDD [17], considers four different modality scenarios for gene expression value distributions within and across biological conditions. A nonparametric method, D3E [18], uses two nonparametric methods, Cramer-von Mises test and the Kolmogorov-Smirnov test, for comparing the distributions of expression values of each gene in different conditions in order to identify DE genes.

A few studies have been conducted to compare and evaluate the performances of the differential gene expression analysis tools for scRNAseq data [19,20]. In these studies, tools specifically developed for scRNAseq, such as SCDE, MAST, Monocle, D3E, and tools developed for bulk RNAseq data that can be used for single cells, such as DESeq [21], Limma [22], and edgeR [23], were compared in details in terms of accuracy of detection, effect of sample sizes (number of cells), and consistency of detection. In general, consistency of detection, that is the consensus of the significantly DE genes called by the tools, is not high across the aforementioned tools. Nonparametric methods, which can capture multimodality in data, show more sensitivity to the sparsity of data [19,20]. This is because the large amounts of zero counts in scRNAseq data affect the expression values’ distributions significantly. When there are a large number of zero counts, they result in high false positive rates in detecting DE genes. By contrast, the model-based methods result in low numbers of false positives since their model are specifically developed for the drop-out events. However, these methods cannot capture the multimodality of the expression data across the cells, leading to low true positive rates. Thus, to identify DE genes with a higher accuracy, characteristics of both multimodality and large amounts of zero counts need to be considered.

One way to reduce the false positives in detecting DE genes for scRNAseq data is employing the information about the expression levels of genes that have interactions with each other. It has been shown that genes in the same pathway tend to exhibit strong co-relationship and have consistent changes in the direction of expression levels [24–26]. References [24,25] have proposed to utilize gene interaction network to identify DE genes for RNAseq data. A Markov random field (MRF) model has been developed and the state of a gene is determined by the states of its neighbors in the interaction network. However, the methods in [24,25] have used a Poisson or Poisson-Beta model, which is not suitable for capturing the multimodality in scRNAseq data.

In this study, we proposed to apply a combination of data imputation, a statistical model and a nonparametric method, aiming at addressing the both challenges of multimodality and large amounts of zero counts in scRNAseq data. To overcome the multimodality, the proposed method employs a nonparametric method, inspired by the EMDomics method for bulk mRNA data [27], to compute the difference between the expression values of a gene in two groups of cells. Unlike the model-based methods that need to estimate the model parameters, EMDomics uses the nonparametric Earth Mover’s Distance (EMD) [28,29], to measure the distance between the distributions of expression values. It has a good performance on high heterogeneous data while it cannot handle the numerous zero counts for scRNAseq data, which results in a low specificity. To handle the large numbers of zero counts, the proposed method uses a logistic regression to model the zero counts in the expression values and uses a Wald test to decide whether or not to consider the zero counts when implementing EMD. In this logistic regression model, the method measures the relationship between the expression binary values and the independent features (e.g. condition) of the cells. For the genes that their zero counts need to be considered, a Lasso regression model is especially employed to impute the zeros through expression values from similar genes. Additionally, to further decrease the false positives, we proposed to employ the gene interaction network. Based on the results of the differential analysis, the method adjusts the final state of a gene by considering the states of its majority neighbors. We used both simulated datasets and real datasets to evaluate the performance of the proposed differential expression analysis method, named SigEMD. Results show that SigEMD is a powerful method for accurately identifying DE genes in multimodal and sparse scRNAseq data. The source code of SigEMD (version 0.21.1) in R language is available at https://github.com/NabaviLab/SigEMD.

The rest of the paper is organized as follows: Section 2 presents the main methods. Section 3 introduces the real and simulated datasets. In Section 4, results are discussed to show the effectiveness of the proposed method. Section 5 concludes the paper.

2. Methods

We developed a method for differential gene expression analysis for scRNAseq data that combines data imputation, logistic regression, and nonparametric distribution comparison approaches to address the both challenges of large number of zero counts and multimodality. The method consists of four steps. First, it uses a logistic regression and the Wald test to identify the genes that their zero counts significantly affect the distribution of expression values (Section 2.1). Then, the method imputes the zero counts of the identified genes in the previous step by using a Lasso regression model (Section 2.2). After fixing the zero values, the method employs a nonparametric approach based on the Earth Mover’s Distance (EMD) for differential analysis of expression values’ distributions and assigning a score to each gene (Section 2.3). And finally, it calculates p-values for the scores from the differential analysis via permutation test to identify significantly DE genes (Section 2.4). We also proposed a method to adjust the significantly DE status of the genes using gene interaction network (Section 2.5).

The regression model and data imputation (steps 1 & 2) are used to reduce the impact of the large number of zero counts; and EMD is used to reflect the difference between the multimodal gene expression values under different conditions (steps 3 & 4). The gene interaction information is employed to further decrease the false positives in detecting DE genes.

2.1. Statistical regression model for zero read counts

Because of the drop-out events, there are a large number of zero counts in scRNAseq data for many genes. For these genes, the distributions of zero counts among all the cells can impact the evaluation of the true difference between the expression levels’ distributions of the two conditions. To reduce the effect of zero counts first, we used a logistic regression statistical model to evaluate whether or not the condition is highly related with the zero counts (described as below), then we imputed the zero counts of those genes that their zero counts need to be considered (described in section sub2.2).

Let $\mathbf{Y}=\left[y_{gm}\right]$ be the expression value matrix, where $y_{gm}$ is the expression value of gene $g(g\in \{1,\dots N\left\}\right)$ in cell $m(m\in \{1,\dots M\left\}\right)$. The total number of genes are $N$ and total number of cells are $M$. We used an indicator $z_{gm}$ for each gene (i.e. $z_{gm}=1$ if $y_{gm}>0$; and $z_{gn}=0$ if $y_{gm}=0$) to indicate if gene $g$ is expressed in cell $m$. Then, we defined the gene expression binary vector as ${\mathbf{z}}_g={\left[z_{g1},\dots z_{gm},\dots z_{gM}\right]}^{\mathrm{T}}$ $(m=1,\dots M)$ for a gene $g$. Based on the logistic regression function, we modeled the expression binary values of gene $g$:

$$Logit\left(p\left({\mathbf{z}}_g=1\right)\right)={\mathbf{X}\mathbf{w}}_{\mathit{g}},$$ (1)

where $\mathbf{X}=\left[{\mathbf{x}}_1,\dots {\mathbf{x}}_{\mathit{l}},\dots {\mathbf{x}}_{\mathit{L}}\right]$ is the design matrix, ${\mathrm{x}}_l$ is a column vector of user-defined feature (e.g. condition) with dimension of $M$, and ${\mathbf{w}}_g={\left[w_{g1},\dots w_{gl},\dots w_{gL}\right]}^{\mathrm{T}}$ is the coefficient vector. We also included cellular detection rate (CDR) [15] as a column of feature in X for each cell, which is defined as ${\mathbf{x}}_{\mathit{l}}={\left[CD{R}_1,\dots CD{R}_m,\dots CD{R}_M\right]}^{\mathrm{T}}$, where $CD{R}_m=(1/N){\sum }_{g=1}^N{z}_{gm}$ ($N$ is the total number of genes).

Then, we examined if the coefficient of “condition” is zero through Wald test. A large p-value ($>0.05$) indicates that the condition is not a meaningful predictor of the outcomes. That is to say the “condition” cannot determine whether the gene $g$ is expressed, due to many zero counts. Therefore, for this gene, we ignored zero counts while computing the difference between the two distributions of the two conditions using EMD (section sub2.3). However, for the remaining genes with low p-values, we imputed the zeros as described in section sub2.2 and then computed the difference between the two distributions of the two conditions using EMD.

2.2. Imputing zero counts

We employed a data imputation method using Lasso regression to estimate nonzero values for the zero counts for the genes that their zero counts cannot be ignored, as described in the previous subsection. Consider the set of genes that need to be imputed, determined by the regression analysis and Wald test, as $S_{IP}$, where $IP$ represents the number of genes that need to be estimated. We defined the gene detection rate $\left(GDR\right)$ for gene $g$ as $GD{R}_g=(1/M){\sum }_{m=1}^M{z}_{gm}$, where $z_{gm}$ indicates if gene $g$ is expressed in cell $m$ (i.e. $z_{gm}=1$ if $y_{gm}>0$; and $z_{gm}=0$ if $y_{gm}=0$) and M is the number of cells. Then, we selected the genes with high GDR ($\ge 80\mathrm{\%}$) as $S_G$, to be used for the regression analysis. The goal is imputing the zero values of genes from set $S_{IP}$ in each cell.

To impute the expression value of gene $g$ in cell $m$, a Lasso model is trained using the expression values of the genes that are correlated with gene $g$. In this method, first, we selected the top $s$ ($s$ can be set to a number, such as $s=150$, or a cutoff value can be used to set $s$) correlated genes from set $S_G$ by calculating Pearson correlation coefficient with gene $g$. Let $S_{G_g}$, be a subset of $S_G\left(S_{G_g}\subseteq S_G\right)$, to represent a set of genes that are correlated with gene $g$. In this step, only the corresponding cells that have nonzero expression values for gene $g$ are used to calculate the correlation coefficient. Then, a Lasso model for gene $g$ in cell $m$ is fitted as:

$$\mathrm{m}\mathrm{i}\mathrm{n}\left\{\frac{1}{\left|S_{G_g}\right|}{‖{\mathbf{y}}_{G_g,m}-{\mathbf{Y}}_{G_g,-m}{\mathbf{\beta }}_{gm}‖}_2^2+\lambda {‖{\mathbf{\beta }}_{gm}‖}_1\right\},$$ (2)

where $\left|S_{G_g}\right|$ is the number of genes in set $S_{G_g},{\mathbf{y}}_{G_g,m}$ is the observed vector of expression values in cell $m$ with $G_g$ elements; ${\mathbf{Y}}_{G_g,-m}$ is the design matrix with $G_g$ rows and ($M-1$) columns; $\lambda$ is the regularization parameter; and ${\mathbf{\beta }}_{gm}$ is the regressed coefficient vector with ($M-1$) rows. Then ${\widehat{y}}_{gm}$, the expression value of gene $g$ in cell $m$, is estimated as:

$$\left\{\begin{array}{ll}{\widehat{y}}_{gm}={\mathbf{y}}_{g,-m}\cdot {\mathbf{\beta }}_{gm}& \text{if}{y}_{gm}=0\text{and}g\in S_{IP}\\ {\widehat{y}}_{gm}=y_{gm}& \text{if}{y}_{gm}\ne 0\text{or}g\notin S_{IP}\end{array},\right.$$ (3)

where ${\mathbf{y}}_{g,-m}$ is the row vector of expression values of gene $g$.

Thus, for the genes belong to set $S_{IP}$, we can apply the EMD method for differential expression analysis, using the imputed expression values.

2.3. Computing EMD score for differential expression analysis

We used a nonparametric method based on the EMD [28,29] for differential analysis of expression values in two conditions. EMD measures the distance between the distributions of expression values or normalized read counts [27]. Suppose the genes are under two different conditions $P$ and $Q$, which include $M_1$ cells and $M_2$ cells, respectively. Then, we can obtain the normalized histograms (or densities) $P=\left\{\left(p_1,w_{p1}\right),\dots \left(p_i,w_{pi}\right),\dots \left(P_{M_1},w_{p{M}_1}\right)\right\}$ and $Q=\left\{\left(q_1,w_{q1}\right),\dots \left(q_j,w_{qj}\right),\dots \right.\left.\left(q_{M_2},w_{q{M}_2}\right)\right\}$, where $p_i$ is the center of each histogram bin in condition $P$, and the weight $w_{pi}$ is the frequency of each histogram bin. $q_j$ and $w_{qj}$ hold the same definition in condition $Q$. The cost of moving from expression value $p_i$ to $q_j$ is defined as the Euclidean distance between them, $d_{ij}=‖p_i-q_j‖$. The flow $f_{ij}$ denotes the portion of the bin i’s weight needed to be moved to bin $j$ to make these two bins even. The total cost is calculated as:

$$COST\left(P,Q,F\right)=\sum _{i=1}^{M_1}\sum _{j=1}^{M_2}{f}_{ij}{d}_{ij}.$$ (4)

The goal is to find a flow $\mathrm{F}=\left[f_{ij}\right]$ to minimize the total cost. After that, the EMD score is defined as the normalized minimum COST:

$$EMD\left(P,Q\right)=\frac{\mathrm{m}\mathrm{i}\mathrm{n}COST}{\sum _{i=1}^{M_1}\sum _{j=1}^{M_2}{f}_{ij}}.$$ (5)

Efficient algorithms to compute the EMD scores are fully described in [28] and [29]. In this work, to compute the EMD score we used the emd2d from the emdist R package [30], which is implemented based on the algorithm in [28].

2.4. Obtaining p-values for EMD scores

We used a permutation test to calculate the p-values of EMD scores. The null hypothesis is that the expression levels of each gene are the same under different conditions. In each permutation iteration, cells are randomly divided into two groups of $M_1$ and $M_2$ cells from the total $M$ cells. Then the EMD score for each gene is calculated in each permutation. For gene $g$, given $EM{D}_g=\left[em{d}_1,\dots em{d}_k,\dots em{d}_K\right]$, a vector of $K$ permuted EMD scores, and the observed score, $em{d}_{obs}$, the $p$-value for gene $g$ is calculated as:

$$p\text{-value}=\frac{\text{numbers}\text{of}em{d}_k\ge em{d}_{abs}}{K}.$$ (6)

After applying the permutation test to all the genes, we obtained a vector of p-values. To decrease the incorrectly rejected null hypothesis under a default threshold, we adjusted the p-values by FDR control [31]. The genes with adjusted p-values under a predefined threshold, such as 0.05, are chosen as significantly DE genes.

2.5. Reducing false positives using genes neighbor matrix

As empirical evidence shows that genes in the same pathway tend to correlate in expression [24–26], the network structure of genes can boost differential expression analysis.

We retrieved the BioGRID [32] gene interaction database and formed the neighbor adjacency matrix for all the genes with binary value, where zero means there is no connection between a pair of genes and one means there is a connection. Then for each gene, u₀ denotes the number of its neighbors with state DE and u₁ denotes the number of its neighbors with state nonDE. The new state of a gene is determined by the state of its majority neighbors.

3. Datasets

In this work, we used both simulated datasets and real datasets to evaluate the performance of the SigEMD for differential expression analysis in scRNAseq data.

3.1. Simulated datasets

Simulated data were employed to assess the detection accuracy. We generated simulated data using the simulation function from the scDD [17] package in R programming language [33]. The parameters were set as default in the R package. 10,000 genes were generated across two conditions within 75 cells for each. 1000 of these 10,000 genes were generated as DE genes and equally divided into four types of scenarios: 1) unimodal distributions with different means (DU), 2) bimodal distributions with different proportions of cells of each mode and the same modes across the two conditions (DP), 3) one unimodal distribution and one bimodal distribution with the same mode between one of the mode of the bimodal distribution and the unimodal distribution (DM), 4) one unimodal distribution and one bimodal distribution with no common mode (DB), which can be seen in Fig. 1a–d. The remaining 9000 genes were generated as nonDE genes. Specifically, half of the 9000 genes were generated from the same unimodal distribution (EE); and the remaining half were generated from the same NB model with two components (EP).

Fig. 1.

Plots of gene expression values in the simulated data. Density plots of expression values of DE genes with four different types of scenarios: a) DU, b) DP, c) DM, and d) DB under the two conditions represented by red and blue lines. e) The cumulative distribution function of percentage of zero counts of each gene in the simulated data and the positive control real data. (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)

Then, large amounts of zero counts were generated to simulate the drop-out event. First, we built the cumulative distribution function (CDF) of the percentage of zeros of each gene using the real data, $F_X(x)$. Then in the simulated data for each gene, we randomly selecteds $c(c\sim F_X(x)$) cells from the total cells and forced their expression values to zero. Thus, the CDF of the percentage of zeros of each gene is similar between the simulated data and the real data (Fig. 1e). In this way, the distribution of the total counts in the simulated data is more similar to that of the real data, which enables us to assess the true positives (TPs) and false positives (FPs) more accurately.

3.2. Real scRNAseq datasets

A real scRNAseq dataset provided by [34] is used as the positive control dataset to validate TPs. It consists of 22,928 genes from 48 mouse Embryonic Stem Cells and 44 mouse Embryonic Fibroblasts. The count matrix is available in GEO database with accession number GSE29087. To validate the TPs detected by the proposed method, we used the top 1000 DE genes from [35] as a gold standard, which are also used in [9,19,20].

The dataset from [36] is used as the negative control dataset to test FPs of the proposed method. 80 samples that are obtained under the same condition were retrieved from GEO database with accession number GSE54695. To apply gene interaction network to reduce the FPs, we only selected the common genes appeared in both the GSE54695 dataset and the BioGRID database. Therefore, the negative control dataset is finally filtered into 80 samples and 7277 genes in each sample. By randomly sampling from the 80 samples, ten datasets were generated to obtain statistical characteristics of the results. For each generated dataset, we randomly selected 40 cells out of the 80 cells as one group and considered the remaining 40 cells as the other group as in [20]. Since all the samples are under the same condition, there exist no DE genes in these ten independent datasets.

For preprocessing the real datasets, we filtered out the genes that are not expressed in all the cells (zero read counts across all the cells) and used the log-transformed transcript per millions (TPM) values as the input to SigEMD afterwards.

4. Results and discussions

4.1. Results on simulated datasets

Because true DE genes are unknown in real datasets, we used simulated data to evaluate the accuracy of detecting DE genes. The simulated data were explained in the Datasets section. For simulated data, we did not employ gene interaction network information to adjust DE gene status. We compared our method with those of three differential expression analysis methods that are developed for scRNAseq data: MAST [15], SCDE [9], and D3E [18] in terms of sensitivity $(=\text{TP}/(\text{TP}+\text{FN}))$, false positive rate $(\text{FPR}=\text{FP}/(\text{FP}+\text{TN}))$, accuracy $(=(\text{TP}+\text{TN})/(\text{TP}+\text{TN}+\text{FP}+\text{FN}))$, and precision $(=\text{TP}/(\text{TP}+\text{FP}))$. For the utilization of D3E, we applied the fast mode (i.e. −m 0) and Cramer-von Mises test (i.e. −t 0) in the Python package. The results of the methods for an adjusted p-value threshold of 0.05 are shown in Table 1.

Table 1.

Detection accuracy for DE genes in simulated data for adjusted p-value threshold = 0.05.

Method	Sensitivity	False positive rate	Accuracy	Precision
SigEMD	0.575	0.0056	0.953	0.920
MAST	0.386	0.0013	0.937	0.971
D3E	0.344	0.0415	0.897	0.480
SCDE	0.335	0.0002	0.933	0.995

Using an adjusted p-value threshold of 0.05, it can be observed that the proposed method outperforms other methods in terms of TPRs (i.e. the number of genes called significant by each method over the known ground truth DE genes). SigEMD has better sensitivity than those of the model-based methods, but it introduces FPs resulting in higher FPRs. Model-based methods, SCDE and MAST, perform better in terms of FPRs, but they detect much fewer TPs than SigEMD method does, as expected. This is because these methods are based on Poisson and NB models and cannot capture intrinsic multimodalities of data. Since SigEMD combines data imputation, statistical model, and nonparametric EMD to identify DE genes, it has much lower FPRs than D3E while provides much better TPRs at the same time. The receiver operating characteristic (ROC) curve in Fig. 2 shows that the area under the curve (AUC) of SigEMD is close to 1, indicating that it has the best detection accuracy in overall. Furthermore, we computed true positive rates for the four differentially expressed scenarios (i.e. DU, DM, DP, and DB) and true negative rates for the two non-differentially expressed scenarios (i.e. EP and EE). The results, as shown in Fig. 3, indicate that SigEMD outperforms other methods in capturing modalities. For the DP and DB scenarios where the level of multimodality is high, model based methods (MAST, SCDE) fail in detecting DE genes.

Fig. 2.

ROC curves for detecting DE genes using simulated data.

Fig. 3.

The percentage of a) true positive genes and b) true negative genes for different scenarios in the simulated data.

4.2. Discussions about the effect of sample size

To investigate the effect of sample size on detecting DE genes, we generated eight cases with 10, 30, 50, 75, 100, 200, 300, and 400 cells for each condition. Sensitivities, FPRs, accuracies and precisions of detecting DE genes, using the four methods are shown in Fig. 4. From Fig. 4 we can observe that the number of the truly identified DE genes tends to increase as the sample size increases. The sensitivity and accuracy of SigEMD increases significantly when the sample size is larger than 100 in each condition and reach to their plateau when the sample size is around 300. In addition, results show that SigEMD could obtain a nearly 100% accuracy when the sample size is large enough. FPR and precision for SigEMD remains steady when the sample size is larger than 50.

Fig. 4.

Effect of sample size on detecting DE genes in terms of a) TPR, b) FPR, c) accuracy, and d) precision. The sample size is in horizontal axis, from 10 cells to 400 cells in each condition. A threshold of 0.05 is used for FDR and adjusted P-value.

4.3. Discussions about the running time for computing p-values

We examined the running time for computing EMD scores and their p-values. Since SigEMD calculates p-values by a permutation test, its running time depends largely on the number of the permutation iterations and the number of genes. We can achieve more accurate p-values by employing more permutation iterations, and as a consequence improve the FPR. However, running time grows linearly with increasing the number of iterations. In our simulation, using 500-iteration results in a lower FPR (0.0041); however, the running time of 500 iterations is nearly five times longer than the running time of 100 iterations (13.53 min for 100 iterations and 68.6 min for 500 iterations, using 75 cells for each condition and 10,000 genes and running on personal computer, iMac with 3.1 GHz CPU and up to 8 gigabytes memory). The running time also increases linearly with the number of genes. The runtimes of SigEMD (100 iterations, no imputation) and the other tools are shown in Table 2 for a sample size of 75 cells in each condition. As expected, SigEMD takes more time than model-based methods, MAST and SCDE; but it is faster than D3E, which is a nonparametric method as well.

Table 2.

Runtime of the four methods for a sample size of 75 cells for each condition.

Method	Runtime
MAST	1.53 min
SCDE	9.88 min
SigEMD	13.53 min
D3E	40.56 min

4.4. Results on real datasets without gene interaction network

We applied the proposed method, SigEMD, to the real datasets that we explained in the previous section. We compared the results of the proposed method with the results of MAST, SCDE, and D3E. Using the “positive control” datasets [34], we observed that the top significantly DE genes varied among the methods. Under a threshold of adjusted p-value < 0.05, among the total filtered 14,900 genes (filtering the genes with zero counts across all the cells), MAST identified 734 DE genes, SCDE identified 2414 DE genes, SigEMD identified 3702 DE genes, and D3E identified 8401 DE genes. Fig. 5a shows the Venn diagram of significantly DE genes across these four methods. As can be seen in Fig. 5a, there are only 405 genes common across these tools.

Fig. 5.

Venn Diagram using real data. a) Venn Diagram of the DE genes called by D3E, SigEMD, MAST, and SCDE under an adjusted p-value of 0.05. b) Venn diagram of the DE genes across the gold standard, SigEMD, and the other three methods under an adjusted p-value of 0.05.

We also compared the significantly DE genes identified by the methods with the 1000 gold standard genes that we explained in the previous section. Among the common 405 DE genes across the methods, only 142 genes intersected with the 1000 gold standard genes. SigEMD detected 488 genes, MAST detected 198 genes, SCDE detected 392 genes, and D3E detected 722 genes in common with the 1000 gold standard genes (Fig. 6). We also showed the Venn diagram of the significantly DE genes across the gold standard, SigEMD, and the other three methods separately in Fig. 5b. The number of common genes with the 1000 gold standard genes is considered as the TPs. Therefore, the sensitivities (or true positive rates) of the MAST, SCDE, SigEMD, and D3E are 0.198, 0.392, 0.488, and 0.722, respectively as shown in Fig. 6. The blue bar plots in Fig. 6 show the intersection between the gold standard genes and the significantly DE genes called by the methods, whereas the yellow bar plots show the number of significantly DE genes that are not among the gold standard genes. Compared with the model-based methods, MAST and SCDE, the nonparametric methods, SigEMD and D3E, performed better in terms of sensitivity. This is because nonparametric methods can capture multimodality inherited in scRNAseq data while model-based methods cannot. The precision of the methods, that is the number of TP genes (in blue bar) over the total number of detected significantly DE genes are 0.27, 0.16, 0.13 and 0.086, for MAST, SCDE, SigEMD, and D3E, respectively. Even though the sensitivity of the SigEMD is worse than the sensitivity of the D3E (better than the sensitivities of the others), SigEMD generates much less false positives. Therefore, it has higher precision compared to D3E.

Fig. 6.

Intersection and no intersection between selected genes called significant by the methods and the 1000 gold standard genes.

We need to mention that we do not have the real true positive DE genes for the positive control dataset. The 1000 gold standard genes are DE genes from the dataset that are validated through qRT-PCR experiment [35]. This dataset has similar conditions as those of the positive control dataset. However, they are not from the same assay and experiment. Therefore, the results we showed here provide information about precision and sensitivities to some degree. To compute true sensitivities and precision, we used simulated data as was discussed in the previous subsection.

We conducted a pathway enrichment analysis of the DE genes detected by the tools using the positive control datasets. We used the “Investigate Gene Sets” function of the web-based GSEA software tool (http://www.broadinstitute.org/gsea/msigdb/annotate.jsp) to examine whether or not the identified genes are biologically meaningful. We investigated the KEGG database (containing 186 gene sets) from the Molecular Signatures Database (MSigDB) for the gene set enrichment analysis (FDR threshold of 0.05). Top 1000 identified DE genes of each tool were used as the input for the KEGG pathway enrichment analysis. Out of the 196 gene sets, 58, 56, 84, 87 gene sets are enriched by the top 1000 DE genes identified by MAST, SCDE, D3E, and SigEMD, respectively. More KEGG pathways with lower FDRs are enriched by the 1000 top-ranked DE genes from SigEMD. This indicates that SigEMD is a powerful approach to identify biologically meaningful DE genes. Box plot in Fig. 7 shows the comparison of FDRs of the top ten most significant gene sets enriched by the 1000 top-ranked DE genes from the tools. Many of the top enriched gene sets are biologically relevant and are common across the tools. However, the top-ranked DE genes from SigEMD yield stronger significant score for highly relevant gene sets. For example, FDRs for the KEGG Regulation of Actin Cytoskeleton gene set are 4.55E-6, 2.10E-4, 1.44E-20, and 3.38E-24 for MAST, SCDE, D3E, and SigEMD, respectively. The ten top-ranked KEGG pathways for the four tools are listed in Supplementary Tables S1–S4.

Fig. 7.

Box plot of FDRs for the top 10 KEGG pathways enriched by the 1000 top-ranked DE genes, using GSEA.

To validate the FPs, we used the negative control dataset from [36] and applied the methods to the ten randomly sampled datasets (explained in the Datasets section). Since cells in the two groups are from the same condition, we expect the methods do not identify any DE gene. MAST and SCDE did not call any gene as DE gene as expected. SigEMD and D3E, respectively identified 44.6 and 160 significantly DE genes, in average over the ten datasets out of 7277 genes (FPRs of 0.006 and 0.022 for SigEMD and D3E, respectively).

4.5. Results on real datasets adjusted by the gene interaction network

To improve the false positive rates of the nonparametric methods, we applied the gene interaction network adjustment method, described in Section 2.5, to the results of SigEMD and D3E. After employing the information of gene interaction network, the number of FPs decreased significantly. SigEMD identified 1.5 genes and D3E identified 51 genes out of 7277 genes as significantly DE genes. Thus, employing the gene interaction network can make a significant contribution to reducing the FPs. However, it can reduce the TPs as well. We also applied the gene interaction network adjustment method to the results from the positive control dataset. After the adjustment, we identified 3316 and 5925 genes as differentially expressed for SigEMD and D3E, respectively, 418 and 589 of which (compared to 488 and 722) are common with the 1000 gold standard genes.

4.6. Discussions about efficiency of imputation

We compared the sensitivity and FPR of detecting DE genes using SigEMD with and without imputation. Using imputation is optional in the SigEMD software. First, we tested the simulated datasets. The average sensitivity and FPR are 0.563 and 0.011, respectively, when we do not employ the imputation of the zero counts. However after imputation, the sensitivity and FPR improve to 0.575 and 0.0055, respectively. Then, we tested the real datasets. In the positive control dataset, the sensitivity (the number of common genes respect to the 1000 gold standard genes) is 0.479 without imputation; and it increases to 0.488 when applying the imputation method. In the negative control dataset, the average FPRs are 0.012 and 0.006, without and with employing the imputation method, respectively. Therefore, the imputation of the zero counts of the genes that their zero counts have an impact on the evaluation of DE genes, as described in Section 2.2, improves the accuracy of the differential gene expression analysis in terms of sensitivities and FPRs; and has a stronger effect on FPRs. We need to mention that employing imputation, especially when the number of genes that their zero counts have an impact is large, increases the overall runtime.

5. Conclusion

In this study, we proposed a novel method, SigEMD, for identifying DE genes in scRNAseq data. We developed this method to address the intrinsic challenges of data multimodality, large amounts of zero counts, and sparsity of scRNAseq data in differential expression analysis. The method is based on the combination of data imputation, a logistic regression model and a nonparametric differential analysis method. The logistic regression model and data imputation are used to handle the zero counts’ problem and the nonparametric method is used for differential analysis of heterogenous scRNAseq data. Specifically, the logistic regression models the zero counts in the gene expression values and the Wald test determines whether or not the zero counts should be considered in the differential analysis. After Wald test selection of a set of genes that need to be imputed, the method employs a data imputation method to estimate zero counts. Then, the nonparametric method for the differential analysis, which is based on the Earth Mover’s Distance (EMD), compares the overall difference between the distributions of expression values in different conditions. In addition, to further reduce the false discovery rates, we propose to adjusted the final gene expression states by utilizing the gene interaction network information.

We used simulated data to further validate the performance of SigEMD in terms of detection accuracy, precision, sensitivity, and false positive rate. Results indicate that SigEMD has an overall powerful performance compared with other methods. Using “positive control” real data, SigEMD yields higher true positive rates compared to model-based methods, and enhances more the detection precision compared to nonparametric methods. Using a “negative control” real data to investigate false positive rates indicates that even though the FPR of SigEMD is lower compared to the other nonparametric method, it is slightly higher than those of model-based methods. We also illustrated the effectiveness of employing gene interaction network in reducing false positives.

SigEMD, a combination of statistical model, data imputation, and nonparametric EMD, has strengths and weaknesses. Unlike the model-based methods, it does not make any assumptions about the distributions of the gene expression values. As a result it can capture the heterogeneous and multimodal pattern of gene expression values. Another advantage is that it decreases the negative impact of large amounts of zero counts by utilizing a logistic regression model and imputing the zeros. However, as SigEMD compares the overall distributions based on histograms, it needs a moderate sample size (more than 50 cells per class). Moreover, since SigEMD calculates p-values by a permutation test, the number of the permutation iterations has also an impact on the detection accuracy and the runtime of the method; the larger the permutation iteration number, the more accurate the p-value and the longer the runtime are. We will continue working on improving the running time, the p-value computation algorithm and reducing FPs.

Supplementary Material

Appendix A. Supplementary data

Supplementary data associated with this article can be found, in the online version, at http://dx.doi.org/10.1016/j.ymeth.2018.04.017.

Abbreviations:

scRNAseq

single cell RNA sequencing

NGS

next generation sequencing

RNAseq

RNA sequencing

DE

differentially expressed

EMD

Earth Mover’s Distance

NB

negative binomial

DU

differential unimodal

DM

differential modality

DP

differential proportion

DB

both DM and DU

CDR

cellular detection rate

GDR

gene detection rate

FDR

false discovery rate

TPM

transcript per million

CDF

cumulative distribution function

ROC

receiver operating characteristic

TPR

true positive rate

FPR

false positive rate

TP

true positive

FP

false positive

TN

true negative

FN

false negative

AUC

area under curve