Model-based dimensionality reduction for single-cell RNA-seq using generalized bilinear models

Abstract

Dimensionality reduction is a critical step in the analysis of single-cell RNA-seq (scRNA-seq) data. The standard approach is to apply a transformation to the count matrix followed by principal components analysis (PCA). However, this approach can induce spurious heterogeneity and mask true biological variability. An alternative approach is to directly model the counts, but existing methods tend to be computationally intractable on large datasets and do not quantify uncertainty in the low-dimensional representation. To address these problems, we develop scGBM, a novel method for model-based dimensionality reduction of scRNA-seq data using a Poisson bilinear model. We introduce a fast estimation algorithm to fit the model using iteratively reweighted singular value decompositions, enabling the method to scale to datasets with millions of cells. Furthermore, scGBM quantifies the uncertainty in each cell’s latent position and leverages these uncertainties to assess the confidence associated with a given cell clustering. On real and simulated single-cell data, we find that scGBM produces low-dimensional embeddings that better capture relevant biological information while removing unwanted variation.

1. Introduction

Single-cell RNA sequencing (scRNA-seq) is a revolutionary technology that allows gene expression to be profiled at the level of individual cells (Saliba et al., 2014). This enables the identification of novel cell types that play critical roles in biological processes. However, the increased resolution provided by scRNA-seq comes at the cost of introducing several statistical and computational challenges. One major challenge is the large size of scRNA-seq datasets, which often contain millions of cells (Cao et al., 2019); thus, new methods must be computationally scalable (Lähnemann et al., 2020). Another challenge is that the extreme sparsity and discreteness of scRNA-seq count data make traditional statistical models based on normal distributions inappropriate (Vallejos et al., 2017). A third major challenge is that cell-level measurements are noisy and contain relatively little information compared to bulk sequencing, making it important to quantify the uncertainty in these measurements and propagate it to downstream analyses (Lähnemann et al., 2020).

Due to the large size of single-cell datasets, it is standard practice to use a dimensionality reduction technique such as principal components analysis (PCA) before clustering and other downstream analyses (Luecken and Theis, 2019). Researchers typically apply a transformation to the count matrix before running PCA, since applying PCA directly to the count matrix can introduce strong technical artifacts. Unfortunately, commonly used transformations such as $\mathrm{l}\mathrm{o}\mathrm{g}(1+x)$ can still lead to substantial biases in the subsequent PCA results (Townes et al., 2019). This has led to the current leading approach of transforming the counts using the Pearson residuals of a probabilistic model, as done by scTransform (Hafemeister and Satija, 2019). However, we find that even scTransform leads to undesirable biases that can mask true signals and upweight technical artifacts. Indeed, it seems unlikely that there exists a transformation that satisfies the necessary requirements simultaneously (Van Eeuwijk, 1995).

Alternatively, some methods perform dimensionality reduction directly using a probabilistic model of the count data matrix. These methods can avoid the artifactual biases of simple transformations and, further, can provide principled uncertainty quantification for downstream analyses and visualization. The GLM-PCA method of Townes et al. (2019) models the entries of the count matrix using a Poisson or negative-binomial distribution and estimates latent factors in the log space, however, GLM-PCA suffers from slow runtime and convergence issues on single-cell datasets with millions of cells (Lause et al., 2021). ZINB-WAVE (Risso et al., 2018) employs a similar model that assumes a zero-inflated negative-binomial distribution for the counts, but it has been noted that ZINB-WAVE can take days to run on large datasets, and furthermore, several studies have shown that the distribution of UMI counts is not zero inflated (Svensson, 2020; Sarkar and Stephens, 2021; Townes et al., 2019). This has prompted the development of the faster version, NewWave (Agostinis et al., 2022), however, we find that even NewWave is computationally burdensome on large datasets. The scVI method of Lopez et al. (2018) uses a variational autoencoder to learn a low-dimensional embedding, but a disadvantage is that this embedding may not be easily interpretable (Svensson et al., 2020), in contrast to PCA where the embedding is defined by linear combinations of genes.

An advantage of a model-based approach to dimensionality reduction is that one can, in principle, quantify uncertainty in the low-dimensional representation of cells. Understanding the confidence in each cell’s latent position is useful for discerning whether the clusters represent biologically distinct groups as opposed to technical or computational artifacts. However, to the best of our knowledge, none of the existing methods use uncertainty in the low-dimensional embedding to provide uncertainty quantification in downstream analyses.

In this paper, we introduce scGBM, a novel approach to dimensionality reduction for scRNA-seq data. Starting from the same underlying model as in GLM-PCA (Townes et al., 2019), we provide three key innovations. First, we develop a new estimation algorithm that is faster than existing approaches and scales up to datasets with millions of cells. Second, we quantify uncertainty in the low-dimensional embedding, enabling calibrated inference in downstream analyses. Third, we use these uncertainties to define a cluster cohesion index (CCI) that measures the stability of each cluster and the relationships between clusters. The CCI provides a quantitative tool for assessing which clusters represent biologically distinct populations, rather than simply being artifacts of sampling variability. Further, when analyzing numerous clusters, the CCIs reveal relationships that would be difficult to see with scatterplots. We evaluate the performance of scGBM on real and simulated data, finding that in many cases the current leading approaches are unable to capture true biological variability, while scGBM is successful in doing so.

The article is organized as follows. In Section 2, we demonstrate some of the limitations of current leading methods for scRNA-seq dimensionality reduction. In Section 3, we introduce our proposed methodology. Section 4 contains empirical results comparing the performance of our method versus leading approaches on simulated and real data. We conclude in Section 5. An scGBM R package is available for download.¹

2. Motivation

It is known that fundamental issues can arise from the commonly used approach of running PCA on $\mathrm{l}\mathrm{o}\mathrm{g}(1+x)$ transformed scRNA-seq data. For instance, Townes et al. (2019) showed that the first principal component is strongly correlated with the number of zeros per cell, even on null data. Methods using count models have been developed to address these issues, however, we find that the current leading approaches still have significant limitations. One of the most popular methods is scTransform (Hafemeister and Satija, 2019), implemented in the widely used Seurat package (Satija et al., 2015). Let $Y\in {\mathbb{R}}^{I\times J}$ be the matrix of unique molecular identifier (UMI) counts, where $I$ is the number of genes and $J$ is the number of cells. The idea behind scTransform is to apply PCA to the matrix of Pearson residuals obtained by fitting negative-binomial GLMs to the count matrix $Y$. Specifically, for each row $i$, scTransform fits the following model to data $Y_{i1},\dots ,Y_{iJ}$:

$$\begin{gathered} Y_{ij}~\mathrm{N}\mathrm{e}\mathrm{g}\mathrm{B}\mathrm{i}\mathrm{n}\mathrm{o}\mathrm{m}\left({\mu }_{ij},{\alpha }_i\right) \\ \mathrm{l}\mathrm{o}\mathrm{g}\left({\mu }_{ij}\right)={\beta }_{0i}+{\beta }_{1i}\mathrm{l}\mathrm{o}\mathrm{g}\left(S_j\right) \end{gathered}$$ (2.1)

where $S_j={\sum }_{i=1}^I{Y}_{ij}$ and $\mathrm{N}\mathrm{e}\mathrm{g}\mathrm{B}\mathrm{i}\mathrm{n}\mathrm{o}\mathrm{m}(\mu ,\alpha )$ is the negative-binomial distribution with mean $\mu$ and variance $\mu +{\mu }^2/\alpha$. Then, PCA is applied to the matrix $Z=\left[Z_{ij}\right]\in {\mathbb{R}}^{I\times J}$ where

$$Z_{ij}:=\frac{Y_{ij}-{\hat{\mu }}_{ij}}{\sqrt{{\hat{\mu }}_{ij}+{\hat{\mu }}_{ij}^2/{\hat{\alpha }}_i}}$$ (2.2)

are the Pearson residuals and ${\hat{\mu }}_{ij}$ and ${\hat{\alpha }}_i$ are regularized versions of the estimated parameters for the model in Equation 2.1. Lause et al. (2021) argued that the scTransform model (Equation 2.1) is overspecified since $\mathrm{l}\mathrm{o}\mathrm{g}\left(S_i\right)$ can be interpreted as an offset which would imply that ${\beta }_{1i}=1$. Additionally, Lause et al. (2021) showed that the overdispersion estimates $\hat{\alpha }$ can have significant bias. We demonstrate potential shortcomings of scTransform using (i) a simulation with three cell types differentiated by single marker genes, and (ii) a simulation with two cell types differentiated by log-fold changes of equal magnitude across all genes.

Single marker genes simulation

In simulation (i), we generated data for $J=1000$ cells from three cell types, where cell types A and B are each distinguished by overexpressing a single marker gene. Specifically, relative to cell type C, cell type A overexpresses gene 1 and cell type B overexpresses gene 2, while the remaining 998 genes have identical mean expression across the three cell types (Figure 1a); see Section S3 for full details of the simulation. We applied scTransform to this dataset and found that, in terms of PCs 1 and 2, scTransform does not reveal the distinction between these three cell types (Figure 1b,c). The reason why this occurs is that scTransform implicitly standardizes the scale of the genes by dividing by the standard deviation, so all genes end up getting similar weight in the PCA. Additionally, the overdispersion parameter ${\alpha }_i$ absorbs some of the latent variability in the first two genes, further reducing the signal. This is illustrated in Figure 1d, which shows that the variance of the Pearson residuals from the 998 noise genes is comparable to (and, in fact, larger than) that of the two marker genes. Thus, the normalization procedure used by scTransform upweights the noise genes and downweights the signal genes.

Figure 1: Limitations of scTransform.

a. Single marker genes simulation: Boxplots of simulated counts for 1000 cells from 3 cell types. Gene 1 is overexpressed in cell type A and gene 2 is overexpressed in cell type B. The remaining genes are Poisson(1) random noise. b. The scores obtained from applying PCA to the Pearson residuals computed by scTransform. c. The corresponding weight of each gene in the first principal component. d. Boxplots of Pearson residuals for gene 1, gene 2, and the remaining “noise” genes. e. Equal log-fold change simulation: Two cell types such that all genes have an absolute log₂ fold change of 1 between the two cell types. The weight given to each gene in the first principal component depends on the baseline expression $\mu$. f. Purified monocytes data: Applying scTransform to monocytes from 10X genomics, the genes with the largest baseline mean (in terms of log mean expression) tend to have the largest PC weights.

Equal log-fold change simulation

In simulation (ii), we generated $J=1000$ cells from two equally sized groups. In the first group, we randomly generated the mean of each gene $i=1,\dots ,I$ by sampling ${\mu }_i~\mathrm{E}\mathrm{x}\mathrm{p}\mathrm{o}\mathrm{n}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{i}\mathrm{a}\mathrm{l}\left(0.1\right)$ independently. In the second group, the mean of gene $i$ was randomly set to either $2{\mu }_i$ or ${\mu }_i/2$, with probability 1/2, independently. In other words, all genes are differentially expressed by a multiplicative factor (fold change) of 2 between the two groups of cells. Each count was generated as independent Poisson with the corresponding mean. Applying scTransform (Figure 1e), we see that the PC1 weights are strongly dependent on the base mean ${\mu }_i$. This is undesirable because, in real data, some genes tend to have higher (or lower) measured expression simply because they are more (or less) abundant due to baseline cell activity or due to technical gene-specific effects unrelated to biology. Moreover, the PC weights are commonly used as a measure of a gene’s influence or importance on the corresponding component (Soumillon et al., 2014; Petropoulos et al., 2016; Roden et al., 2006).

The reason why the peculiar dependence in Figure 1e occurs is that scTransform decomposes latent effects additively in the mean rather than in the log of the mean. Although scTransform adjusts for gene-specific and cell-specific intercepts in log space via $\mathrm{l}\mathrm{o}\mathrm{g}\left({\mu }_{ij}\right)={\beta }_{0i}+{\beta }_{1i}\mathrm{l}\mathrm{o}\mathrm{g}\left(S_j\right)$, the latent effects are decomposed in linear space because applying PCA to the Pearson residual matrix $Z$ is equivalent to finding an approximate low-rank factorization $U\Sigma V^T\approx Z$ where $U\in {\mathbb{R}}^{I\times M},V\in {\mathbb{R}}^{J\times M}$, and $\Sigma =\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{g}\left({\sigma }_1,\dots ,{\sigma }_M\right)$, that is,

$$\sum _{m=1}^M{\sigma }_m{u}_{im}{v}_{jm}\approx \frac{Y_{ij}-{\hat{\mu }}_{ij}}{\sqrt{{\hat{\mu }}_{ij}+{\hat{\mu }}_{ij}^2/{\hat{\alpha }}_i}}.$$ (2.3)

Purified monocytes data

To illustrate this issue on real data, we considered a dataset of $J=\mathrm{2,612}$ purified monocytes downloaded from the 10X genomics website (www.10xgenomics.com). Since these data come from a single cell type, we expect minimal true latent variation. Since the true mean is unknown, we computed the log mean count for each gene and plotted it against the PC1 weights obtained from scTransform (Figure 1f). Although the weights are more variable than in the simulated example, we again observe that the genes with the largest baseline mean expression tend to have the largest weights.

3. Methods

In this section, we introduce our proposed model (Section 3.1), iteratively reweighted singular value decomposition (Section 3.2), step-by-step estimation algorithm (Section 3.3), projection technique for scaling to large matrices (Section 3.4), and uncertainty quantification approach (Section 3.5).

3.1. scGBM fits a Poisson bilinear model to the count matrix

Our proposed method, referred to as scGBM, handles normalization, transformation, and latent factorization in a unified way using a Poisson bilinear model. Specifically, we model the entries of the UMI count matrix $Y\in {\mathbb{R}}^{I\times J}$ (with genes on rows and cells on columns) as

$$\begin{gathered} Y_{ij}~\text{Poisson}\left({\mu }_{ij}\right) \\ \text{log}\left({\mu }_{ij}\right)={\alpha }_i+{\beta }_j+\sum _{m=1}^M{\sigma }_m{u}_{im}{v}_{jm} \end{gathered}$$ (3.1)

for $i=1,\dots ,I$ and $j=1,\dots ,J$, where ${\alpha }_i$ is a gene-specific intercept, ${\beta }_j$ is a cell-specific intercept, $U:=\left[u_{im}\right]\in {\mathbb{R}}^{I\times M}$ is a matrix of latent factor loadings or weights, $V:=\left[v_{jm}\right]\in {\mathbb{R}}^{J\times M}$ is a matrix of latent factor scores, and $\Sigma :=\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{g}\left({\sigma }_1,\dots ,{\sigma }_M\right)\in {\mathbb{R}}^{M\times M}$ is a diagonal matrix of scaling factors (or singular values) ${\sigma }_m>0$. For intuition, this model can roughly be thought of as applying PCA inside the link function of a generalized linear model (GLM). As in PCA, we constrain the columns of $U$ to be orthonormal and order the columns such that ${\sigma }_1>\cdots >{\sigma }_M>0$, so the first factor captures the greatest amount of latent structure. Identifiability constraints and more background on the model are provided in Section S1.1.

Following Choulakian (1996), we refer to the model in Equation 3.1 as a generalized bilinear model (GBM), but other names such as GAMMI (Van Eeuwijk, 1995) or eSVD (Lin et al., 2021) have been used in the statistical literature. We refer to Miller and Carter (2020) for an extensive discussion of the literature on GBMs. In the single-cell literature, Equation 3.1 is typically referred to as GLM-PCA (Townes et al., 2019). Although scGBM is based on the same model as GLM-PCA, there are some key differences between the two methods. First, scGBM is able to estimate cell-specific intercepts ${\beta }_j$, whereas GLM-PCA treats the ${\beta }_j’\mathrm{s}$ as fixed offsets. Second, GLM-PCA does not enforce identifiability constraints in the intercepts or the factors. Finally, GLM-PCA does not quantify uncertainty in the parameter estimates.

A GBM that only includes intercepts and latent factors, as in Equation 3.1, is likely to be sufficient for most single-cell studies. However, in some cases, it is beneficial to control for known batches such as groups of samples sequenced in different locations or on different dates. To achieve this, scGBM can estimate a batch-specific intercept for each gene; see Section S1.5 for details. More generally, the model can be augmented to include arbitrary sample covariates and gene covariates, as well as interactions between these (Miller and Carter, 2020). However, in this paper, we only consider binary sample covariates since other types of covariates are rarely used during this stage of single-cell analyses.

In contrast to many bulk RNA-seq models, the scGBM model assumes a Poisson outcome and thus does not allow for overdispersion. In fact, there is increasing evidence that the technical sampling distribution of UMI counts is close to Poisson and that any additional dispersion is due to biological variability such as heterogeneous cell types or cell states (Sarkar and Stephens, 2021; Lause et al., 2021). Since the goal of scGBM is to remove technical variability while preserving all biological variability for downstream analyses, it is natural to use a Poisson outcome distribution.

3.2. Fast estimation using iteratively reweighted singular value decomposition

Existing methods for fitting GBMs are not scalable to the large datasets encountered in scRNA-seq. To address this limitation, we propose a new approach to fitting the Poisson GBM that combines iteratively reweighted least squares (IRLS) and singular value decomposition (SVD). This pairing is natural since IRLS is the standard way to fit GLMs and SVD is the standard way to perform PCA.

Define $X=U\Sigma V^T$, and suppose we have current estimates $\hat{\alpha },\hat{\beta }$, and $\hat{X}$. By a second-order Taylor approximation at these estimates (see Section S1.2 for details), maximizing the log-likelihood of the scGBM model in Equation 3.1 is approximately equivalent to solving the weighted low-rank problem

$$\underset{X:\text{rank}\left(X\right)=M}{\text{min}}\sum _{i,j}{W}_{ij}{\left(Z_{ij}-X_{ij}\right)}^2$$ (3.2)

where $W_{ij}={\hat{\mu }}_{ij}/{\hat{\mu }}_{*},{\hat{\mu }}_{*}={\mathrm{m}\mathrm{a}\mathrm{x}}_{ij}{\hat{\mu }}_{ij},Z_{ij}={\hat{X}}_{ij}+\left(Y_{ij}-{\hat{\mu }}_{ij}\right)/{\hat{\mu }}_{ij}$, and ${\hat{\mu }}_{ij}=\mathrm{e}\mathrm{x}\mathrm{p}\left({\hat{\alpha }}_i+{\hat{\beta }}_j+{\hat{X}}_{ij}\right)$. Unlike the unweighted case, there is not a closed-form solution to Equation 3.2, unfortunately. However, Srebro and Jaakkola (2003) introduced a simple and efficient method for solving this type of weighted low-rank problem (with fixed $W$ and $Z$) by iteratively applying the equation

$${\hat{X}}^{\text{(new)}}={\mathrm{S}\mathrm{V}\mathrm{D}}_M(\hat{X}+W\circ (Z-\hat{X}\left)\right)$$ (3.3)

where ${\mathrm{S}\mathrm{V}\mathrm{D}}_M(\cdot )$ denotes the truncated SVD of rank $M$ and ∘ is the Hadamard product. In turn, when $\hat{X}$ is held fixed, the maximum likelihood estimates of $\hat{\alpha }$ and $\hat{\beta }$ have closed-form solutions.

Thus, we alternate between updating $\hat{X}$ by applying Equation 3.3 with $\hat{\alpha }$ and $\hat{\beta }$ fixed, and updating $\hat{\alpha }$ and $\hat{\beta }$ to their maximum likelihood estimates with $\hat{X}$ fixed. The truncated SVD automatically factors $\hat{X}$ as $\hat{U}\hat{\Sigma }{\hat{V}}^T$, yielding estimates of the loadings, scores, and singular values. See Section 3.3 for a step-by-step description of the proposed algorithm, which we call iteratively reweighted singular value decomposition (IRSVD).

There are several advantages of the IRSVD algorithm. First, it is asymptotically faster than Fisher scoring, the technique used by Miller and Carter (2020) and Townes et al. (2019); see Section 3.3 for the computational complexity. Further, one can leverage special properties of Poisson GLMs to obtain vectorized updates for the intercepts—that is, the entries of $\alpha$ and $\beta$ can be updated simultaneously—which greatly reduces the runtime in practice. Finally, the identifiability constraints such as orthogonality are preserved at every iteration of the algorithm.

3.3. Estimation algorithm and computational complexity

Our proposed IRSVD estimation algorithm for the scGBM model in Equation 3.1 is as follows.

• Initialize. Set ${\hat{\beta }}_j\leftarrow \mathrm{l}\mathrm{o}\mathrm{g}\left({\sum }_i{Y}_{ij}\right),{\hat{\alpha }}_i\leftarrow \mathrm{l}\mathrm{o}\mathrm{g}\left({\sum }_j{Y}_{ij}\right)-\mathrm{l}\mathrm{o}\mathrm{g}\left({\sum }_j\mathrm{e}\mathrm{x}\mathrm{p}\left({\hat{\beta }}_j\right)\right),W_{ij}\leftarrow \mathrm{e}\mathrm{x}\mathrm{p}\left({\hat{\alpha }}_i+{\hat{\beta }}_j\right)$, and

  $$\hat{X}\leftarrow \mathrm{c}\mathrm{l}\mathrm{i}\mathrm{p}\left({\mathrm{S}\mathrm{V}\mathrm{D}}_M\left(\right(Y-W)/\mathrm{s}\mathrm{q}\mathrm{r}\mathrm{t}(W\left)\right)/\mathrm{s}\mathrm{q}\mathrm{r}\mathrm{t}\left(W\right)\right)$$ (3.4)

  where square roots, division, and $\mathrm{c}\mathrm{l}\mathrm{i}\mathrm{p}\left(x\right)=\mathrm{m}\mathrm{a}\mathrm{x}\left(\mathrm{m}\mathrm{i}\mathrm{n}\right(x,c),-c)$ are applied entry-wise, with $c=8$.
• Iterate the following steps until convergence.

  1. Update intercepts.

     $${\widehat{\alpha }}_i\leftarrow \text{log}\left(\sum _{j=1}^J{Y}_{ij}\right)-\text{log}\left(\sum _{j=1}^J\text{exp}\left({\widehat{\beta }}_j+{\widehat{X}}_{ij}\right)\right),$$ (3.5)

     $${\widehat{\beta }}_j\leftarrow \text{log}\left(\sum _{i=1}^I{Y}_{ij}\right)-\text{log}\left(\sum _{i=1}^I\text{exp}\left({\widehat{\alpha }}_i+{\widehat{X}}_{ij}\right)\right).$$ (3.6)
  2. Update latent factors. Compute ${\hat{\mu }}_{ij}\leftarrow \mathrm{e}\mathrm{x}\mathrm{p}\left({\hat{\alpha }}_i+{\hat{\beta }}_j+{\hat{X}}_{ij}\right),{\hat{\mu }}_{*}\leftarrow {\mathrm{m}\mathrm{a}\mathrm{x}}_{ij}{\hat{\mu }}_{ij},W_{ij}\leftarrow {\hat{\mu }}_{ij}/{\hat{\mu }}_{*}{,Z}_{ij}\leftarrow {\hat{X}}_{ij}+\left(Y_{ij}-{\hat{\mu }}_{ij}\right)/{\hat{\mu }}_{ij}$, and

     $$\hat{X}\leftarrow {\mathrm{S}\mathrm{V}\mathrm{D}}_M(\hat{X}+\rho W\circ (Z-\hat{X}\left)\right).$$ (3.7)

• Output. Return the estimates ${\hat{\alpha }}_i,{\hat{\beta }}_j,\hat{U},\hat{\Sigma }$, and $\hat{V}$ such that $\hat{X}=\hat{U}\hat{\Sigma }{\hat{V}}^T$.

For the convergence criterion, we terminate the algorithm when the relative change in log-likelihood is below a specified tolerance (default 10⁻⁴) or after a specified maximum number of iterations (default 100). As a default, we set $M=20$. We find that a step size of $\rho =1$ works well, but that the rate of convergence can be improved by using an adaptive scheme where $\rho$ is increased by a multiplicative factor of 1.05 when the log-likelihood increases and is decreased by a multiplicative factor of 1/2 when the log-likelihood decreases. In addition, the latent factors update (Equation 3.7) can be modified to use Nesterov acceleration (Tuzhilina and Hastie, 2021); see Equation S1.5. Therefore, we use a slightly refined version of the algorithm employing Nesterov acceleration and the adaptive step size described above.

See Section S1.2 for the rationale behind each part of the algorithm. The initialization is based on solving the initial weighted low-rank problem exactly when $W$ is rank 1. The intercept update step is based on maximum likelihood estimation holding $\hat{X}$ fixed. In the latent factors update step, we augment the Srebro and Jaakkola (2003) iteration by allowing a step size $\rho >0$ other than 1, and allowing $W$ and $Z$ to depend on the current value of $\hat{X}$ in order to form a local approximation to the weighted low-rank problem.

Computational complexity analysis

• Initialize. The truncated SVD of rank $M$ takes $O\left(IJM\right)$ time (Halko et al., 2011). The other initialization operations only take $O\left(IJ\right)$ time, so this step is $O\left(IJM\right)$ altogether.

• Update intercepts. Each ${\hat{\alpha }}_i$ update takes $O\left(J\right)$ time, and each ${\hat{\beta }}_j$ update takes $O\left(I\right)$ time. Thus, the entire step takes $O\left(IJ\right)$ time per iteration.

• Update latent factors. The entry-wise operations are $O\left(IJ\right)$ and the truncated SVD of rank $M$ takes $O\left(IJM\right)$ time (Halko et al., 2011). Thus, this step takes $O\left(IJM\right)$ time per iteration.

Hence, the runtime for initialization and each iteration of IRSVD is $O\left(IJM\right)$. This is faster than the Fisher scoring algorithm of Miller and Carter (2020), which is $O\left(IJ{M}^2\right)$ per iteration. The quadratic complexity in $M$ can be practically significant since $M$ is often chosen to be around 20–50. Further, IRSVD is simpler than the Miller and Carter (2020) algorithm, making it easier to implement and yielding a smaller constant.

3.4. Scaling up by fitting on a subset of samples and projecting

Even with the improvement due to IRSVD, the computation time scales linearly with the number of samples, making it burdensome on datasets with millions of cells. Thus, to further improve scalability, we propose using a combination of subsampling and projection. Specifically, we first randomly select a subset of cells and run the algorithm in Section 3.3 to estimate the parameters using only the data from these cells. Then, holding $U$ and $\alpha$ fixed at these estimated values, column $j$ of the Poisson bilinear model in Equation 3.1 is simply a GLM with covariate matrix $U$ and coefficients ${\beta }_j,{\sigma }_1{v}_{j1},\dots ,{\sigma }_M{v}_{jM}$, which can be fit using standard software. We refer to this as the “projection method”.

We refer to this subsampling-projection version of the algorithm as “scGBM-proj”, in contrast to running the algorithm on the full dataset, which we refer to as “scGBM-full”. The observation that fixing $U$ or $V$ yields a standard GLM has been used in previous RNA-seq techniques (Leek and Storey, 2007; Sun et al., 2012; Risso et al., 2014). Importantly, the GLMs can be fit in parallel since each column is processed independently. This also reduces memory requirements since the count matrix $Y$ can remain stored in sparse form and only needs to be loaded into memory one column at a time, making it suitable for on-disk processing tools like DelayedArray (Pagès, 2020).

3.5. Quantifying uncertainty in the latent factors

Since scGBM performs dimensionality reduction using a statistical model, uncertainty in the low-dimensional representation can be quantified using the classical method of inverting the Fisher information matrix. Miller and Carter (2020) use this approach, accounting for joint uncertainty in $U$ and $V$ as well as their identifiability constraints by inverting the constraint-augmented Fisher information. However, this is computationally expensive on single-cell datasets.

Thus, to quantify uncertainty in $V$, we use a rough approximation by inverting the diagonal blocks of the Fisher information for $V$, that is, the submatrices $F_1,\dots ,F_J\in {\mathbb{R}}^{M\times M}$ in which entry $\left(m,m^{\prime }\right)$ of $F_j$ is

$$F_{j,m,m^{\prime }}=-E\left(\frac{{\partial }^2\ell }{\partial v_{jm}\partial v_{j{m}^{\prime }}}\right)$$ (3.8)

for $j=1,\dots ,J$ and $m,m^{\prime }=1,\dots ,M$. For the Poisson bilinear model in Equation 3.1,

$$\frac{{\partial }^2\ell }{\partial v_{jm}\partial v_{j{m}^{\prime }}}=-\sum _{i=1}^I{\sigma }_m{u}_{im}{\mu }_{ij}{\sigma }_{m^{\prime }}{u}_{i{m}^{\prime }}$$ (3.9)

by differentiating the log-likelihood $\ell$. Therefore, in matrix form,

$$F_j=\left(U\Sigma {)}^T{\tilde{W}}_j\right(U\Sigma )$$ (3.10)

where ${\tilde{W}}_j=\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{g}\left({\mu }_{1j},\dots ,{\mu }_{Ij}\right)\in {\mathbb{R}}^{I\times I}$. By plugging in parameter estimates and taking the square root of the diagonal entries of $F_j^{-1}$, we obtain approximate standard errors for the estimates of $v_{j1},\dots ,v_{jM}$. This approximation will tend to underestimate the uncertainty in $V$ since it treats the other parameters $\alpha ,\beta ,U$, and $\Sigma$ as fixed. Nonetheless, on simulated data, we find that it yields confidence intervals with coverage only slightly below the target coverage, for large $I$ and $J$ (Figure S1). Standard errors for $U$ can be approximated in a complementary way.

4. Results

We evaluate the performance of scGBM by comparing to GLM-PCA (Section 4.1), testing the accuracy and speed of the projection method (Section 4.2), comparing to PCA coupled with scTransform or $\mathrm{l}\mathrm{o}\mathrm{g}(1+x)$ transformation (Section 4.3), and applying our uncertainty quantification method to clustering (Section 4.4). For a description of all datasets used in this section, see Section S2.

4.1. Comparing to GLM-PCA

In this section, we compare the performance of our estimation algorithm (scGBM-full) to three algorithms provided by the GLM-PCA R package as implemented by Townes and Street (2020): Fisher scoring, Avagrad (Savarese et al., 2021), and stochastic gradient descent (SGD). We generated simulated datasets with $I=1000$,$J\in \left\{2000,{10}^5\right\}$, and $M=10$ by sampling $U$ and $V$ uniformly from the Stiefel manifold, setting $\Sigma =\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{g}\left({\sigma }_1,\dots ,{\sigma }_M\right)$ where ${\sigma }_m=(\kappa m/M)(\sqrt{I}+\sqrt{J})$, generating intercepts ${\alpha }_i$ and ${\beta }_j$ independently from $𝒩\left(\mathrm{0,1}\right)$, and generating $Y$ according Equation 3.1. On each dataset, we ran the estimation algorithms to produce parameter estimates $\hat{U},\hat{V}$, and $\hat{\Sigma }=\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{g}\left({\hat{\sigma }}_1,\dots ,{\hat{\sigma }}_M\right)$.

To assess accuracy, we simulated 100 datasets as above with $J=2000$ and $\kappa =2$. For each $m=1,\dots ,10$, Figure 2a shows the absolute value of the correlation between column $m$ of (i) $U$ and $\hat{U}$ and (ii) $V$ and $\hat{V}$. Additionally, for each dataset, we computed the root mean squared error between the true $X=U\Sigma V^T$ and the estimate $\hat{X}=\hat{U}\hat{\Sigma }{\hat{V}}^T$ produced by each algorithm; see Figure 2b. In terms of both accuracy metrics, we found that scGBM-full performed significantly better (higher correlation, lower error) than GLM-PCA.

Figure 2: Comparison of scGBM-full (IRSVD) and GLM-PCA on simulated data.

a. Absolute value of the correlation between estimated columns of $U$ and $V$ and the ground truth. Points shown are the median over 100 simulated datasets with $I=1000,J=2000$, and $M=10$. b. Root mean squared error (RMSE) between the ground truth latent effects $X=U\Sigma V^T$ and the estimate $\hat{X}$ from each algorithm, for the 100 simulated datasets. GLM-PCA (Fisher) was not included in plots a and b since it has significantly longer runtime, making it computationally burdensome. c. Log-likelihood versus wall clock time on a low latent variability (LLV) simulated dataset with $I=1000$ and $J={10}^5$. d. The same plot for a high latent variability (HLV) simulated dataset with $I=1000$ and $J={10}^5$.

To compare computation time in a way that accounts for convergence rate and goodness of fit, we plot the log-likelihood ${\sum }_{i,j}\left(Y_{ij}\mathrm{l}\mathrm{o}\mathrm{g}\left({\mu }_{ij}\right)-{\mu }_{ij}\right)$ (with additive constants removed) versus the wall clock time, that is, the elapsed time since the start of the algorithm. Figure 2 shows the results for datasets with $J={10}^5$ under two simulation settings: $\kappa =2$, representing low latent variability (Figure 2c), and $\kappa =5$, representing high latent variability (Figure 2d). We observe that the initialization used by scGBM-full has significantly higher log-likelihood, and scGBM-full reaches better solutions more quickly than the GLM-PCA algorithms. In particular, GLM-PCA (AvaGrad) suffers from low log-likelihood and requires many runs to find a suitable learning rate.

For a real data comparison, Table 1 shows the runtime of scGBM-full and the three GLM-PCA algorithms on three real datasets: 10X immune cells $(J=\mathrm{3,994}$, Zheng et al., 2017), COVID-19 Atlas ($J=\mathrm{44,721}$, Wilk et al., 2020), and 10X mouse brain ($J=\mathrm{1,308,421}$, Lun and Morgan, 2020); see Table 1 caption for details. Figure S2 shows plots of log-likelihood versus runtime for the 10X immune cells and COVID-19 Atlas.

Table 1:

Runtime comparison of scGBM, GLM-PCA, and NewWave on three datasets: 10X immune cells, COVID-19 Atlas, and 10X mouse brain; see Section S2 for dataset references.

Dataset	# of cells	scGBM-proj	scGBM-full	GLM-PCA (Fisher)	GLM-PCA (AvaGrad)	GLM-PCA (SGD)	NewWave
10X immune cells	3,994	0.14 min	1.68 min	55.72 min	1.58 min	1.72 min	2.95 min
COVID-19 Atlas	44,721	1.89 min	22.77 min	468.24 min	19.52 min	20.66 min	35.31 min
10X mouse brain	1,308,421	57.67 min	782.40 min	9878.20 min*	811.09 min	505.82 min	>2907 min

For each dataset, $I=1000$ variable genes were selected using the Seurat package (Satija et al., 2015). Each method was run for a maximum of 100 iterations. For scGBM-proj, we used subsamples of sizes 400, 4,000, and 100,000 on the three datasets, respectively, and 64 cores were allocated for parallel processing. For GLM-PCA (SGD), the minibatch size was set equal to the subsample size used by scGBM-proj. For GLM-PCA, the learning rate (for Avagrad and SGD) and the penalty (for Fisher scoring) were set sufficiently small (or large) to prevent divergence. Latent factor dimensionality of $M=20$ was used for all algorithms. When applying NewWave to the 10X brain data, the job timed out after 2907 minutes.

^*The runtime for GLM-PCA (Fisher) on the 10X mouse brain data was estimated by extrapolating the time taken for 10 iterations.

Overall, we found that scGBM-full was roughly an order of magnitude faster than GLM-PCA (Fisher). GLM-PCA (Avagrad) took approximately the same time as scGBM-full, and GLM-PCA (SGD) was similar or somewhat faster, however, GLM-PCA (Avagrad) and GLM-PCA (SGD) often converged to significantly less accurate solutions than scGBM-full (Figure 2a,b). Another limitation of the gradient-based methods implemented in GLM-PCA is that they are very sensitive to the learning rate. For instance, Table S1 shows an example where changing the learning rate by just 0.01 leads to divergence. This significantly increases the computational burden in practice, since the algorithm must search for a suitable learning rate.

4.2. Testing the performance of the projection method

Here, we assess the accuracy and computational efficiency of the projection method for scaling to large data (Section 3.4). To test it on real data, we consider the 10X immune cell and COVID-19 Atlas datasets from Table 1. We first applied scGBM-full to the whole datasets, obtaining ${\hat{V}}_{\text{full}}\in {\mathbb{R}}^{J\times M}$, and then we used scGBM-proj with various subset sizes to obtain ${\hat{V}}_{\text{proj}}\in {\mathbb{R}}^{J\times M}$. To quantify the accuracy of the projection method, we computed the absolute value of the correlation between ${\hat{V}}_{\text{full}}$ and ${\hat{V}}_{\text{proj}}$ for each column $m=1,\dots ,M$; see Figure 3. We found that the first column of $V$ (that is, the one corresponding to the largest scaling factor, ${\sigma }_1$) can be reliably estimated even using a small fraction of the cells to estimate $U$. These results suggest that using the projection method with ≈ 10–15% of the data is sufficient for capturing the most dominant sources of biological variation. We obtained similar results when testing the projection method on simulations with known ground truth; see Figure S3.

Figure 3:

Testing the accuracy of the projection method. We first applied scGBM-full to the 10X immune cell dataset $(J=\mathrm{3,994})$ to estimate $V$. Then for subsets of various sizes, we used scGBM-proj to estimate $V$, and for each $m=1,\dots ,10$, computed the correlation between the two estimates (scGBM-full and scGBM-proj) of the $m$th column of $V$. The points shown represent the median of the absolute value of the correlation, over 100 runs of scGBM-proj. As the subset fraction increases, scGBM-proj agrees more closely with scGBM-full, with 10–15% being sufficient to accurately capture the largest factors. Running the same analysis on the COVID-19 Atlas dataset $(J=\mathrm{44,721})$ yielded similar conclusions. The 10X mouse brain dataset was omitted due to the computational burden of numerous runs required.

To assess the computational efficiency of the projection method, Table 1 reports the runtime of scGBM-proj on the 10X immune cell, COVID-19 Atlas, and 10X mouse brain datasets. We find that scGBM-proj has significantly faster runtimes compared to the other algorithms. In particular, the total runtime on a dataset with 1.3 million cells was less than one hour using a high-performance computing environment with 64 cores. Figures 2c,d and S2 show the log-likelihood attained by scGBM-proj versus wall clock time on simulated and real data. These plots show that scGBM-proj consistently attains the highest log-likelihood for small wall clock times. On the data with high latent variation (HLV), scGBM-proj and scGBM-full converge to solutions with similar log-likelihood, but scGBM-proj does so much more rapidly.

4.3. Comparing to PCA coupled with scTransform or $\mathrm{l}\mathrm{o}\mathrm{g}(1+x)$ transformation

Next, we assess the quality of scGBM results compared to two commonly used approaches, finding that it overcomes the limitations observed in Section 2. The first approach is “log+scale+PCA”, which applies PCA to the matrix log(1+CPT) with all rows standardized to have mean 0 and standard deviation 1. Here, CPT denotes the $I\times J$ matrix of “counts per ten-thousand.” The second approach is SCT+PCA, which applies PCA to the matrix of Pearson residuals produced by scTransform as described in Section 2. Both methods are implemented in the Seurat package (Satija et al., 2015).

One might think that SCT+PCA and scGBM would yield very similar results since, as stated by Townes et al. (2019), applying PCA to the Pearson residuals can be viewed as an approximation to fitting the Poisson bilinear model used by scGBM. This approximation is justifiable when the scale of the latent variation is small (see the discussion following Proposition S1.1), however, when the scale of the latent variation is large, SCT+PCA and scGBM can produce very different results.

Single marker genes simulation

To illustrate, first consider the single marker genes simulation from Figure 1. Figure 4a plot the factor scores from the three methods, showing that scGBM is the only method able to distinguish the three simulated cell types. Even after applying tSNE (Van der Maaten and Hinton, 2008) to SCT+PCA and log+scale+PCA, there is no separation between the cell types (Figure S4). This indicates that the biological signal is not sufficiently captured by any of the PC dimensions in SCT+PCA or log+scale+PCA. In terms of the first factor loading, scGBM clearly prioritizes genes 1 and 2 whereas the other two methods do not (Figure 4b). The first scGBM factor loading can be interpreted qualitatively as saying that moving left in the scGBM factor score plot in Figure 4a corresponds to an increase in gene 1 expression, and moving right corresponds to an increase in gene 2.

Figure 4:

Applying scGBM to the simulations from Figure 1. a. Single marker genes simulation: the first two columns of $V\Sigma$ for scGBM compared to embeddings from SCT+PCA and log+scale+PCA. As in Figure 1, cell type A is colored in red, B is colored in green, and C is colored in blue. b. The weight of each gene in the first factor. The first two genes (colored red) are the only ones that are truly differentially expressed in the simulation. c. Equal log-fold change simulation: Factor 1 weights for scGBM on the simulated data from Figure 1c. d. Purified monocytes data: Factor 1 weights for scGBM on the purified monocytes from Figure 1f. The red dots are stabilized weight estimates using adaptive shrinkage based on sampling variability (Section S1.3).

For this simulation, we used the default number of latent factors, $M=20$, for scGBM. To check for sensitivity to the choice of $M$, we also applied scGBM with $M=50$ latent factors. The estimated factor scores for $M=20$ and $M=50$ agreed well with each other ($r^2=0.94$ for factor 1, $r^2=0.92$ for factor 2). Meanwhile, for GLM-PCA, although it was able to separate the clusters with $M=50$, we found that the results were highly sensitive to the random initialization used by the algorithm. Occasionally, the first two GLM-PCA scores were very strongly correlated, meaning that the second estimated factor was redundant (Figure S5). In contrast, scGBM is not a randomized algorithm and did not suffer from this issue.

Equal log-fold change simulation

Next, consider the equal log-fold change simulation (Figure 1e) where there are two cell types and all genes are differentially expressed with a log₂ fold change of ±1. Unlike SCT+PCA, scGBM does not assign higher weights to genes simply because they have a higher baseline expression (Figure 4c). Indeed, appealingly, the scGBM factor weights are not correlated with the base mean $\mu$. GLM-PCA performed similarly to scGBM on this example (Figure S6).

Purified monocytes data

As a real data example, we revisit the purified monocytes data (Figure 1f). Here, we find that the GLM-PCA weights are strongly correlated with the log mean count (Figure S6), whereas scGBM does not exhibit this bias. However, we observe that both the scGBM and GLM-PCA weights have a higher variance for genes with low mean, which is undesirable. This is due to the inherent large uncertainty in $\mathrm{l}\mathrm{o}\mathrm{g}\left({\mu }_{ij}\right)$ when ${\mu }_{ij}$ is close to zero, leading to unstable estimates of $u_{im}$ (Figure 4d, raw). To address this, we take advantage of the fact that scGBM is model based, which enables us to quantify uncertainty in the parameter estimates. Using the classical method of inverting the Fisher information matrix (Section 3.5), we approximate the standard error corresponding to each entry of $U$ and use the Adaptive Shrinkage (ash) method of Stephens (2017) to perform post hoc shrinkage towards 0; see Section S1.3 for details. Compared to the usual shrinkage approach based on penalized likelihood, the ash method has the advantage of adaptively setting the amount of shrinkage for each parameter, rather than requiring it to be set a priori. Since ash shrinks genes with larger uncertainties more strongly (Figure 4d, stabilized), this effectively stabilizes the scGBM weights such that low mean genes are not given overly large weights simply due to noise.

Semi-simulated hybrid T/B cells

For this example, we used real data to construct a semi-simulated dataset with cells that lie on a “gradient” between two different cell types. Beginning with a dataset of B cells and naive T cells from the DuoClustering2018 package (Duò et al., 2018), we simulated hybrid cells by combining a fraction of the genes from each cell type. Specifically, we used the following procedure to generate a dataset with $I=6168$ genes and $J=5000$ hybrid cells, where each cell is generated as follows:

1. Randomly select one B cell and one naive T cell.

2. Sample $I^{\prime }~\mathrm{U}\mathrm{n}\mathrm{i}\mathrm{f}\{1,\dots ,\mathrm{I}\}$ and randomly choose a subset of $I^{\prime }$ genes.

3. Construct a simulated cell such that the counts for the $I^{\prime }$ randomly chosen genes are equal to those of the memory T cell, and the remaining $I-I^{\prime }$ genes counts are equal to those of the naive T cell.

We then tested the ability of each method to capture the biological gradient, which is mathematically represented by the mixture proportion $I^{\prime }/I$ (Figure 5a). Visually, scGBM appears to capture the true latent structure better than SCT+PCA and log+scale+PCA, as both methods appear to be sensitive to the presence of a few outlier cells Figure 5a. The first PC scores from both SCT+PCA and log+scale+PCA were more strongly correlated with the number of UMI counts per cell, suggesting that these methods were less successful at removing the unwanted variability due to sequencing depth (Figure 5b). The spurious groups of cells persist even after uniform manifold approximation (UMAP) is applied to the PCA scores (Figure S7).

Figure 5:

Comparing methods on semi-simulated data where cells are drawn to lie on a gradient between B cells and Naive T cells. a. The embeddings from scGBM, SCT+PCA, and log+scale+PCA. Lower values of the mixture proportion (red) indicate that the cell is more like a B cell whereas higher values (light blue) indicate that the cell is more like a Naive T cell. b. Factor 1 score versus the log of the mean UMI count per cell. c. A “volcano” plot of the genes driving the first scGBM factor. Specifically, the loading weight for each gene is plotted against its −log¹⁰ p-value. This plot was made using the EnhancedVolcano R package (Blighe et al., 2023).

An additional strength of scGBM is that the low-dimensional embeddings are directly interpretable. Specifically, a unit change along the $x$-axis of the scGBM plot in Figure 5a corresponds to a log fold-change of 1 with respect to the linear combination of genes defined by the first factor loadings. Because scGBM provides a $p$-value for each element $u_{i1}$, one can visualize the important genes in each factor using a volcano plot (Figure 5c). Theoretical calculations suggest that genes with absolute weight $\left|u_{i1}\right|$ greater than $2/\sqrt{I}\approx 0.025$ should be considered as relevant to the factor (Section S4). For example, FCRL2 $\left(u=-0.035,p=1.28\times {10}^{-14}\right)$ is known to be preferentially expressed by B cells (Li et al., 2014), whereas TCEA3 $\left(u=0.028,p=1.53\times {10}^{-15}\right)$ is reported to be enhanced in naive and memory T cells (Uhlén et al., 2015, Human Protein Atlas, www.proteinatlas.org).

4.4. Uncertainty quantification and the cluster cohesion index

Clustering typically occurs downstream of dimensionality reduction, as a post hoc analysis step (Kiselev et al., 2019). Uncertainty in the estimated low-dimensional representation is likely to influence clustering results and other downstream analyses, but the effect of this uncertainty has not been thoroughly investigated in previous work. For example, one might wonder whether different clusters would be identified if the same set of cells was re-sequenced and the same analysis was performed on the resulting new count matrix.

To illustrate, we simulated a count matrix consisting of random Poisson noise, $Y_{ij}~\mathrm{P}\mathrm{o}\mathrm{i}\mathrm{s}\mathrm{s}\mathrm{o}\mathrm{n}\left(1\right)$ i.i.d. for $i=1,\dots ,I$ and $j=1,\dots ,J$ with $I=1000$ and $J=5000$, and we used scGBM to obtain estimated scores $\hat{V}$. Then we applied the Louvain algorithm (Blondel et al., 2008) to $\hat{V}$ to identify clusters of cells, since this is the default clustering algorithm used in Seurat (Satija et al., 2015) (FindClusters() with default resolution equal to 0.8). The Louvain algorithm identified 9 clusters, even though the cells were simulated to be homogeneous. The fact that standard single-cell clustering algorithms produce too many clusters on null data such as this has also been noted in other studies (Morelli et al., 2021; Grabski et al., 2022).

With scGBM, we can use the uncertainty in $\hat{V}$ to quantify the uncertainty in these clusters. First, a simple visualization of the uncertainty in $\hat{V}$ can be obtained by drawing an ellipse around each point, with the dimensions of the ellipse equal to the estimated standard errors; see Figure 6a (left). The considerable overlap among these ellipses suggests that there are no clearly separated clusters. Meanwhile, when there are true clusters, the ellipses for cells from different clusters tend not to overlap as much. For example, on the 10X immune cell data, there is relatively little overlap between the uncertainty ellipses for the distinct cell populations ($\mathrm{T}$ cells, B cells, monocytes); see Figure 6a (right). However, the qualitative visualizations in Figure 6a may be difficult to interpret or use, especially when there are many clusters.

Figure 6:

Cluster cohesion results for two datasets: random Poisson noise and the 10X immune cells. In the 10X immune data, B = “B cells”, M = “Monocytes”, mT = “Memory T”, NC = “Naive cytotoxic”, NK = “Natural Killer”, nT = “Naive T”, rT = “Regulatory T”, and Th = “T helper”. a. The first two columns of $\hat{V}\hat{\Sigma }$. The uncertainty in each point is visualized by drawing an ellipse with axis lengths equal to the estimated standard errors. b. The distribution of the CCI fractions $f_{k,k}$; see Equation S1.16. c. Heatmaps of inter-CCIs for the random Poisson noise data and the 10X immune cells dataset. The inter-CCIs enable one to visualize the uncertainty in the distinctions between different clusters.

Cluster cohesion index (CCI)

To provide a quantitative measure of the uncertainty in each cluster, we introduce the cluster cohesion index (CCI). Given estimates ${\hat{v}}_{jm}$ and corresponding standard errors $\mathrm{s}\mathrm{e}\left({\hat{v}}_{jm}\right)$, we randomly generate perturbed values ${\tilde{v}}_{jm}~𝒩\left({\hat{v}}_{jm},\mathrm{s}\mathrm{e}{\left({\hat{v}}_{jm}\right)}^2\right)$ independently, perform clustering on the rows of $\tilde{V}=\left[{\tilde{v}}_{jm}\right]$, and compute the fraction of pairs of cells that are still in the same cluster. Then, for each original cluster, the CCI is defined as the mean of this fraction over many repetitions; see Section S1.4 for details. The CCI takes a value between 0 and 1, with 1 indicating that all pairs of cells are still in the same cluster after resampling.

Figure 6b shows the CCIs for the random Poisson noise dataset and the 10X immune cell dataset from Figure 6a. As expected, the CCIs for the random noise dataset are all low, reflecting the fact that any identified clusters are artifacts of sampling variability. Meanwhile, on the 10X immune cell data, the CCIs for the true cell types are all higher. We can define a significance threshold for CCIs by computing the values that would have been generated under the null model of $V=0$; see Section S1.4 for details. As one would hope, all of the random Poisson noise CCIs fall below the significance threshold, while all of the 10X immune cell CCIs are above the threshold; see dashed blue lines in Figure 6b.

Inter-cluster cohesion index

To quantify the overlap between clusters in a way that accounts for model-based uncertainty, we define the inter-cluster cohesion index (inter-CCI) as follows: out of all pairs of points $j$ and $j^{\prime }$ that were originally in clusters $k$ and $k^{\prime }$, respectively, the inter-CCI is the mean of the fraction that are in the same cluster after resampling; see Section S1.4 for details. Note that the inter-CCI between a cluster and itself coincides with the CCI of that cluster.

The inter-CCIs can be visualized using a heatmap (Figure 6c). For the random Poisson noise data, all of the inter-CCIs are of similar magnitude to the CCIs themselves, indicating low confidence in all of the clusters, as expected. On the 10X immune cell data, the inter-CCIs for the T-cell subpopulations (nT, mT, rT, Th) are also relatively high, indicating lower confidence in the distinction between these groups. Meanwhile, the remaining subpopulations (B, NK, NC, and M) have high CCIs and low inter-CCIs, indicating high confidence in these clusters. The inter-CCIs allow one to visualize and quantify cluster relationships that would be difficult to discern using scatterplots alone.

Application to breast cancer data

To further demonstrate the utility of inter-CCIs, we consider a dataset of 100,064 cells from the tumor microenvironment of 26 breast cancer patients (Wu et al., 2021); see Figure 7a. Since Wu et al. (2021) used the standard Seurat pipeline (Satija et al., 2015) to cluster cells and used Garnett (Pliner et al., 2019) for annotation, we computed the inter-CCIs for this Seurat-based clustering; see heatmap in Figure 7b. The blocks along the diagonal indicate groups of cell types with substantial overlap, accounting for model-based uncertainty. These blocks appear to correspond to biologically relevant classes of cells such as lymphoid (T cells, B cells, NK cells), myleoid (monocytes and DCs), and endothelial cells. The inter-CCI heatmap enables us to see the relationships among all of the 29 minor cell types, which would be difficult or impossible to see with a scatterplot.

Figure 7:

a. We applied scGBM to the Wu et al. (2021) dataset consisting of 100,064 breast cancer cells and plotted the first two scGBM scores (columns of $\hat{V}$) colored by major cell type. b. The heatmap of inter-cluster cohesion indices (inter-CCIs) for the minor cell types. The inter-CCIs clearly reveal relationships between the clusters that are not visible in the scatterplot, and even enable one to analyze the numerous minor cell types, which would be very difficult to visualize in a 2-d scatterplot.

5. Discussion

Currently, standard practice is to visualize single-cell data by applying UMAP or t-SNE (McInnes et al., 2018; Van der Maaten and Hinton, 2008) to the PCA scores. Although UMAP could instead be applied to the scores produced by scGBM, our results from Figures 4 and 5 show that the raw scGBM scores $V\Sigma$ are already sufficient for visualizing the dominant sources of biological variability. An additional advantage of scGBM is that the visualizations have a direct biological interpretation by looking at the genes included in the corresponding factors. Since UMAP embeddings been shown to induce undesirable distortions (Chari and Pachter, 2023), we believe that scGBM can provide a useful alternative for users who prefer a non-linear yet still model-based embedding.

In future work, there are several interesting directions for improving upon our current approach. Although our scGBM algorithm is faster than GLM-PCA, it is still much slower than PCA — especially because commonly used transformations preserve sparsity, which allows for the use of more efficient SVD algorithms (Baglama and Reichel, 2005). This limitation could potentially be addressed by modifying our IRSVD algorithm to avoid applying the SVD to a dense matrix. A second limitation of scGBM is that, unlike PCA, the latent factors depend on the choice of $M$. Although there are heuristics for choosing the number of latent factors $M$, developing principled techniques for choosing $M$ remains an area for future work. In this paper, we mainly considered quantifying uncertainty in the scores $V$, but it would be also be interesting to develop additional approaches that use the uncertainty in the gene loadings $U$. For example, one could propagate uncertainty into gene set enrichment analysis (Subramanian et al., 2005). A final area for future research would be to induce sparsity in the weights $U$, to improve biological interpretability. Sparsity in $U$ would mean that only a few genes have non-zero weights, making it easier to connect the factors to known biology, for instance, in terms of gene sets and pathways. To this end, it would be interesting to explore whether ideas from sparse PCA (Zou et al., 2006) can be extended to bilinear models.