Dissecting tumor transcriptional heterogeneity from single-cell RNA-seq data by generalized binary covariance decomposition

Abstract

Profiling tumors with single-cell RNA sequencing has the potential to identify recurrent patterns of transcription variation related to cancer progression, and to produce therapeutically relevant insights. However, strong intertumor heterogeneity can obscure more subtle patterns that are shared across tumors. Here we introduce a statistical method, generalized binary covariance decomposition (GBCD), to address this problem. We show that GBCD can decompose transcriptional heterogeneity into interpretable components—including patient-specific, dataset-specific and shared components relevant to disease subtypes—and that, in the presence of strong intertumor heterogeneity, it can produce more interpretable results than existing methods. Applied to data on pancreatic ductal adenocarcinoma, GBCD produced a refined characterization of existing tumor subtypes, and identified a gene expression program prognostic of poor survival independent of tumor stage and subtype. The gene expression program is enriched for genes involved in stress responses, and suggests a role for the integrated stress response in pancreatic ductal adenocarcinoma.

Profiling tumors with single-cell RNA sequencing (scRNA-seq) technologies continues to uncover tumor transcriptional heterogeneity at a resolution unparalleled by bulk transcriptome profiling, and has the potential to reshape our understanding of cancer biology¹. In particular, by analyzing patterns of transcriptional variation, studies can identify ‘gene expression programs’ (GEPs)—sets of genes whose transcription tends to vary in a coordinated fashion^2-5. Some GEPs may be patient-specific, whereas others may be shared across (subsets of) patients. Crucially, shared GEPs may characterize different molecular subtypes or cellular states, and so may provide insights into disease etiology. Further, associating such GEPs with disease progression or therapeutic response may provide medically relevant insights.

A fundamental challenge to identifying shared GEPs is that tumors often exhibit substantial intertumor heterogeneity (differences in expression from one tumor or patient to another)^1,6. Thus, in scRNA-seq datasets involving multiple tumors, malignant cells typically cluster by patient, with little overlap between patients. This strong intertumor heterogeneity tends to obscure the more subtle, but nonetheless important, patterns of heterogeneity shared among tumors. Indeed, as we illustrate in simulations below, integrative analyses that first combine expression data from malignant cells across all tumors, then apply off-the-shelf statistical methods (for example, non-negative matrix factorization (NMF)⁷), tend to identify the patient-specific GEPs, but often miss the shared GEPs. Furthermore, the combined analysis of multiple studies—which is desirable due to the typically small number of tumors in individual scRNA-seq studies—may introduce dataset or batch effects on top of patient effects, further masking the shared GEPs.

To address these issues, many recent studies have taken a tumor-by-tumor approach in which GEPs that capture intratumor heterogeneity (differences in expression among cells in a given tumor or patient) are identified separately in each tumor (for example, by NMF), then compared across tumors to compile a candidate set of shared GEPs^8-14. However, this strategy may fail to identify some shared GEPs. For example, when tumors are each comprised of cells that are homogeneous in molecular subtype identity, the tumor-by-tumor strategy may miss GEPs that distinguish among cancer subtypes (see, for example, the head and neck squamous cell carcinoma case study below). Moreover, this strategy does not benefit from the improved statistical power of an integrated analysis that combines information across tumors.

Another strategy, which tries to combine the benefits of integrated and tumor-by-tumor analyses, treats patient effects as ‘unwanted variation’ to be removed^15-17—a process sometimes known as ‘harmonization’—then applies clustering, NMF or other methods to the ‘harmonized’ data^11,12,18-21. Many harmonization methods have been developed recently for scRNA-seq data^22-27. While such methods can successfully harmonize datasets from nondiseased tissues²⁸, they work best when cell types and states are present in comparable proportions across individual samples, and this is often not the case for tumor data⁶. A recent benchmark study²⁹ showed that Harmony²², Conos²⁶ and reciprocal principal component (PC) analysis (PCA) (implemented in Seurat 3; ref. 24) tend to ‘overharmonize’ cancer datasets, removing genuine variation between patients and thus failing to separate patients with different glioma types.

Therefore, there is a need for improved statistical methods that can better dissect tumor transcriptional heterogeneity in scRNA-seq data into both patient-/dataset-specific and shared GEPs. Here we introduce such a method based on matrix factorization that we call GBCD. We illustrate in simulations that GBCD addresses limitations of existing approaches and yields more interpretable results in the presence of strong intertumor heterogeneity. To demonstrate its potential to yield biological insights, we applied GBCD to data from a study on head and neck squamous cell carcinoma (HNSCC) and to data from three studies on pancreatic ductal adenocarcinoma (PDAC). In the HNSCC data, our approach captured subtype-related GEPs contributing to shared patterns of intertumor heterogeneity that were missed by a previous tumor-by-tumor NMF analysis⁸. In the PDAC data, our analyses refined known clinically relevant subtypes, and identified a GEP related to stress response pathways that was prognostic of poor survival even after accounting for known prognostic factors such as tumor stage and subtype.

Results

Overview of GBCD

We assume that the combined scRNA-seq data from all tumors are contained in an $N\times J$ matrix $Y$ with entries $y_{ij}$, where $i=1,\dots ,N$ indexes cells and $j=1,\dots ,J$ indexes genes. In typical applications, $Y$ contains unique molecular identifier (UMI) counts transformed using the shifted logarithm³⁰ (‘shifted log counts’ for brevity). GBCD is ‘unsupervised’ in that it does not use information about which cell comes from which tumor, unlike tumor-by-tumor and many harmonization approaches.

GBCD is a matrix factorization method that decomposes $Y$ into two matrices, $L$ and $F$, such that $Y\approx L{F}^T$, or, equivalently

$$y_{ij}\approx \sum _{k=1}^K{l}_{ik}{f}_{jk}$$ (1)

Our goal is that the $K$ components in equation (1) be interpretable as GEPs: $l_{ik}$ represents the membership of cell $i$ in GEP $K$, and $f_{jk}$ represents the effect of GEP $k$ on expression of gene $j$. When the $y_{ij}$ are shifted log counts, each $f_{jk}$ approximately represents the log-fold change (LFC) of gene $j$ in a cell $i$ with membership $l_{ik}=1$ relative to a cell $i^{\prime }$ with membership $l_{i^{\prime }k}=0$. We refer to the $f_{jk}$ values as LFCs, and we call ${\mathbf{f}}_k≔{(f_{1k},\dots ,f_{jk})}^T$ the ‘signature’ of GEP $k$, where $T$ denotes the matrix transpose.

We use GEP as a term of art for the $K$ components in equation (1), but we emphasize that some components may not have a coherent biological function. Since our method does not use information about which cells come from which tumors, patient-specific or batch effects may also appear as GEPs. Determining which GEPs are most biologically relevant is done in subsequent analyses of the gene signatures (see below).

We make two key assumptions about the matrix factorization (equation (1)):

$$\begin{gathered} \text{Memberships}{l}_{ik}\text{are non-negative},\text{and often nearly} \\ \text{binary (close to}0\text{or}1). \end{gathered}$$ Assumption 1:

$$\text{The GEP signatures}{\mathbf{f}}_k\text{are mutually orthogonal}.$$ Assumption 2:

Assumption 1 is motivated by our interest in capturing discrete substructures such as patient effects and tumor subtypes, where cell memberships are binary. However, since nondiscrete structures may also be of interest, and can arise naturally from continuous biological processes, we implement assumption 1 as a ‘soft’ assumption; that is, GBCD encourages, but does not require, binary memberships. We refer to this assumption as the generalized binary assumption.

Assumption 2 is motivated by our observation that, without this assumption, a shared GEP can get absorbed into multiple patient-specific effects. The orthogonality assumption helps to preserve the shared GEP because absorbing it into multiple patient-specific GEPs would make the patient-specific signatures dependent, and so nonorthogonal. Note that, unlike the memberships $l_{ik}$, the LFCs $f_{jk}$ can be both positive or negative, allowing them to model both upregulation and downregulation of genes in a GEP.

Note that these assumptions allow for overlapping structures in the cell population, in that multiple nonzero memberships $l_{ik}$ are permitted for each cell $i$. For example, a given cell might have membership in four GEPs—one dataset-specific GEP, one patient-specific GEP, one tumor subtype GEP and one cellular state GEP—and this configuration would be represented by four nonzero $l_{ik}$. This ability to model overlapping structures is a key feature that distinguishes GBCD from clustering methods that identify nonoverlapping structures only.

Combining equation (1) with assumption 2 implies that

$$Y{Y}^T\approx L{F}^{T}F{L}^T=LD{L}^T$$ (2)

where $D≔F^{T}F$ is a diagonal matrix. In practice, our approach first finds $L$ and $D$ in equation (2), then estimates $F$ based on equation (1); both steps are performed using the empirical Bayes matrix factorization (EBMF) methods from ref. 31. The matrix $Y{Y}^T$ in equation (2) is related closely to the cell-by-cell covariance matrix, so we refer to the matrix factorization in equation (2) as the GBCD. For brevity, we also use GBCD to refer to our overall approach.

We assessed the biological relevance of each GEP by examining its top ‘driving genes,’ that is, genes with largest LFCs by magnitude. Then we performed gene set enrichment analyses (GSEA)³² to identify biological processes enriched for the driving genes. Driving genes can be either upregulated (positive $f_{jk}$) or downregulated (negative $f_{jk}$). Our analyses have focused on upregulated genes to facilitate comparisons with previously reported GEPs, but we note that downregulated genes may also be interesting.

Illustrative example

To illustrate the difference between GBCD and existing approaches, we performed a simple simulation designed to mimic the features of a multitumor scRNA-seq dataset. Briefly, we simulated scRNA-seq counts for $J=10,000$ genes in $N=3,200$ cells from eight patients with three different types of GEPs (Fig. 1a):

Fig. 1 ∣. GEP identification in simulated multipatient scRNA-seq datasets.

a, $N\times 11$ membership matrix, $L$, used to simulate the data, and estimates of $L$ in one of the simulated datasets ($N=3,200$). Rows are cells and columns are GEPs. Each column was scaled so that the largest value was 1. All methods identified one or more components correlated strongly with the cellular detection rate—a principal source of nuisance variation⁵⁵; these ‘nuisance’ components are not shown. See also Supplementary Fig. 1. b, $t$-SNE visualizations of the same dataset, in which cells are colored by membership to the continuous (top), subtype (middle) or patient-of-origin (bottom) GEPs. c, Performance of the different methods compared in the 20 simulated datasets. See Supplementary Note for details on these methods and how they were applied to the simulated datasets. Performance was measured as the Pearson correlation between the true GEP membership and the estimated membership for the estimated GEP with the highest correlation. The boxes in the boxplot give the interquartile range of the Pearson correlations; the median is shown as the horizontal line; points represent values outside the interquartile range.

Patient-specific GEPs (P1–P8): each of these GEPs was unique to a single patient, with cells having binary memberships to these GEPs. These GEPs mimic the strong patient effects in single-cell tumor data.

Subtype-related GEPs (S1, S2): these GEPs were active in patients 1–4 and 5–8, respectively, with cells having binary memberships to these GEPs. These GEPs mimic cancer subtype programs shared by multiple patients.

Continuous GEP: this GEP was active in 600 cells chosen at random, with nonbinary memberships. This GEP mimics a cellular process or activity whose intensity varies continuously across cells, and may be unrelated to cancer subtype.

The continuous GEP contributes to patterns of intratumor heterogeneity shared across patients, whereas the subtype-related GEPs do not. The subtype-related GEPs induce hierarchical relationships among the patients in that patients 1–4 are more closely related to each other than to patients 5–8; however, this is not a strict hierarchy due to the continuous GEP. The differentially expressed genes were unique to each GEP, satisfying the orthogonality assumption (see Methods for more details on data generation).

We simulated 20 scRNA-seq datasets in this way. The results of a typical simulation are shown in Fig. 1a; see also Supplementary Figs. 1 and 2. The strong patient effects are reflected in the $t$-SNE embedding³³, in which the primary structure is the clustering by patient (Fig. 1b).

GBCD and both versions of ‘combined NMF’—NMF applied to the combined scRNA-seq counts or the shifted log counts from all patients— successfully identified the patient-specific GEPs (Fig. 1a and Supplementary Fig. 1). However, the GBCD membership estimates were visually the least noisy. All three methods accurately identified the continuous GEP (Supplementary Fig. 2). Notably, GBCD accurately estimated the memberships of both the patient-specific GEPs and the continuous GEP, highlighting the flexibility of the generalized binary assumption.

The key difference among the three methods was that GBCD accurately identified the two subtype-related GEPs, while the combined NMF methods did not (Fig. 1a and Supplementary Fig. 1). Without an orthogonality constraint, the combined NMF methods absorbed the subtype-related GEPs into the patient-specific GEPs; that is, they estimated the patient-specific gene signatures as ${\mathbf{f}}_{\mathrm{P}1}+{\mathbf{f}}_{\mathrm{S}1}$, ${\mathbf{f}}_{\mathrm{P}2}+{\mathbf{f}}_{\mathrm{S}1}$, ${\mathbf{f}}_{\mathrm{P}3}+{\mathbf{f}}_{\mathrm{S}1}$, ${\mathbf{f}}_{\mathrm{P}4}+{\mathbf{f}}_{\mathrm{S}1}$, ${\mathbf{f}}_{\mathrm{P}5}+{\mathbf{f}}_{\mathrm{S}2}$, ${\mathbf{f}}_{\mathrm{P}6}+{\mathbf{f}}_{\mathrm{S}2}$, ${\mathbf{f}}_{\mathrm{P}7}+{\mathbf{f}}_{\mathrm{S}2}$ and ${\mathbf{f}}_{\mathrm{P}8}+{\mathbf{f}}_{\mathrm{S}2}$. This decomposition of $Y$ failed to separate the shared GEPs from the patient-specific GEPs, and, accordingly, did not recover the hierarchical relationships between patients. GBCD ruled out this solution because it did not satisfy the orthogonality assumption (for example, ${\mathbf{f}}_{\mathrm{P}1}+{\mathbf{f}}_{\mathrm{S}1}$ and ${\mathbf{f}}_{\mathrm{P}2}+{\mathbf{f}}_{\mathrm{S}1}$ are not orthogonal to each other because they share ${\mathbf{f}}_{\mathrm{S}1}$), and chose a different matrix factorization that revealed the hierarchical relationships.

To check that the differences shown above were systematic, rather than the chance outcome of a single simulation, we repeated these comparisons in 20 replicate simulations. The results confirmed that both GBCD and combined NMF consistently detected the patient-specific and shared continuous GEPs, whereas GBCD more reliably recovered the subtype-related GEPs (Fig. 1c).

Next, we compared with PCA applied to the combined shifted log counts, which is another widely used approach for dimension reduction of scRNA-seq data. In Fig. 1a, PCA identified both the subtype-related and the continuous GEP, but it failed to identify the patient-specific GEPs. Instead, PCA captured the patient effects using combinations of PCs; for example, PCs 3–8 were each loaded on cells belonging to a seemingly random subset of patients. This is a problem for interpretation because these PCs might be interpreted, falsely, as capturing features shared across patients. Part of the problem with PCA here is that it constrains the ${\mathbf{l}}_k$ to be orthogonal to each other, so it cannot correctly capture hierarchical relationships (for example, ${\mathbf{l}}_{\mathrm{S}1}$ and ${\mathbf{l}}_{\mathrm{P}1}$ are not orthogonal to each other). Results across the 20 replicate simulations confirmed that PCA had substantially worse specificity for GEP discovery than GBCD in that a much larger proportion of the GEPs estimated by PCA (the PCs) did not correlate strongly with any of the 11 true GEPs (Supplementary Fig. 3).

Other strategies similarly had difficulty identifying the subtype-related GEPs (Fig. 1c and Supplementary Fig. 4). Consensus NMF⁵, which averages the solutions of combined NMF over multiple runs, failed to identify the subtype GEPs because none of the individual NMF runs identified them. Patient-by-patient NMF, which fits NMF to the shifted log counts separately for each patient then clusters the estimated GEPs, failed to identify subtype GEPs because these effects did not vary among the cells within a patient. A variant of GBCD that replaced assumption 1 with a generic sparsity assumption using point-exponential priors (‘CD, point-exp prior’) identified some subtype effects, but not as reliably as GBCD, and was worse at distinguishing active versus inactive cells in GEPs with binary memberships. This demonstrates the importance of assumption 1 for identifying discrete structures. We also compared with two other EBMF methods that were similar to GBCD but dropped assumption 2 (‘EB-SNMF, point-exp prior’ and ‘EB-SNMF, GB prior’; see Supplementary Note for details). Similar to NMF, these methods tended to miss subtype-related GEPs. ‘Batch effect correction’-type methods that integrate scRNA-seq data from multiple samples, including LIGER²³, CCA implemented in Seurat²⁴, MNN Correct²⁵ and Conos²⁶, likewise failed to identify subtype effects, consistent with the observation in ref. 29 that these methods tend to overcorrect. Among the batch-correction methods, MNN Correct performed best in identifying the subtype-related GEPs, and MNN Correct also makes an orthogonality assumption.

HNSCC data

We analyzed scRNA-seq data from primary tumors in ten HNSCC patients, and lymph node metastases in five of these patients⁸. Puram et al.⁸ found that each of the ten patients mapped to one of three HNSCC molecular subtypes previously defined from an analysis of bulk expression data³⁴. We show here that GBCD can learn the molecular subtypes de novo from the single-cell data, whereas other methods struggle to do this.

As noted in ref. 8, the $N=2,176$ malignant cells demonstrate strong patient effects that are typical of cancer data; the main structure in the $t$-SNE plot is the clustering of the cells by patient (Fig. 2a). Because of these strong patient effects, Puram et al.⁸ took a tumor-by-tumor analysis approach: they applied NMF separately to each tumor to identify GEPs, then they clustered the GEPs to identify six consensus GEPs that they called meta-programs. However, this failed to identify GEPs that classify the patients into their molecular subtypes. This illustrates a limitation of the tumor-by-tumor approach, which can only identify GEPs whose memberships vary across cells within a tumor.

Fig. 2 ∣. GBCD analysis of HNSCC data.

a, $t$-SNE plot showing $N=2,176$ malignant cells in ten primary tumors and five matched lymph node metastases. Cell color varies by patient of origin and tumor stage (primary tumor, lymph node (LN) metastasis), and shape varies by tumor molecular subtype (classical, basal, atypical). b, Cell memberships in the 19 GEPs estimated by GBCD. Membership values were rescaled separately for each GEP so that the maximum membership in each GEP was always 1. Numbers in parentheses give the number of cells per tumor. Measures of patient-specific expression and interpatient variation in expression are shown. See Methods for definitions. c, LFC estimates of the ten upregulated genes with the largest LFC for each shared GEP (see also Supplementary Table 1). d, Concordance between the shared GEPs and six previously identified meta-programs⁸ by $P$ values from one-sided Wilcoxon rank-sum tests ($P$ values were not corrected for multiple tests, and only $P$ values less than 10⁻¹⁰ are shown).

We applied GBCD to the 2,176 malignant cells. In total, GBCD identified 19 GEPs (not including an ‘intercept’ GEP on which all cells were loaded). Of these GEPs, 12 were active in multiple patients (shared), and the remaining seven were specific to an individual patient (patient-specific) as indicated in Fig. 2b. We annotated the shared GEPs by the driving genes and enriched gene sets (Table 1, Supplementary Fig. 5 and Supplementary Tables 1-2), and showed the LFC estimates for the top ten driving genes in each shared GEP (Fig. 2c). Each of the six meta-programs from ref. 8 overlapped with at least one shared GEP (Fig. 2d). However, other shared GEPs were not correlated strongly with any of the six meta-programs, for example, GEP 11 (respiratory electron transport) and GEP 12 (mRNA splicing).

Table 1 ∣.

Top driving genes and enriched gene sets for the shared GEPs identified from HNSCC data

GEP	Top ten driving genes	Top significantly enriched gene sets
1	GSTA1, EPCAM, GSTM1, SPP1, CES1, ALDH3A1, OSGIN1, GPC3, ALDH1A1, AKR1C1	Biological oxidations, phase II—conjugation of compounds, genes upregulated through activation of mTORC1 complex, metabolism of lipids, phase I—functionalization of compounds
2	S100A9, S100A8, MT2A, GJB2, TGFBI, CD74, SAA1, PLEK2, LGALS1, COL17A1	Genes defining epithelial-mesenchymal transition, genes upregulated in response to IFNγ, interferon signaling, IFNγ signaling, IFNα/β signaling
3	UBE2T, GINS2, MCM2, MCM5, RFC4, CDC6, MAD2L1, FEN1, PCNA, MCM7	Genes encoding cell cycle related targets of E2F transcription factors, cell cycle, cell cycle (mitotic), cell cycle checkpoints, genes involved in the G2/M checkpoint
4	CCNB1, CDC20, CCNB2, CDKN3, PTTG1, BIRC5, PLK1, TPX2, KIF20A, HMGB2	Genes involved in the G2/M checkpoint, cell cycle, cell cycle (mitotic), mitotic prometaphase, genes encoding cell cycle related targets of E2F transcription factors
5	RRM2, CENPW, ZWINT, PCLAF, TK1, MCM5, TYMS, MCM3, MCM7, ASF1B	Genes encoding cell cycle related targets of E2F transcription factors, cell cycle, cell cycle (mitotic), DNA strand elongation, activation of ATR (ataxia telangiectasia-mutated and Rad3-related) kinase in response to replication stress
6	GPNMB, LAPTM4A, KRT14, B2M, DUSP1, CD9, FTL, SAT1, H3F3B, MTRNR2L2	Genes involved in p53 pathways and networks, genes upregulated in response to low oxygen levels (hypoxia), genes regulated by NF-κB in response to TNF, cell junction organization
7	FABP5, KRT16, LY6D, S100A7, S100A8, AQP3, SPRR1B, KRT6B, GPR15LG, S100A9	Formation of the cornified envelope, keratinization, developmental biology, neutrophil degranulation, Genes defining late response to estrogen
8	S100A9, APOBEC3A, S100P, LCN2, KLK7, CST6, SLPI, CEACAM6, KRT80, KLK6	Formation of the cornified envelope, keratinization, antimicrobial peptides
9	TNC, MMP2, VIM, MMP10, TPM1, ITGB6, DKK3, PLAU, TGFBI, MMP13	Genes defining epithelial-mesenchymal transition, ECM organization, collagen formation, assembly of collagen fibrils and other multimeric structures, ECM proteoglycans
10	LAMC2, ANXA3, PLAUR, LGALS1, SERPINE1, ITGA5, UPP1, ANXA1, TNFRSF12A, TNC	Genes defining epithelial-mesenchymal transition, genes regulated by NF-κB in response to TNF, hemostasis, cell junction organization, genes encoding components of apical junction complex
11	SNHG29, SET, IFI27, B2M, COX6A1, HNRNPH1, C6orf62, SLIRP, RAN, HMGB1	Respiratory electron transport, ATP synthesis by chemiosmotic coupling, the citric acid cycle and respiratory electron transport, cellular response to chemical stress, a subgroup of genes regulated by MYC (version 1)
12	MTRNR2L8, MTRNR2L2, MIR4461, HSPA1A, GNAS, MTRNR2L1, UGDH-AS1, TMSB10, H3F3AP4, MTRNR2L10	mRNA splicing, processing of capped intron-containing pre-mRNA, mRNA splicing—minor pathway, processing of replication-dependent histone pre-mRNAs, RNA polymerase II transcription termination

Middle column, genes with the largest expression increases (largest positive LFC); right-hand column, enriched MSigDB Hallmark and CP:Reactome gene sets^32,52,53 ($P<{10}^{-6}$). See Supplementary Table 2 for a description of the $P$ values and for more GSEA results.

Most importantly, GBCD identified GEPs corresponding closely to the previously defined molecular subtypes³⁵: GEP 1 was largely active only in cells from the two classical patients, and GEP 2 was mainly active in cells from the seven basal patients. Therefore, GEPs 1 and 2 were sufficient to classify the ten patients into the three subtypes. Analysis of variance of the GEP memberships revealed that GEPs 1 and 2 had much greater interpatient than intrapatient variation, whereas the opposite was true for the other shared GEPs (Fig. 2b). This is consistent with the observation that tumor cells from a single HNSCC patient are largely homogeneous in molecular subtype identity⁸.

The results from applying other methods highlighted behaviors qualitatively similar to those observed in the simulations (Supplementary Figs. 6-13). For example, NMF identified the patient-specific programs but failed to identify the molecular subtypes. PCA was the only other method that identified the molecular subtypes, but it failed to capture patient-specific effects as individual PCs. Furthermore, the GSEA results for many of the shared PCs were difficult to interpret (Supplementary Fig. 12).

Pancreatic ductal adenocarcinoma data

PDAC is highly lethal and demonstrates extensive heterogeneity in disease progression and treatment response among patients³⁶. Many studies have attempted to define clinically relevant subtypes with the goal of personalizing treatment and improving outcomes³⁷. An early study³⁸ applied NMF to microarray expression data and identified ‘classical’ and ‘basal’ subtypes. Later studies used newer technologies to refine these subtypes (Supplementary Note).

To investigate whether GBCD can contribute biological insights, we performed a de novo analysis of scRNA-seq profiles of 35,670 malignant cells from 59 PDAC tumors. These data were from three studies using two different scRNA-seq technologies (Table 2). Since GBCD can effectively disentangle shared GEPs from patient- and study-specific effects, we expected our analysis to provide a deeper characterization of recurrent gene expression patterns by pooling tumors from multiple studies to achieve a larger sample size. Also, we focused on malignant cells to avoid the influence of immune and stromal cell infiltration on the analysis of tumor transcriptional heterogeneity. This is known to be an issue for bulk RNA-seq of PDAC tumors given their typically low tumor purity³⁶.

Table 2 ∣.

PDAC datasets

Label	Study	Assay	Specimens	$N$
A	Chan-Seng-Yue et al.36	10x 3′ v.2	13 primary, 1 metastatic	18,920
B	Peng et al.54	10x 3′ v.2	24 primary	10,936
C	Raghavan et al.41	Seq-Well	21 metastatic	5,814

The right-most column ($N$) gives the number of high-quality malignant cells retained for analysis.

A $t$-SNE plot of the 35,670 scRNA-seq profiles shows that the cells cluster by patient and cohort (Fig. 3a). This substantial interpatient and interstudy structure may obscure other biologically interesting expression patterns that are shared across tumors.

Fig. 3 ∣. GBCD analysis of PDAC data.

a, $t$-SNE plot of $N=35,670$ malignant cells from 59 PDAC tumor samples (Table 2). Cells are colored by patient of origin. Note two patients (A1 and B17) are labeled twice since the cells from those patients split into two distinct clusters. b, Memberships of the cells in the 33 GEPs identified by GBCD. Cells are arranged top-to-bottom by study and patient, and within each study patients are ordered by the proportion of cells expressing GEP 1 which correlates strongly with the classical subtype of PDAC (see c). Membership values were rescaled separately for each GEP so that the maximum membership in each GEP is always 1. Measures of patient-specific expression, interpatient variation in expression and chromosomal structure are shown. See Methods for definitions. c, Summary of concordance between the shared GEPs and literature-derived expression signatures for classical and basal PDAC subtypes^36,38,41,42 (CK, Chan-Seng-Yue et al.³⁶) by $P$ values from one-sided Wilcoxon rank-sum tests ($P$ values were not corrected for multiple tests, and only $P$ values less than 10⁻¹⁰ are shown). d, LFC estimates of the ten upregulated genes with the largest LFC for each shared GEP (Supplementary Table 3).

Applying GBCD to these data identified a total of 33 GEPs, not including an intercept GEP. We split these GEPs into two broad groups. The first group included GEPs 15–33 (Fig. 3b), which were driven primarily by patient or dataset effects. GEPs 21–33 were each active predominantly in one patient, and thus mainly captured patient-specific effects, in part due to the influences of aneuploid copy number profiles that are common in most human tumors and can vary from patient to patient³⁹. GEP 15 captured mainly dataset effects and distinguished cohorts A and B from cohort C, which differed by tumor stage (metastatic versus primary) and sequencing platform (Seq-Well versus 10x). GEPs 16–20 were each active in, at most, a few patients and demonstrated much less within-patient than between-patient variation (Fig. 3b), and may capture patterns of copy number variation common to a few patients. GEPs 16–33 exhibited markedly stronger chromosomal structure than GEPs 1–14 (Fig. 3b and Supplementary Fig. 15), suggesting a link to aneuploidy in tumor cells.

The second group included GEPs 1–14 (Fig. 3b). These GEPs were shared across multiple tumors and represented potentially interesting components of transcriptional variation in PDAC tumors. Shared GEPs were annotated by the driving genes and enriched gene sets (Table 3, Supplementary Fig. 14 and Supplementary Tables 3-4). GEPs 1–8 aligned with literature-derived classical or basal-like PDAC subtype signatures (Fig. 3c). This result confirms that these subtypes can be identified de novo from scRNA-seq data. The classical-subtype-associated programs (GEPs 1–3) were enriched for biological processes including O-linked glycosylation of mucins (GEP 1), cholesterol homeostasis (GEP 2) and MYC regulation (GEP 3). GEPs 4–8 were associated with basal-related signatures and were enriched for different biological processes including epithelial-mesenchymal transition, extracellular matrix (ECM) organization, p53 pathway and cornification. Many of these biological processes have also been found to be enriched in literature-derived subtype signatures^36,40-42. The classical-associated GEPs 1 and 2 and the basal-associated GEPs 4–6 were largely active in two separate subsets of cells, but these cells were sometimes from the same tumor (for example, B17), consistent with the finding that basal and classical programs can co-exist intratumorally^36,43. GEPs 9–14 did not correlate with previously defined subtype signatures. These GEPs were enriched for interferon signaling (GEP 9), cell cycle (GEP 10), tumor necrosis factor–nuclear factor kappa B (TNF–NF-κB) signaling (GEP 11), respiratory electron transport (GEP 12), translation (GEP 13) and stress-induced pathways (GEP 14). Figure 3d shows, for each shared GEP, the LFC estimates for the top ten upregulated genes.

Table 3 ∣.

Top driving genes and enriched gene sets for the shared GEPs identified from PDAC data

GEP	Top ten driving genes	Top significantly enriched gene sets
1	TFF1, CLDN18, PHGR1, REG4, TFF3, DPCR1, TFF2, CEACAM5, SPINK1, MUC5AC	Cell–cell communication, O-linked glycosylation of mucins, cell junction organization, O-linked glycosylation, genes defining late response to estrogen
2	REG4, CES2, AKR1B10, CA2, PHGR1, OLFM4, ALDH1A1, TFF3, CLDN18, CA9	Metallothioneins bind metals, genes involved in cholesterol homeostasis
3	CA2, REG4, AHCY, AKR1B10, KCNE3, TMEM97, GPATCH4, IMPA2, CLDN18, FAM195A	Mitochondrial translation, a subgroup of genes regulated by MYC (version 2), a subgroup of genes regulated by MYC (version 1), translation, rRNA processing
4	MT2A, S100A2, CAV1, MIR205HG, VIM, MT1X, CST6, LGALS1, TGFBI, MT1E	Genes defining epithelial-mesenchymal transition, regulation of IGF transport and uptake by IGFBPs, metallothioneins bind metals, response to metal ions
5	CAV1, DCBLD2, MT2A, S100A2, KRT6A, VIM, PTRF, COL17A1, MT1E, HMGA2	Type I hemidesmosome assembly, ECM organization, genes defining epithelial-mesenchymal transition, laminin interactions, cell junction organization
6	KRT13, LY6D, KRT5, AQP3, S100A2, LYPD3, SCPEP1, S100A9, CCL20, MTRNR2L2	Genes involved in p53 pathways and networks, genes regulated by NF-κB in response to TNF, formation of the cornified envelope, keratinization, genes defining late response to estrogen
7	TMSB4X, B2M, C15orf48, TACSTD2, FTH1, LCN2, TMSB10, SAT1, HLA-A, S100A6	Neutrophil degranulation, innate immune system, hemostasis, genes encoding components of the complement system, regulation of IGF transport and uptake by IGFBPs
8	HLA-DRB1, HLA-DRA, SERPINA1, CLU, CD74, SPINK1, MMP7, PDIA3, HSP90B1, ENO1	Neutrophil degranulation, innate immune system, antigen presentation: folding, assembly and peptide loading of MHC class I, MHC class II antigen presentation, regulation of IGF transport and uptake by IGFBPs
9	TNFSF10, IFI6, ISG15, IFI16, MX1, UBE2L6, IFIT3, IFI44L, PARP14, CD74	Genes upregulated in response to IFNα proteins, genes upregulated in response to IFNγ, interferon signaling, IFNα/β signaling, cytokine signaling in immune system
10	CENPF, TOP2A, PTTG1, MKI67, HMGB2, UBE2C, STMN1, BIRC5, CCNB1, NUSAP1	Genes encoding cell cycle related targets of E2F transcription factors, cell cycle, cell cycle (mitotic), mitotic prometaphase, genes involved in the G2/M checkpoint
11	ISG20, PLAUR, CLDN18, CXCL5, CLDN23, KLK10, CD55, CDKN1A, SNX9, GULP1	Genes regulated by NF-κB in response to TNF, cytokine signaling in immune system, IFNα/β signaling, tight junction interactions, genes mediating apoptosis by activation of caspases
12	TMSB10, NCL, LYZ, EIF5A, TFF2, HSP90AB1, NME1, HMGA1, HSPD1, SNRPF	A subgroup of genes regulated by MYC (version 1), respiratory electron transport, ATP synthesis by chemiosmotic coupling, metabolism of RNA, cellular response to chemical stress
13	GNB2L1, EEF1A1, SLC25A6, ADIRF, GLTSCR2, UQCRH, HNRNPA1, SYT8, PABPC1, PTMA	Eukaryotic translation initiation, activation of mRNA upon binding of cap-binding complex and eIFs, eukaryotic translation elongation, translation
14	ZFAS1, FTH1, DDIT3, EPB41L4A-AS1, NDRG1, EIF4A2, TAF1D, SLC3A2, GULP1, FTL	Genes upregulated in response to low oxygen levels (hypoxia), genes upregulated during unfolded protein response, a cellular stress response related to the endoplasmic reticulum, genes upregulated through activation of mTORC1 complex, response of EIF2AK1 (HRI) to heme deficiency, unfolded protein response

Middle column, genes with the largest expression increases (largest positive LFC); right-hand column, enriched MSigDB Hallmark and CP:Reactome gene sets^32,52,53 ($P<{10}^{-6}$). See Supplementary Table 4 for a description of the $P$ values and for more GSEA results.

Despite the close association of GEPs 1–8 with literature-derived subtype signatures, they do not simply recapitulate previous results. Some previously identified signatures (for example, the squamoid signatures from ref. 42) are not correlated strongly with any of our GEPs. Other previously identified signatures also reflect some combination of GEPs 1–8 (Fig. 3c). Thus, GEPs 1–8 represent a different decomposition of classical- and basal-related transcriptional heterogeneity than has previously been described.

A stress signaling program is prognostic of poor survival.

To assess the prognostic value of GEPs 1–14, we scored each GEP signature in bulk RNA-seq data of 391 primary PDAC tumors from four studies^40,44-46 involving 260 deaths, and performed a survival analysis. Specifically, we performed stepwise variable selection of the 14 gene signatures, together with the previously identified subtype signatures, in a Cox proportional hazards regression model with overall survival as the endpoint, adjusting for age, sex and tumor stage. The final model selected GEP 4 and GEP 14 ($P<0.01$), and selected none of the previously identified subtype signatures (Fig. 4a). GEP 4 was prognostic of poor survival (hazard ratio (HR) = 1.58; $P=2\times {10}^{-10}$), and was the only selected program related to subtype (basal). This suggests that GEP 4 is more strongly prognostic of survival than the previously defined subtype signatures. GEP 14 was also prognostic of poor survival (HR = 1.32; $P=5\times {10}^{-5}$), and represented a prognostic expression signature even after accounting for tumor subtype and tumor stage. While GEP 14 showed strong association with survival in the overall population (Fig. 4b), further analyses stratified by tumor stage and classical/basal subtype showed that GEP 14 was particularly prognostic of patient survival in earlier stage basal-like tumors (Fig. 4c).

Fig. 4 ∣. Prognostic relevance of GEPs assessed in bulk RNA-seq of 391 primary PDAC tumors.

a, Cox proportional hazards model selected by the stepwise variable selection. For each covariate, the HR estimate (black square), 95% confidence interval (horizontal error bar) and $P$ value from two-sided Wald test are given. ISR, integrated stress response. b,c, Kaplan–Meier plots comparing survival of patients with high and low levels of GEP 14 (high versus low values are split at the median value) in all 391 patients (b), and subgroups of patients stratified by tumor subtype (GEP 4 dichotomized at the median value) and tumor stage (c). $P$ values are from two-sided log-rank tests, and are not adjusted for multiple testing.

Inspection of the top driving genes in GEP 14 revealed interesting connections with previous literature on PDAC. The top driving gene is ZFAS1, a long noncoding RNA gene that has been shown to promote PDAC metastasis⁴⁷. Other top driving genes include two genes related to iron metabolism, FTH1 and FTL, both of which have been linked recently with PDAC progression arising from increased ferritinophagy^48,49. Enrichment analysis of a longer list of driving genes highlights several gene sets related to stress response (Supplementary Fig. 14). These include genes upregulated in response to hypoxia (for example, BNIP3L, DDIT3, VEGFA), heme deficiency and misfolded protein accumulation in the endoplasmic reticulum. Notable driving genes related to stress response include ATF4—the main effector of the integrated stress response⁵⁰—and several of its validated targets (Supplementary Note).

Discussion

In this paper, we introduced a matrix factorization method, GBCD, and demonstrated its ability to yield insights into transcriptional heterogeneity from scRNA-seq tumor data in multiple patients and studies. GBCD assumes that expression programs are mutually orthogonal, and hence avoids absorbing shared expression programs into patient-specific expression programs. When cells show strong interpatient heterogeneity, which is typical of malignant cells, GBCD can isolate shared patterns of variation that existing methods miss. Unlike NMF, GBCD does not constrain expression programs to be non-negative, allowing GBCD to capture both transcriptional repression and upregulation.

From scRNA-seq tumor data, GBCD identified GEPs that were shared across tumors and datasets, as well as GEPs that characterized patient and dataset effects. Some of the shared GEPs correspond to previously defined tumor subtypes or generic cellular processes such as cell cycle and immune activation. Most notably, GBCD identified an expression program, GEP 14, which was an independent predictor of poor prognosis in patients with primary PDAC. GEP 14 is associated with hypoxia and many other stress response processes, including many genes that are induced by ATF4 as part of the integrated stress response⁵¹ (Supplementary Note).

Like other matrix factorization and clustering methods, GBCD should be viewed as a useful exploratory tool for identifying interesting substructures in large datasets. Like other existing tools, GBCD has limitations that users should be aware of. For example, GBCD is solving a nonconvex optimization problem, so its solutions will typically depend on initialization. In NMF, this issue is sometimes addressed by combining results across multiple runs with different initializations⁵; it is possible that a similar strategy could be helpful for GBCD. Another complication is how to choose $K$, the number of GEPs; see Methods for discussion of this.

Finally, we note that we do not expect GBCD to be uniformly superior to other methods such as NMF. For example, in the simulations combined NMF was essentially clustering cells by patient, which in some contexts could be the desired goal. The assumptions of GBCD—and in particular the assumption that the GEPs are orthogonal to each other—may be more appropriate in some applications than others. Because it may be difficult to know in advance which assumptions are appropriate for a given dataset, we recommend applying multiple methods and comparing their results. Our analyses showed that GBCD can produce results that differ considerably from existing methods, giving it the potential to provide new scientific insights.

Statistics and reproducibility

Details on how the statistical analyses were conducted are given in Methods and Supplementary Note. We used previously collected datasets; therefore, no statistical method was used to predetermine sample size, the experiments were not randomized and the investigators were not blinded to allocation during experiments and outcome assessment. Nonmalignant cells were excluded from our analyses because our method was intended specifically for analyzing malignant cells. Additional criteria for excluding cells and genes from the datasets are detailed in Methods.

Methods

All analyses used publicly available, deidentified data, so no ethical approval was required.

GBCD model and fitting

Below, we describe the steps that were taken to apply GBCD to scRNA-seq data and obtain the statistical quantities used in interpreting and reporting results. This involved the following four steps: (1) prepare the scRNA-seq data; (2) estimate the GEP memberships; (3) estimate the GEP signatures; and (4) quantify uncertainty in the GEP signatures. Each of these steps is described in the sections below. We also discuss some practical considerations in applying GBCD to scRNA-seq data.

GBCD is implemented using the EBMF methods in the R package flashier^31,56-58.

Preparation of scRNA-seq data

First, we combined the scRNA-seq data across tumors and studies into an $N\times J$ counts matrix, $X$. For the PDAC data and the simulated datasets, $X$ was a matrix containing UMI counts in which element $x_{ij}$ was the UMI count of gene $j$ in cell $i$ (see below for additional details about the PDAC data and the simulated datasets). The following steps were taken to prepare the PDAC data and the simulated datasets for GBCD.

Let $s_i$ denote the ‘size factor’ for cell $i$, which is defined as the total count over all genes, $s_i={\sum }_{j=1}^J{x}_{ij}$. (We note that other size factor definitions^59-61 are possible, and in some settings may be preferred.) We scaled the data by the size factors, added a ‘pseudo-count’ $c>0$, then took the log,

$$y_{ij}=\log (c+\tilde{s}∕s_i\times x_{ij})-\log c$$ (3)

in which $\tilde{s}≔\text{median}\{s_1,\dots ,s_N\}$ denotes the median size factor. We subtracted log $c$ from the log-transformed data to ensure that all $y_{ij}$ were non-negative. This also preserved sparsity in the original counts $x_{ij}$; that is, the shifted log count $y_{ij}$ was zero if, and only if, the original count $x_{ij}$ was zero.

Regarding the choice of $c$, different pseudocounts have been proposed (for example, ref. 30). Generally speaking, pseudocounts that are too small overemphasize differences in normalized expression between zero and nonzero UMI counts, whereas pseudocounts that are too large reduce differences in normalized expression⁶². We chose a ‘middle ground’ value of $c=0.1$; see refs. 30,56 for further discussion.

Note that the downloaded HNSCC data were already log-transformed and normalized. (See below for additional details about the HNSCC data.)

Estimating the GEP memberships

We estimated the GEP membership matrix $L$ by decomposing $Y{Y}^T$ using the EBMF framework³¹. Specifically, we modeled $Y{Y}^T$ as

$$\begin{gathered} Y{Y}^T=L{L}^T+ϵI_N+E \\ {e}_{ij}\stackrel{\text{i.i.d.}}{\sim }N(0,{\phi }^2) \end{gathered}$$ (4)

where $I_N$ denotes the $N\times N$ identity matrix, $E$ is an $N\times N$ matrix with entries $e_{ij}$, and $ϵ$, ${\phi }^2>0$ are unknowns to be estimated.

If the rows of $Y$ are centered, $Y{Y}^T$ is proportional to the cell-by-cell sample covariance matrix. Thus, we refer to equation (4) as a ‘covariance decomposition’ since it approximates $Y{Y}^T$ by $L{L}^T$, where $L$ is an $N\times K$ non-negative matrix.

GBCD assigns a ‘generalized binary’ prior independently to each entry of $L$,

$$l_{ik}\sim (1-{\pi }_k^l){\delta }_0+{\pi }_k^l{N}_{+}({\mu }_k,{\sigma }_k^2)$$ (5)

with parameters ${\mu }_k\ge 0$, ${\sigma }_k^2>0$, ${\pi }_k^l\in [0,1]$, in which ${\delta }_0$ denotes the point mass at zero and $N_{+}(\mu ,\sigma )$ denotes the left-truncated normal distribution with mean $\mu$ and variance ${\sigma }^2$, with the truncation point at zero. This prior was chosen to balance modeling flexibility—the ability to capture both discrete and continuous structures—and computational convenience. We call equation (5) the generalized binary (GB) prior because as ${\sigma }_k^2\to 0$, the prior supports only two possible values, 0 or ${\mu }_k$. We set the ratio ${\sigma }_k∕{\mu }_k$ to a small value to encourage discrete structures, while still allowing for continuous structures when they better fit the data. The other hyperparameters, ${\pi }_k^l$ and ${\mu }_k$, were estimated using the EBMF framework. Note that the diagonal scaling matrix, $D$, which is in the original model in equation (2) does not appear in equation (4). That is because the GB prior equation (5) already takes care of this scaling. The scaling matrix in equation (2) can be recovered as $D=\text{diag}({\mu }_1^2,\dots ,{\mu }_K^2)$.

Instead of fitting equation (4) directly, we actually fit a ‘relaxed’ matrix decomposition,

$$Y{Y}^T=L{\tilde{L}}^T+ϵI_N+E$$ (6)

with independent GB priors (equation (5)) on $L$ and $\tilde{L}$. Clearly, equation (6) is the same as equation (4) when $L=\tilde{L}$. While the model fitting algorithm does not require $L$ and $\tilde{L}$ be the same, fitting equation (6) usually produced highly concordant ${\mathbf{l}}_k$ and ${\tilde{\mathbf{l}}}_k$. After fitting equation (6), we retained only the components $k$ such that the Pearson correlation between ${\mathbf{l}}_k$ and ${\tilde{\mathbf{l}}}_k$ was greater than 0.8, and we (arbitrarily) used the left-hand factors, $L$, for the final GEP membership estimates.

It was shown in ref. 31 that fitting the ‘relaxed model’ (equation (6)) for any prior family reduces to solving a sequence of much simpler EBNM problems with the same prior family³¹. Fitting equation (6) with GB priors (equation (5)) was implemented in the R package flashier⁵⁸ together with a custom EBNM solver⁵⁷ for the GB prior. See the Supplementary Note for more details on this.

Estimating the GEP signatures

Once $L$ was estimated, we estimated $F$ by fitting an EBMF model to $Y$ with fixed $L$,

$$Y=L{F}^T+\tilde{E}$$ (7)

$${\tilde{e}}_{ij}\sim N(0,{\tilde{\phi }}_j^2)$$ (8)

where $\tilde{E}$ is an $N\times J$ matrix of independent error terms ${\tilde{e}}_{ij}$, and ${\tilde{\phi }}_j^2>0(j=1,\dots ,J)$ are additional unknowns to be estimated. This model is a semi-non-negative matrix factorization (SNMF)⁶³ because it places a nonnegativity constraint on $L$ but not $F$.

For GEP $k$, $f_{jk}$ approximately represents the LFC of gene $j$. It is reasonable to assume that only a subset of genes exhibit differential expression in GEP $k$. Thus, we assigned point-Laplace priors independently to all $f_{jk}$, which encourages shrinkage of small coefficients to zero while avoiding over-shrinkage of large coefficients:

$$f_{jk}\sim (1-{\pi }_k^f){\delta }_0+{\pi }_k^f\text{Laplace}({\lambda }_k)$$ (9)

where $\text{Laplace}(\lambda )$ denotes the zero-centered Laplace distribution with scale parameter $\lambda$. The parameters ${\pi }_k^f\in [0,1]$, ${\lambda }_k\ge 0$ were estimated within the EBMF framework. The EBMF model (equation (7)) with point-Laplace priors (equation (9)) was fitted to the data using flashier with an EBNM solver⁵⁷ for the point-Laplace prior. The initial estimate of $F^T$ was set to ${(L^{T}L)}^{-1}{L}^{T}Y$.

Quantifying uncertainty in the GEP signatures

The EBMF framework uses mean-field variational approximations for fast posterior computations³¹. Since mean-field variational approximations are known to underestimate uncertainty in the posteriors, we performed additional calculations to better estimate uncertainty in the LFCs. These calculations were then used to compute posterior $z$-scores⁶⁴ and other statistical quantities.

First, for each gene $j$, we fit a linear regression model in which the response variable was the vector of shifted log counts for gene $j$, and the regressor matrix was the matrix of GEP membership estimates $L$. We fit these linear regression models using the lm function in R. After fitting the regression model for each gene $j$, we extracted the LFC estimates ${\widehat{f}}_{j1},\dots ,{\widehat{f}}_{jK}$ and their corresponding standard errors ${\widehat{s}}_{j1},\dots ,{\widehat{s}}_{jK}$.

Next, we performed adaptive shrinkage, separately on the LFC estimates for each GEP $k$, to improve the accuracy of the standard errors. To implement this step, we used the ash function from the ashr R package⁶⁵. Adaptive shrinkage accepts as input a vector of effect estimates ${\widehat{f}}_{1k},\dots ,{\widehat{f}}_{jk}$ and associated s.e. ${\widehat{s}}_{1k},\dots ,{\widehat{s}}_{jk}$. The revised posterior mean estimates and s.e. values returned by adaptive shrinkage were then used to calculate the final test statistics, including posterior $z$-scores (defined as the posterior mean divided by the posterior s.e.) and local false sign rates (lfsr)⁶⁵.

Note that this approach to quantifying uncertainty in the LFC estimates assumes that the membership estimates are known, so this does not fully account for uncertainty in the matrix factorization. In practice, we found it to be an improvement over the uncertainty estimates provided by flashier.

Choice of $K$ in GBCD

Like other matrix factorization methods, GBCD requires user judgment in choosing $K$, the number of factors. The EBMF method that underlies GBCD requires users to set an upper bound, $K_{\max }$, on the number of factors, then provides a method to automatically select $K$ (up to $K_{\max }$). Thus, one possible strategy is to simply set $K_{\max }$ to be very large and rely on this automatic method to select $K$. However, for several reasons—not least that computation increases with $K$—in practice one might want to more strictly limit $K$. In experimenting with various values of $K$ in our analyses, we found that GEPs identified with a smaller $K$ were generally also present when a larger $K$ was used (Supplementary Fig. 16). Therefore, a larger $K$ generally allows for identifying finer structure at the expense of higher computational cost. The main disadvantage to limiting $K$ is that GBCD may miss interesting structure in the data. In practice, the value of $K$ needed to capture most of the interesting structure will depend on the complexity of the data, and this will likely increase with the number of cohorts, patients and tumor cells analyzed.

Computing requirements

The fast model fitting algorithms for EBMF make it possible to apply GBCD to moderately sized datasets (Supplementary Fig. 17). However, when $N$ is large (for example, in the PDAC dataset), it can be costly to compute $Y{Y}^T$—an $N\times N$ nonsparse matrix—then apply EBMF to that matrix. Thus, we developed an alternative implementation of EBMF that performs the model fitting computations using only the (sparse) matrix $Y$ without ever explicitly forming $Y{Y}^T$. This variant is mathematically equivalent to the original implementation but reduces computing costs when $N$ is large and $Y$ is sparse. In particular, this implementation reduces the per iteration computational complexity from $O(N^{2}K)$ to $O(SK)$, where $S$ is the number of nonzero entries in $Y$, and reduces the memory usage from $O(N^2)$ to $O(NK+S)$. For the HNSCC data, this implementation reduced running time and memory from 2 h and 16 Gb to 50 min and 2 Gb; for the PDAC data, this reduced running time and memory from 50 h and 200 Gb to 21 h and 48 Gb. The results contained in this paper were from the original implementation that explicitly computed $Y{Y}^T$. When we compared the two variants, the results were highly consistent (Supplementary Fig. 17). Computing costs could be reduced further by optimizing the number of backfitting iterations for estimating $L$ (Supplementary Note). Other improvements to the GBCD pipeline will probably be needed for handling very large scRNA-seq datasets.

Simulation design

We took the following steps to simulate scRNA-seq datasets. First, we simulated a UMI count matrix $X^{\text{null}}$ containing scRNA-seq measurements of $J=10,000$ genes in $N=3,200$ cells. The elements of this matrix were simulated independently using the following model:

$$x_{ij}^{\text{null}}\sim \text{Poisson}(s_i{2}^{{\theta }_{ij}})$$ (10)

$$\Theta =1_N{\alpha }^T$$ (11)

where $\Theta$ is an $N\times J$ matrix with entries ${\theta }_{ij}$, $\alpha$ is a $J\times 1$ vector of gene-specific intercept terms, and $1_N$ denotes the vector of ones of length $N$. The parameters of this model were estimated by fitting the Splat simulation model⁶⁶ to one of the PDAC datasets. Then we modified $X^{\text{null}}$ using binomial thinning⁶⁷ to produce an $N\times J$ matrix $X$ with elements $x_{ij}$ such that

$$x_{ij}\sim \text{Poisson}(s_i{2}^{{\tilde{\theta }}_{ij}})$$ (12)

$$\tilde{\Theta }=1_N{\tilde{\alpha }}^T+L{F}^T$$ (13)

where $\tilde{\alpha }$ is a $J\times 1$ vector of gene-specific intercept terms modified by the binomial thinning, and $L$ and $F$ are, respectively, $N\times K$ and $J\times K$ matrices representing GEP memberships and signatures for $K=11$ GEPs. Figure 1a shows the $L$ matrix for one of the simulated datasets.

So as not to unfairly disadvantage the NMF-based approaches, the true GEP signatures were all non-negative (that is, all differentially expressed genes were upregulated). Each GEP had 75–500 upregulated genes, with fold changes ranging from 1.5 to 11 (average of 3). The upregulated genes were different in each GEP.

We repeated this simulation procedure to generate a total of 20 datasets.

HNSCC data

The HNSCC data consisted of transcriptional profiles for 5,902 cells from 18 patients generated using SMART-Seq2 (ref. 8). Cells were previously classified as malignant or normal using several complementary approaches. Our GBCD analysis focused on tumors with at least 50 malignant cells. In total, we retained 2,176 malignant cells from ten primary HNSCC tumors and five matched lymph node metastases from these patients. We removed mitochondrial genes, ribosomal genes and genes expressed in fewer than 20 cells, leaving a total of 17,113 genes. Since SMART-Seq2 does not produce UMI counts, we ran GBCD on the normalized expression data, $y_{ij}={\log }_2({\text{TPM}}_{i,j}∕10+1)$, where ${\text{TPM}}_{i,j}$ is the transcript-per-million (TPM) value for gene $j$ in cell $i$. Note that these normalized data are equivalent to shifted log counts (equation (3)) (up to a constant of proportionality), with $c=\tilde{s}∕{10}^5$. See ‘Preparation of scRNA-seq data’ for details.

PDAC data

The PDAC data were from published scRNA-seq studies (A, B and C in Table 2). We excluded neuroendocrine tumors from all datasets, and analyzed only the tumors that were confirmed to be PDAC by histology. We further focused only on analyzing malignant cells, which were previously distinguished from normal cells based on transcriptional profiles and inferred copy number variation. We filtered out low-quality cells with a large fraction of mitochondrial counts (>10%), and genes expressed in fewer than 25 cells in any single cohort. We also removed all mitochondrial genes and ribosomal genes. These filtering steps left a total of 13,844 genes, and reduced the number of cells in datasets A, B and C from 31,195, 41,986 and 23,042 to 18,920, 10,936 and 5,814, respectively (Table 2). We normalized the UMI counts as described in ‘Preparation of scRNA-seq data’.

Gene set enrichment analyses

For each GEP, we performed a GSEA of the most strongly upregulated genes, which we defined as genes with an estimated fold change of at least 1.5 and being among the top 2% of genes by fold change. We tested gene sets from the MSigDB Hallmark and CP:Reactome gene sets^32,52,53.

Measuring concordance with literature-derived signatures

We compared our results with literature-derived gene expression signatures, which were ordered (ranked) sets of genes. For a GEP $k$ identified by GBCD (or another method) and a given literature-derived gene expression signature, we compared the distributions of the estimated $f_{jk}$ in two sets of genes: (1) the top-ranked 100 genes from the literature-derived expression signature, and (2) a set of control genes selected at random but matched to have similar average expression levels to (1). The matched control genes were selected using a similar approach to that in ref. 8: we grouped all genes into 50 bins based on their average expression level in all cells, then for each gene in (1) we randomly chose 100 genes from the same bin and added them to (2) so that the control set had a comparable distribution of average expression levels. Then we reported the $P$ value from the one-sided Wilcoxon rank-sum test that tests whether the median of $f_{jk}$ in the top-ranked genes (1) is smaller than the median in the control set (2).

Annotation of GEPs

We defined the following measures to better understand what sort of structure was captured by each GEP.

Patient-specificity score.

Let $m\in \{1,\dots ,M\}$ denote a patient and let $p_{m,k}$ denote the proportion of cells $i$ from patient $m$ that have membership to GEP $k$, which we defined as cells with $l_{ik}>0.05$. We also defined $m(k)$ as the patient with the highest proportion for GEP $k$, that is, $m(k)={\text{argmax}}_{m=1,\dots ,M}{p}_{m,k}$. We then defined the ‘patient-specificity score’ for GEP $k$ as

$${\upsilon }_k=1-\frac{\underset{m\ne m(k)}{\max }{p}_{m,k}}{p_{m(k),k}}$$ (14)

Note that ${\upsilon }_k$ is always between 0 and 1: a ${\upsilon }_k$ close to 1 means that GEP $k$ is predominantly active in a single patient; a ${\upsilon }_k$ close to 0 means that another patient has a similar percentage of cells with membership to GEP $k$. In our analyses, we annotated GEP $k$ as patient-specific if ${\upsilon }_k>0.5$; otherwise, we annotated it as shared.

Interpatient variation.

For each shared GEP $k$, we performed one-way analysis of variance on the GEP membership values ${\mathbf{l}}_k$, with patient as the explanatory variable. We then computed the proportion of total variation in membership explained by patient. This proportion also quantified how much the memberships varied between patients relative to within patients. We referred to this quantity as interpatient variation in our results.

Chromosomal score.

We also assessed to what extent a GEP signature could be predicted by the genes’ base-pair positions on the chromosome. For GEP $k$, we fit a nonparametric regression model with $f_{jk}$ as the response and the base-pair position of gene $j$ as the predictor. The position of gene $j$ was defined as the middle point between the transcription start site and transcription end site, in which the transcription start site and transcription end site were obtained from the GENCODE 19 database. We fit a separate nonparametric regression model for each autosomal chromosome. We fit these models using the R package mgcv⁶⁸. We then defined the chromosomal score as

$${\rho }_k=1-\frac{{\sum }_{r=1}^{22}{\text{deviance}}_{kr}}{{\sum }_{r=1}^{22}\text{null}\_{\text{deviance}}_{kr}}$$ (15)

where ${\text{deviance}}_{kr}$ is the deviance of the model fit to chromosome $r$, and $\text{null}\_{\text{deviance}}_{kr}$ is the deviance of a null model with no predictor other than the intercept. A larger ${\rho }_k$ means that the chromosomal positions of the genes better predict the LFCs.

Survival analysis in PDAC bulk RNA-seq data

We obtained PDAC bulk RNA-seq data and clinical annotations from refs. 40,44-46. For the survival analysis, we retained only the resected primary tumors that were histologically confirmed to be PDAC with no evidence of metastasis at diagnosis. This resulted in a total of 391 samples (TCGA⁴⁴, $n=102$; CPTAC⁴⁶, $n=87$; QCMG⁴⁰, $n=70$; PanCuRx⁴⁵, $n=132$). The gene expression data were log₂-transformed TPM values.

For each GEP 1–14 and for each literature-derived subtype signature, we computed an expression score from bulk RNA-seq data of each tumor, using the AddModuleScore function in Seurat²⁴. Specifically, we calculated the average expression level of the 200 top-ranked genes of the gene signature subtracted by the average expression level of a control gene set, similar to ref. 8. For GBCD GEPs, the 200 top-ranked genes were defined as those with the largest LFCs that were upregulated. These expression scores were then ‘$z$-score normalized’ for all tumors in a given study to have zero mean and unit s.d.

We performed stepwise variable selection in a Cox proportional hazards regression model with overall survival as the endpoint, and with the following variables: age, sex, tumor stage, and all the expression scores computed using Seurat. This analysis was implemented using the R package My.stepwise⁶⁹. In this analysis, age, sex and tumor stage were designated as variables that were always included in the model.

Computing environment

Computations with simulated and real datasets were performed in R v.4.1.0 (ref. 70), linked to the OpenBLAS v.0.3.13 optimized numerical libraries, on Linux machines (Scientific Linux v.8) with Intel Xeon E5-2680v4 (Broadwell) processors.

Data availability

The datasets analyzed are available for download from the following URLs: Puram et al.⁸ scRNA-seq data, Gene Expression Omnibus (https://www.ncbi.nlm.nih.gov/geo/), accession GSE103322; Chan-Seng-Yue et al.³⁶ scRNA-seq data, European Genome-Phenome Archive (https://ega-archive.org), accession EGAD00010001811; Peng et al.⁵⁴ scRNA-seq data, Genome Sequence Archive (https://ngdc.cncb.ac.cn/gsa/), accession PRJCA001063; Raghavan et al.⁴¹ scRNA-seq data, Single Cell Portal (https://singlecell.broadinstitute.org), accession SCP1644; PanCuRx bulk RNA-seq data⁴⁵, European Genome-Phenome Archive (https://ega-archive.org), accession EGAS00001002543; PDAC bulk RNA-seq data from TCGA, CPTAC and QCMG cohorts^40,44,46, cBioPortal (https://www.cbioportal.org); Molecular Signatures Database (MSigDB), https://www.gsea-msigdb.org; GENCODE release 19, https://www.gencodegenes.org. Some of these datasets, as well as results from our analyses of these datasets, are available via Zenodo at https://doi.org/10.5281/zenodo.8271036 (ref. 71).

Code availability

The gbcd R package is available via GitHub at https://github.com/stephenslab/gbcd/; gbcd v.0.1-20 is also available via Zenodo at https://doi.org/10.5281/zenodo.13921972 (ref. 71). The R package includes a step-by-step tutorial on using gbcd to analyze scRNA-seq data. The code used to perform the data analyses is available via Zenodo at https://doi.org/10.5281/zenodo.8271036 (ref. 72). Other software and R packages used in this work include: ashr v.2.2-54, https://github.com/stephens999/ashr/; batchelor v.1.8.1, https://doi.org/10.18129/B9.bioc.batchelor; cNMF v.1.4.3, https://github.com/dylkot/cNMF/; conos v.1.5.0, https://github.com/kharchenkolab/conos/; ebnm v.1.0-11, https://github.com/stephenslab/ebnm/; fastTopics v.0.6-135, https://github.com/stephenslab/fastTopics/; flashier v.0.2.32, https://github.com/willwerscheid/flashier/; infercnv v.1.8.1, https://github.com/broadinstitute/infercnv/; mgcv v.1.9-0, https://cran.r-project.org/package=mgcv; My.stepwise v.0.1.0, https://cran.r-project.org/package=My.stepwise; NNLM v.0.4.4, https://github.com/linxihui/NNLM/; Python v.3.10.10, https://www.python.org; rliger v.1.0.0, https://github.com/welch-lab/liger/; R v.4.1.0, https://www.r-project.org; Seurat v.4.1.1, https://satijalab.org/seurat/; scran v.1.20.1, https://doi.org/10.18129/B9.bioc.scran; seqgendiff v.1.2.3, https://github.com/dcgerard/seqgendiff/; and splatter v.1.16.1, https://github.com/Oshlack/splatter/.