Generative pretraining from large-scale transcriptomes for single-cell deciphering

Summary

Exponential accumulation of single-cell transcriptomes poses great challenge for efficient assimilation. Here, we present an approach entitled generative pretraining from transcriptomes (tGPT) for learning feature representation of transcriptomes. tGPT is conceptually simple in that it autoregressive models the ranking of a gene in the context of its preceding neighbors. We developed tGPT with 22.3 million single-cell transcriptomes and used four single-cell datasets to evalutate its performance on single-cell analysis tasks. In addition, we examine its applications on bulk tissues. The single-cell clusters and cell lineage trajectories derived from tGPT are highly aligned with known cell labels and states. The feature patterns of tumor bulk tissues learned by tGPT are associated with a wide range of genomic alteration events, prognosis, and treatment outcome of immunotherapy. tGPT represents a new analytical paradigm for integrating and deciphering massive amounts of transcriptome data and it will facilitate the interpretation and clinical translation of single-cell transcriptomes.

Graphical abstract

Highlights

• •tGPT models gene rankings via autoregressive language modeling
• •tGPT works effectively on fundamental single-cell analysis tasks
• •tGPT captures distinctive features of different cell types
• •tGPT learns expression signatures linked to genomic and clinical features

Automation in bioinformatics; Data processing in systems biology; Transcriptomics

Introduction

Rapid advancement in single-cell RNA sequencing leads to dramatic drop in sequencing cost and allows for millions of single-cell transcriptomes to be digitized in a single experiment simultaneously. The whole human body is estimated to have 30 trillion cells. Single-cell transcriptome sequencing provided an unprecedented resolution to distinguish different cell type clusters, depict hierarchical cell arrangement and decipher transitional cell states. To achieve this goal, multiple single-cell atlasing projects have been established internationally, including Human Cell Atlas (HCA),¹ Single Cell Expression Atlas (SCEA),² COVID-19 Atlas,³ Tabula Muris Atlas⁴ and Mouse Cell Atlas.⁵ The HCA project¹ aims to digitize all cells and create a reference map of the human body through community-driven initiative that researchers all around the world can contribute. SCEA² compiles and annotates publicly available single-cell transcriptomes across multiple species and different studies. The COVID-19 Atlas³ aims at elucidating molecular mechanism and therapeutic target of COVID-19 by generating single-cell atlas of SARS-CoV-2 infection in COVID-19 patients. The Tabula Muris⁴ and MCA⁵ atlases constitute the single-cell reference maps of mouse with millions of cells obtained from different organs. These atlasing projects pose tremendous challenge in the integration of diverse transcriptomes from different projects. However, single-cell transcriptomes are generated by different platforms and experimental protocols. They are sparse, noise and prone to batch effect.^6,7 Therefore, an analytical method to efficiently integrate ten millions of cells are urgently needed.

Over the past few years, deep learning approaches have led to seismic changes in image recognition and natural language understanding. The success of deep learning could largely attribute to the availability of big data, advancement in computational infrastructure, expressivity and scalability of the computational model. The deep learning model could adeptly handle super large-scale high dimensional data and assimilate real-world information. Owing to the exponential accumulation of millions of cell transcriptomes, elucidation of the reference map of single-cell transcriptomes with deep learning becomes an attractive application. Deep learning methods such as scVI,⁸ SAUCIE⁹ and INSCT¹⁰ have been developed for the analysis of single-cell transcriptomes.

The progress of artificial intelligence is undergoing a paradigm shift in computer vision and natural language processing. Deep neural networks based on transformer are becoming the de facto approach in wide variety of scenarios such as vision, language and reasoning.¹¹ Transformer-based models pretrained on broad data at scale continues to achieve state-of-the-art progress in image classification^12,13 and language understanding.^14,15,16 The success of these pretrained models can be attributed to their high expressivity and scalability enabled by transformer to assimilate feature representation from massive amount of unlabeled data. However, the investigation of single-cell transcriptome pretraining at scale has not been well studied.

In this study, we present a deep learning approach entitled tGPT toward integration of unlimited number of cells. tGPT is built on transformer that has been widely used in natural language understanding and image recognition. The transformer is an essential component and key success of foundation models because of its high expressivity and scalability.¹¹ tGPT takes as input the expression rankings of top-expressing genes rather than the actual expression levels. Rank-based methods for gene expression have been demonstrated to be insensitive to batch effects and data normalization.^17,18,19,20 tGPT is conceptually simple and empirically efficient. It models the occurrence of a gene in the context of its preceding neighbors’ rankings. We developed tGPT with 22.3 million cells and systematically evaluated tGPT on four heterogeneous datasets for sensitivity to batch-effect, delineation of clustering performance and inference of developmental lineages. We applied tGPT to bulk cancer tissue sequencing samples and found that features obtained from tGPT are significantly associated with diverse genomic alteration events, patients’ prognosis and treatment outcome of immunotherapy. tGPT represents a new analytical paradigm to integrate and decipher large-scale single-cell transcriptomes. It will facilitate the integration and clinical translation of large volume of single-cell transcriptome data.

Results

An overview of tGPT and its downstream applications

The analytical framework of tGPT (Figure 1) consists of three components: Development of tGPT, applications of tGPT for single-cell clustering, inference of developmental lineage, and interrogation of feature representation of bulk tissues in relation to genomic alterations, prognosis and treatment response of immunotherapy.

Figure 1.

A flowchart illustrating the framework of tGPT and its downstream applications

It consists of three components: development of tGPT, applications of tGPT for single-cell and bulk tissue transcriptomes.

tGPT is formatted as an autoregressive language model in that the output from the previous step is used as input to the next step. The input to tGPT is a sequence of gene symbols that are ranked by their expression levels. The purpose is to predict the index of the next gene in the dictionary in the context of all previous genes. The dictionary consists of 20706 protein-coding genes. tGPT is trained as an unsupervised generative pretraining task.¹⁶ Specifically, for a given cell, let $\mathcal{G}=\left\{G_1,G_2,\dots G_n\right\}$ denote the gene symbols that are sorted in a descending order according to their expression levels. We use the standard language modeling objective $\mathcal{L}\left(\mathcal{G}\right)={\sum }_i\mathit{log}P\left(G_i|G_{i-k},\dots ,G_{i-1};\mathit{\theta }\right)$ to maximize the likelihood. Here, $k$ is the width of context window and $\mathit{\theta }$ are the parameters of tGPT that is used to model the conditional probability. The neural network consists of 8 transformer decoder blocks²¹ with 1024 hidden units and 16 attention heads.

Quantitative evaluations of tGPT on clustering

We systematically evaluated the clustering performance of tGPT on four heterogeneous single-cell datasets of different sizes (50-586k cells) from different species and two bulk tissue sequencing datasets (Tables S1 and S2). These four single-cell datasets include Human Cell Atlas Census of Immune Cells²² (HCA, n = 282,558), Human Cell Landscape²³ (HCL, n = 586,135), Tabula Muris⁴ (n = 54,862) and Macaque Retina²⁴ (n = 124,965) dataset (See STAR Methods for description). The two bulk tissue datasets are Geno-type-Tissue Expression²⁵ (GTEx, n = 11,688) derived from 30 organs and The Cancer Genome Atlas²⁶ (TCGA, n = 9,318) consisted of 33 cancer types.

We observed that tGPT is insensitive to batch effect as benchmarked against with the other methods that support batch-correction such as ComBat,²⁷ MNN,²⁸ Harmony,²⁹ Seurat,^30,31 BBKNN,³² Scanorama,³³ Pegasus,³⁴ scVI,⁸ scArches,³⁵ iMAP³⁶ and DESC³⁷ as measured on the HCA dataset. tGPT achieved a comparable kBET acceptance rate³⁸ of 0.87 among the aforementioned batch-correction methods (Figure S1L). The UMAP plots of these batch-correction methods and their clustering metrics and grid-search results are provided in Figures S1A–S1K, 2, and 3, respectively.

Figure 2.

The clustering performance of tGPT on four single-cell and two bulk tissue datasets

(A) Radar charts depicting clustering metrics of tGPT, Pegaus, scVI and Scanpy across these six datasets.

(B–G) UMAP visualization of feature representations learned by tGPT on the HCA (B), HCL (C), Tabula Muris (D), Macque Retina (E), GTEx (F) and TCGA (G).NMI, Normalized Mutual information; ARI, Adjusted Rand Index; FMI, Fowlkes-Mallows Index.

Figure 3.

Distinct features of different cell types from the HCA dataset learned by tGPT

(A) Heatmap representation of attention head entropy for different cell types, and hierarchical clustering plot clustered these attention heads.

(B) Heatmap representation of attention head importance for different cell types.

(C) Dot plot illustrating the attribution scores for cell type specific genes, gray cell types annotated the clusters of marker genes.

The clustering performance of tGPT is robust with respect to the numbers of top-expression genes being used. We found that the performance of tGPT pretrained on the ranking of top 62 and 126 genes were comparable across these six datasets (Figure S4). In addition, we observed that clustering performance on features extracted from different transformer layers [Layer-1, …, Layer-8] are comparable and better than features extracted from the embedding layer across all these six datasets (Figure S4). For each method, we reported the best performance via grid-search to identify optimal values of two parameters that are most relevant to clustering (see STAR Methods). The results from grid-search were provided in Figures S5–S10 and Data S1. Quantitatively, tGPT achieved an Normalized Mutual information (NMI) ranged from 0.75 on HCA to 0.90 on GTEx, Adjusted Rand Index (ARI) from 0.53 on HCL to 0.84 on Tabula Muris and Fowlkes-Mallows Index (FMI) from 0.55 on HCL to 0.85 on Tabula Muris (Figure 2A). The clustering performance achieved by tGPT are comparable to the other methods such as Scanpy,³⁹ Pegasus³⁴ and scVI⁸ (Figures 2A and S11–S13). Grid-search results of these methods were provided in Figure S14. Running time of these methods were provided in Table S3.

Across these datasets, tGPT was capable of grouping cells with the same or similar types (Figures 2B–2G). On the HCA dataset, tGPT was able to identify cells at different developmental phases. For example, it can delineate B cells of different types such as naive B cells, precursor B (pre-B) cells and progenitor B (pro-B) cells and homologous cells, such as conventional DCs (cDCs) and plasmacytoid DCs (pDC), CD14⁺ and CD16⁺ monocytes. Less represented cell types such as megakaryocytes (0.32%) and MSCs (0.10%) were also captured by tGPT (Figure 2B). On the HCL dataset, tGPT was able to distinguish between immune cells and nonimmune cells as well as different cell types from fetus and adult such as fetal enterocytes and adult enterocytes (Figures 2C and S12). On the Tabula Muris dataset, tGPT was also able to delineate 55 distinct cell types originated from 20 mouse organs (Figures 2D and S13). On the Macaque Retina dataset, distinctive cell clusters from foveal and peripheral regions of fascicularis retina defined by tGPT are well matched with cell types defined in the original literature²⁴ (Figure 2E). On the GTEx dataset, tGPT is able to identify different tissues originated from lineage of organs (NMI = 0.90), and samples with similar histological structure are close together such as colon, small intestine and stomach (Figure 2F). On the TCGA dataset, different cancer types are well separated (NMI = 0.77). Cancer types with the same tissue of origin tend to clump together in the feature representation spaces captured by tGPT. For example, adenocarcinomas and squamous cell carcinomas are closely related in the UMAP plots, respectively. Different cancer subtypes originated from the same tissues are well separated such as lung cancer subtypes (e.g., LUAD and LUSC; Figure 2G), kidney cancer subtypes (e.g., KIRC, KIRP and KICH; Figure 2G) and breast cancer subtypes (e.g., Luminal A, Luminal B, HER+ and Basal cell carcinoma; Figure S15). In addition, tGPT achieved Bilingual Evaluation Understudy (BLEU) scores ≥0.69 four datasets examined in this study (Table S4). This suggested that tGPT obtained good quality in gene ranking generation.

Distinct features learned by tGPT are connected to cell types

We observed that the head entropy and importance of different cell types from the HCA dataset (See STAR Methods) are distinctive from each other. Cells of similar lineages or functions such as T-lineage cells exhibited similar entropy patterns (Figure 3A). The head importance is varying considerably for different cell types, however, cells of similar types are alike as compared with the other cell types (Figure 3B). For each cell type, we calculated the contribution of each gene on the cell final feature representation (See STAR Methods). Celltype specific genes have higher attribution scores (Figure 3C). For example, NKG7, FGFBP2, PRF1, GNLY, GZMA and GZMB are highly represented in cytotoxic T cells and NK cells (Figure S16A). PPBP and PF4 are also highly represented in megakaryocytes (Figure S16B). B-lineage cells have high attribution scores for both CD79A and CD79B. Attribution scores of MS4A1 and MZB1 are relative higher in memory B cells and plasma cells, respectively (Figure S16C). The attribution score of CST3 is higher among CD14⁺ monocytes, CD16⁺ monocytes, cDCs and pDCs. In addition, each specific cell types can be defined by specific genes with high attribution scores, for instance plasmacytoid dendritic cells (pDCs, IRF7), conventional dendritic cells (cDC, FCER1A), CD14⁺ monocytes (CD14) and CD14⁺ monocytes (FCGR3A) (Figure S16D).

Inference of developmental lineage

We used the feature representations learned by tGPT to construct cell pseudo-temporal trajectories on HCA and HCL datasets (See STAR Methods). On the HCA dataset, the developmental trajectories originated from stem cells and differentiated toward multiple biologically functional cell branches (Figure 4A): HSCs to erythroid cells⁴⁰ or DCs and monocytes (Figure 4B); naive T cells to cytotoxic T cells and NK cells⁴¹ (Figure 4C); pro-B cells to pre-B cells, then followed by matured naive B cells, and finally bifurcated into memory B cells or plasma cells⁴² (Figure 4D). In addition, we observed that the cell state signatures are aligned with cell developmental lineages (See STAR Methods). For instance, HSCs and pro-B cells are manifested by apparent progenitor signaling (Figure 4E). Naive and mature T cells are featured by distinguishable patterns (Figures 4F and 4G).

Figure 4.

Diffusion pseudo-time analysis on the HCA and HCL datasets

(A) The diffusion map of HCA dataset.

(B) Hematopoietic stem cells (HSCs) to erythroid cells or dendritic cells (DCs) and monocytes.

(C) Naive T cells to cytotoxic T cells and nature killer (NK) cells.

(D) Pro-B cells to plasma cells.

(E–G) Cell state signatures for progenitor signaling, naive signaling and cytotoxic signaling.

(H–K) The diffusion map of HCL dataset and its main branches. Token score is the norm of the learned token features extracted from tGPT.

On the HCL dataset, the developmental tree depicted three differential trajectories of fetal mesenchymal progenitor cells into different mature cell types (Figure 4H) with fetal cells at the center of the landscape. The fetal mesenchymal progenitor cells are differentiated into biologically functional fibroblasts (Figure 4I), enterocytes (Figure 4J), astrocytes and oligodendrocytes (Figure 4K).

Clinical significance of tGPT in bulk sequencing sample

Here, we demonstrated that tGPT is able to capture clinically significant patterns. On the TCGA dataset, we found that the importance scores are varying considerably for different attention heads among different layers. The importance score patterns can cluster different cancer types into distinct groups in that cancer of the same tissue-of-origin are closely related whereas cancers of different origins are well separated (See STAR Methods, Figure 5A). For example, skin cutaneous melanoma (SKCM) and uveal melanoma (UVM), glioblastoma multiforme (GBM) and brain lower grade glioma (LGG) are respectively located in the same clustering branches. In addition, we examined the association between attention head entropy and molecular alteration events (See STAR Methods). There are several attention heads exhibited significant association with tumor mutation burden (TMB) in the TCGA pan-cancer cohort and specifically in bladder urothelial carcinoma (BLCA), LUAD and LUSC (Figure 5B). We observed that attention heads also showed significant association with TP53 mutations at the pan-cancer level and across 9 cancer types (Figure 5C). There are also attention heads exhibited significant association with homologous recombination deficiency (HRD) and genome doubling (Figures 5D and 5E) at the pan-cancer level. The association of attention heads with HRD and genome doubling are statistically significant across 4 and 14 cancer types, respectively. Meanwhile, the attention heads exhibited prognostic significance at pan-cancer level (Figure 5E) and across 7 cancer types (Figure S17).

Figure 5.

The association of features learned by tGPT versus genomic alteration events and clinical phenotype

(A) Heatmap representation of attention head importance score across different cancer types on the TCGA dataset.

(B–F) Association of attention head entropy versus tumor mutation burden (B), TP53 mutation (C), homologous recombination deficiency (D), genome doubling (E) and overall survival (F) on the TCGA cohort.

(G and H) Heatmap representation of attention head importance and entropy on the urothelial carcinoma stratified by RECIST response. CR, complete response; PR, partial response; SD, stable disease; PD, progress disease.

(I) The varying entropy patterns from SD to PR to CR with PD as baseline.

(J) Exemplified violin plots depicting attention head entropy in SD/PD versus CR/PR.

(K) The varying of patterns of tumor evasion signature from SD to PR to CR with PD as baseline.

(L) The varying of patterns of T cell infiltration signature from SD to PR to CR with PD as baseline.

(M) Association between attention head entropy and overall survival on the urothelial carcinoma dataset.

(N) Exemplified survival curves stratified by attention head entropy.

In addition, we examined the attention head patterns in relation to immunotherapy in an immune checkpoint block (ICB) clinical trial of urothelial carcinoma consisted of 298 patients: 25 patients with CR, 43 with PR, 63 with SD and 63 with PD (See STAR Methods). We found that importance and entropy scores are distinguishable amongst patients with different therapeutic outcome (Figures 5G and 5H). We observed gradually varying entropy values from SD to PR to CR by taking the PD baseline (Figure 5I) and significant difference among 5 attention heads in patients with CR/PR versus SD/PD (Figure 5J). We quantified expression signatures such as tumor evasion and T cell immune infiltration attended by different attention heads (See STAR Methods). By taking PD as baseline, we observed a gradually decreasing patterns of tumor evasion and increasing patterns of T cell immune infiltration from SD to PR to CR (Figures 5K and 5L). The attention heads also exhibited prognostic significance in this clinical trial (Figures 5M and 5N).

Discussion

Efficient integration of accumulating large-scale single-cell transcriptomes is urgently needed. Here, we introduced a conceptually simple approach toward the integration of unlimited number of single-cell transcriptomes and examined its potential clinical translational relevance. The paradigm underpinning tGPT in essence is to predict the occurrence of a given gene with its previous context. We developed tGPT on a super large-scale single-cell transcriptome dataset that consists of 22.3 million cells and systematically evaluated its representation learning ability on different single-cell analysis tasks. We noted that tGPT was insensitive to batch effect and achieved competitive performance as compared with benchmark tools. The purpose of this study is to verify the validity of this new paradigm in deciphering large-scale transcriptome data, especially at the level of single-cell atlas. In addition, we showed that the pretrained tGPT model can be applied to bulk tissue sequencing samples to extract a variety of features exhibiting significant association with genomic alterations and response to immunotherapy treatment.

Artificial intelligence is undergoing a paradigm shift and the pretraining models based on transformer are becoming de facto standard in natural language processing and computer vision, achieving state-of-the-art across a wide range of tasks such as natural language understanding, image classification, video and audio recognition.¹¹ Representative pretraining models include BERT¹⁴ and GPT.¹⁵ The advantage of these pretraining models lie in its ability to assimilate real-world information from super large-scale unlabeled and high-dimensional data. This advantage brings an attractive solution for deciphering single-cell transcriptomes as millions of cells have been sequenced, which exemplified by 22.3 million cells collected in our study. This number is expected to increase exponentially in years ahead. There is no analytical tool that is designed and evaluated on such large volume of data. The high expressivity and scalability of transformer enable tGPT to learn rich representation from transcriptomes in a self-supervised manner. The high clustering performance in single-cell cluster delineation is probably attributable to better feature representation learned by tGPT. In addition, feature representation from tGPT is insensitive to batch effect as the acceptance rate of kBET derived from tGPT is evenly distributed among the other tools that explicitly used batch information for batch-correction. This is probably because of the use of rankings of top expressing genes rather than actual expression levels by tGPT. tGPT is quite different from the other integration tools^30,31,39 as the later use the actual expression levels of highly variable genes (HVGs) and the batch information. The independence of tGPT on batch information makes it attractive for integration of super large-scale transcriptomes because the batch information is not always available and often neglected by researchers.

The clustering performance in delineating single-cell clusters is robust with respect to the number of top expressing genes used and feature representation extracted from different tGPT transformer layers. The clustering metrics obtained from 62 top-expressing genes are comparable to the use of 126 top-expressing genes (Figure S4). This suggested that the rankings of 62 top-expressing genes are sufficient for cell cluster definition. The idea underpinning tGPT is to predict the occurrence of a gene in the context of the occurrences of its preceding neighbors. Thistype of pretraining is not directly related to cell clustering. This does not guarantee that feature representation from the last transformer layer could give rise to better clustering as compared with representation from its preceding layers. In our evaluation, the cluster metrics obtained from different transformer layers are comparable and consistently better than the embedding layer (Figure S4). In addition, we observed that cell-type specific genes have high attribution scores albeit only the rankings are used during pretraining. This finding can partially explain why features derived from tGPT could lead to high performance in cell clustering. Although this study also uses gene rankings as we did in our previous study,⁴³ they are theoretically different. tGPT builds on autoregressive language modeling¹⁶ whereas the model developed in our previous study used masked language modeling.¹⁴ More importantly, we explored the feature patterns learned by tGPT in bulk tissues, which were not investigated in our previous work.

A new finding emerged from our study is that the pretrained tGPT model can be applied to bulk tissues. On the GTEx dataset, the feature representations of different organs extracted from tGPT can divide samples into distinct clusters, aligning with organs. On the TCGA dataset, we observed that different cancer types are well separated and cancers of the same origins are more closely related, which is consistent with previous report.⁴⁴ In addition, the feature patterns of TCGA samples exhibited consistent and significant association with genomic alterations. This indicated that rankings of top-expressing genes carry information about alterations in tumor tissues. Meanwhile, the feature patterns derived from tGPT are distinctive among patients with different treatment outcomes for immunotherapy. Token together, our finding would facilitate translational research enabled by super large-scale transcriptomes.

We focused two main directions of tGPT for future development. First, tGPT can be used to generate large-scale reference mapping with the availability of large-scale disease reference datasets and phenotypes. Second, tGPT can be further investigated for clinical application such as treatment guiding and prognostic prediction.

Conclusion

In summary, we systematically verified a new, simple and effective analytical paradigm for integration of super large-scale transcriptomes and its implications in clinical translation.

Limitations of the study

First, tGPT uses only the top-expressing genes; therefore, it may miss the information that is specifically represented within the low-expressing genes. Second, tGPT uses gene expression rankings but not actual expression levels. Thus, the fold changes among genes are neglected and this can affect biological interpretation. Third, the language modeling objective function used by tGPT is to predict the gene rankings, which is not directly related to biological issues. Therefore, further study is required to investigate associations between prediction of gene rankings and biological functions.

STAR★Methods

Key resources table

REAGENT or RESOURCE	SOURCE	IDENTIFIER
Deposited data
Raw and analyzed data	https://data.humancellatlas.org/explore/projects/cc95ff89-2e68-4a08-a234-480eca21ce79?catalog=dcp1	Census of Immune Cells
Raw and analyzed data	https://www.ncbi.nlm.nih.gov/geo/download/?acc=GSE134355&format=file	GSE134355
Raw and analyzed data	https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE118546; https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE118852	GSE118480
Raw and analyzed data	https://www.ncbi.nlm.nih.gov/geo/download/?acc=GSE96583	GSE96583
Software and algorithms
Scanpy (v1.6.0)	(Wolf et al., 2018)39	https://github.com/theislab/scanpy
Pegasus (v1.4.3)	(Bo Li el at., 2020)	https://github.com/lilab-bcb/pegasus
scVI(v0.6.8)	(Lopez et al., 2018)8	https://github.com/theislab/scvelo
MNN (v1.8.0)	(Laleh Haghverdi et al.,2018)28	https://github.com/MarioniLab/MNN2017
Combat (v1.8.0)	(Jean-Philippe Fortin et al.,2017)	https://github.com/Jfortin1/ComBatHarmonization
Harmony (v0.1.6)	(Ilya Korsunsky et al., 2019)29	https://github.com/immunogenomics/harmony
Seurat (v3.1.5)	(Butler et al., 2018)30	https://satijalab.org/seurat
Scanorama (v1.7.1)	(Brian Hie el al.,2019)33	https://github.com/brianhie/scanorama
DESC (v2.1.1)	(Xiangjie Li et at., 2020)37	https://eleozzr.github.io/desc/
iMAP (v1.0.0)	(Dongfang Wang et at., 2021)36	https://github.com/Svvord/iMAP
scArches (v 1.7.0)	(Mohammad Lotfollahi et al., 2021)35	https://github.com/theislab/scarches
BBKNN (v 1.7.1)	(Krzysztof Polański et al., 2020)32	https://github.com/Teichlab/bbknn
tGPT	(Hongru Shen et al., 2023)	https://github.com/deeplearningplus/tGPT

Resource availability

Lead contact

Further information and requests for resources and materials should be directed to andwill be fulfilled by the lead contact, Xiangchun Li (lixiangchun2014@foxmail.com).

Materials availability

This study did not generate new unique reagents.

Experimental model and subject details

We collected the transcriptomes of 22.3 million single-cells (Table S1), 9318 bulk tissue transcriptomes of TCGA cohort from the supplemental data of pan-cancer immune landscape study,²⁶ 11,688 bulk tissue transcriptomes from GTExdatabase²⁵ and 298 bulk tissue transcriptomes from the clinical trial study on immunotherapy for urothelial carcinoma.⁴⁵ We discarded mitochondrial genes, ribosomal genes and non-protein coding genes for the single-cell data. Four single-cell and two bulk tissue sequencing datasets are used in downstream evaluation of tGPT. Annotated cell labels provided by the original studies are used as the ground truth label (Table S2).

Human Cell Atlas Census of Immune Cells (HCA)

Bone marrow cells (n = 282,588) from 64 healthy donors in Human Cell Atlas (HCA) project. The data are subjected to 10x sequencing protocol²² and contained 18 cell types such as hematopoietic stem cells (HSCs), mesenchymal stem cells (MSCs), erythrocytes, megakaryocytes and different kinds of immune cells.

Human cell Landscape (HCL)

HCL dataset includes 586,135 human cells obtained from a Chinese Han population,²³ the dataset encompasses samples of fetal and adult tissue and covered 60 human tissue types, and are subjected to Microwell-seq protocol.

Tabula Mursi

The Tabula Muris dataset (n = 54,865) is consisted of single-cells sorted by FACS from Mouse Cell Atlas⁴ across 20 different organs subjected to 10x and Smart-seq2 sequencing protocols.

The Cancer Genome Atlas (TCGA)

The TCGA dataset is consisted of 9,318 bulk samples with primary cancer and matched normal samples spanning 33 cancer types.

Genotype-Tissue Expression Project (GTEx)

The GTEx dataset includes 11,688 bulk samples across 30 organs obtained from healthy donors.

Known marker genes of different cell types are curated from CellMarker database⁴⁶: plasma cell (MZB1), DCs and monocytes (CST3, FCER1A, IRF7, CD14 and FCGR3A), megakaryocyte (PPBP and PF4), B cell (CD79A, CD79B and MS4A1) NK and cytotoxic T cell (NKG7, FGFBP2, GNLY, GZMA, GZMB and PRF1).

Cell state signatures are curated from CellMarker database,⁴⁶ including progenitor signaling (STMN1, TUBA1B and HIST1H4C), naïve signaling (CCR7, LEF1 and SELL) and cytotoxic signaling (GZMA, CD8A, CD8B, GZMB, PRF1, IL2, GNLY, GAMK, IFNG and NKG7).

T cellinfiltration signature is obtained from CellMarker database⁴⁶; it consists of CD3D, CD3E and CD8A.

Tumor evasion signature is curated from the Figure 1 of a previous study.⁴⁷

Method details

Input preprocessing

The input sequence list of top-expressing genes was obtained via descending sorting. The input to tGPT was formulated as [<s>, G₁, G₂, G₃, …,<e>], where G₁, G₂ and G₃ are gene symbols and<s>and<e>are two special tokens respectively added to the start and end of the input sequence. The input sequence is padded with special token<pad>if its length is less than a predefined value. The input sequence list is truncated if its length exceeds the predefined value. We evaluated a length of 64 and 128 in this study. The dictionary used by tGPT consists of 20706 protein-coding genes.

The architecture of tGPT

Embedding layer transforms the input gene symbols into a real-value matrix that carries the information on gene token embedding and position encoding. The gene token embedding was obtained via an embedding layer (parameterized as $W_e$) that maps the indices of input genes obtained from the gene symbol dictionary to real-value space. The position encoding (parameterized as $W_p)$ carries information on the sorted gene rankings. For an input sequence $U=\left\{G_{-k},\dots G_{-1}\right\}$, where k is the width of context window, the embedding layer injects position encoding onto gene token embedding as:

$$h_0=U{W}_e+W_p$$

Transformer decoder blocks applies multi-headed masked self-attention over the input embeddings followed by position-wise feed-forward layers, then through a softmax layer. tGPT use a multi-layer of transformer decoder.²¹

$$h_l=\psi \left(h_{l-1}\right)\forall i\in \left[1,n\right]$$

$$P\left(U\right)=\phi \left(h_n{W}_t^T\right)$$

where $\psi$ is the transformer decoder block and $\phi$ is the softmax layer, and $W_t$ is the embedding matrix of the $l$^th decoder block.

Masked Self-Attention is a variant self-attention mechanism.⁴⁸ Each attention head adopts the scale dot-product attention to map a query and a set of key-value pairs to an output. The input consists of query and key of dimension $d_k$, and value of dimensions $d_v\text{.}$ Self-attention is calculated as dot products of the query ($Q_i$) with key ($K_i$) divided each by $\sqrt{d_k}$ and multiply with value ($V_i$) after sofmax transformed⁴⁹:

$$SelfAtt{n}_i\left(Q_i,K_i,V_i\right)=soft\mathit{max}\left(\frac{Q_i{K_i}^{\mathrm{T}}}{\sqrt{d_k}}\right)V_i$$

Masked self-attention is implemented with the aid of attention mask. It basically always scores the future tokens as 0 so tGPT cannot pick from future. The multi-head self-attention is formulated as:

$$MultiHead\left(Q,K,V\right)=Concat\left(SelfAtt{n}_1,...,SelfAtt{n}_h\right)W^O$$

where $W^O\in {\mathbb{R}}^{h{d}_v\times d_{model}}$ denotes the learned output projection matrix.

Position-wise FeedForward neural network is a layer with fully-connected feed-forward layer. This layer consists of two linear transformations with a ReLu activation function in between:

$$FFN\left(x\right)=\mathit{max}\left(0,x{W}_1+b_1\right)W_2+b_2$$

where W₁ and W₂ are weight matrices and $b_1$ and $b_2$ are the bias.

Training scheme

tGPT was pretrained with a batch-size of 64 for 100 epochs. We used Adam with β₁ = 0.9, β₂ = 0.95, weight decay of 0.01 and a learning rate of 0.003. The learning rate is warmed up for four epochs, and then decays to 0 following a cosine schedule.¹² tGPT was trained with PyTorch (version 1.7.1) and transformers (version 4.10.0) on NVIDIA DGX A100 with 8 GPUs each with 40 Gb memory.

Clustering on feature representation from tGPT

We respectively extracted the feature representations from the embedding layer and 8 different transformer layers. The extracted features were used to construct K-Nearest Neighbors (KNN) graphs for subsequent community detection by Leiden algorithm⁵⁰ implemented in Scanpy (version 1.8.1). We performed grid-search to identify optimal values of two parameters n_neighbors and resolution that are the most relevant for clustering. Batch-correction was not applied in clustering. The value of n_neighbors examined was ranged from 5 to 100 with step of 5. The value of resolution examined was ranged from 0.1 to 2 with step of 0.2. The uniform manifold approximation and projection (UMAP) visualization⁵¹ is used.

Features derived from self-attention

Entropy of the self-attention matrices for a given input sequence is calculated as⁵²:

$$Entrop{y}_{\alpha }\left(x_i\right)=-\sum _{j=1}^i{\alpha }_{i,j}\left(x\right)\log \left({\alpha }_{i,j}\left(x\right)\right),$$

where α is the self-attention matrix and ${\alpha }_{i,j}$ is the attention weight between the i^th and j^th tokens. We averaged the entropy of all cells in a cluster to derive a cluster-level entropy.

Head importance score⁵³ is defined as the influence of input on head output. It is calculated via gradient backpropagation, formulated as:

$$I_h={\mathbb{E}}_{x\sim X}\left|{{SelfAttn}_h\left(x\right)}^T\frac{\partial \mathcal{L}\left(x\right)}{{\partial SelfAttn}_h\left(x\right)}\right|$$

where x is the input sequence and $\mathcal{L}\left(x\right)$ is the corresponding loss given the input. $I_h$ is high score while ${SelfAttn}_h\left(x\right)$ is liable to have a large effect on the model.

Token attribution score is defined as the norm of the learned token features ($x_i$) extracted from tGPT, which is defined as:

$$Attribution=\sqrt[2]{x_1^2+x_2^2+...+x_N^2}$$

Attention analysis in relation to signaling

We define an attention-based pathway signaling score in a similar way as⁵⁴:

$$p_{\alpha }\left(f\right)=\sum _{x\in X}\sum _{i=1}^x\sum _{j=1}^{x}f\left(i,j\right){\alpha }_{i,j}\left(x\right)/\sum _{x\in X}\sum _{i=1}^x\sum _{j=1}^x{\alpha }_{i,j}\left(x\right)$$

where ${\alpha }_{i,j}$ is the attention weight between the i^th and j^th gene. For a given gene signature, we set $f\left(i,j\right)$ = 1 if the i^th gene or j^th gene occurs in that gene set.

Diffusion pseudo-time maps construction

We constructed the diffusion pseudo-time maps using package Pegasus³⁴ (v1.4.3), and the cell trajectory was visualized with force-directed layout embedding (FLE) algorithm.⁵⁵ We set δ and nδ to its default values: δ = 2.0 and nδ =5,000.

Firstly, we used the features obtained from last transformer decoder blocks to construct affinity matrix of cells $W_{n\times n}$, and the top-k nearest neighbor cells were find by community detection algorithm⁵⁶ and the HNSW algorithm,⁵⁷ and the formula of affinity matrix is define as:

$$K\left(x,y\right)={\left(\frac{2{\sigma }_x{\sigma }_y}{{\sigma }_x^2+{\sigma }_y^2}\right)}^{\frac{1}{2}}\exp \left(-\frac{{\Vert x-y\Vert }^2}{{\sigma }_x^2+{\sigma }_y^2}\right)$$ (Equation 1)

$$k^{\prime }\left(x,y\right)=\frac{K\left(x,y\right)}{q\left(x\right)q\left(y\right)}$$ (Equation 2)

$$W\left(x,y\right)=\{\begin{array}{l}{k}^{\prime }\left(x,y\right),y\in n\left(x\right)/x\in n\left(x\right)\\ 0,otherwise\end{array}$$ (Equation 3)

The Equation 1 represented the distance between cell-x and cell-y, ${\sigma }_x$ is the x’s local kernel width, x and y are features of last transformer decoder block for cell-x and cell-y. The affinity matrix W was calculated as the density-normalized kernel according to Equation 3.

We then calculated the Markov chain transition matrix P and the symmetric transition matrix Q as the formula:

$$D=diag\left(\sum _{y}W\left(x,y\right)\right)$$

$$P=D^{-1}W,Q=D^{-\frac{1}{2}}W{D}^{-\frac{1}{2}}$$

The symmetrical matrix Q can be decomposed as UAU^T. Let $\mathrm{\Psi }=D^{-\frac{1}{2}}U$. A family with parameter timescale of t for approximated diffusion maps ${\left\{{\mathrm{\Psi }}_t\right\}}_{t\in \mathbb{N}\cup \left\{\infty \right\}}$ is defined as:

$${\mathrm{\Psi }}_t\left(x_i\right)=\left(\begin{array}{l}{\lambda }_1^t{\mathrm{\Psi }}_1\left(i\right)\\ {\lambda }_2^t{\mathrm{\Psi }}_2\left(i\right)\\ ⋮\\ {\lambda }_{n-1}^t{\mathrm{\Psi }}_{n-1}\left(i\right)\end{array}\right)$$

$${{\mathrm{\Psi }}_t}^{\prime }\left(x_i\right)=\sum _{t^{\prime }=1}^t{\mathrm{\Psi }}_{t^{\prime }}\left(x_i\right)=\left(\begin{array}{l}{\lambda }_1\frac{1-{\lambda }_1^t}{1-{\lambda }_1}{\mathrm{\Psi }}_1\left(i\right)\\ {\lambda }_2\frac{1-{\lambda }_2^t}{1-{\lambda }_2}{\mathrm{\Psi }}_2\left(i\right)\\ ⋮\\ {\lambda }_{n-1}\frac{1-{\lambda }_{n-1}^t}{1-{\lambda }_{n-1}}{\mathrm{\Psi }}_{n-1}\left(i\right)\end{array}\right)$$

Benchmark methods

We also performed single-cell analysis using Scanpy (version 1.6.0), Pegasus (version 1.4.3) and scVI (version 0.13.0). Batch-correction was performed with MNN (version 1.8.0),²⁸ Combat (version 1.8.0),⁵⁸ Harmony (version 0.1.6),²⁹Seurat (version 3.1.5),^30,31 Pegasus (version 1.4.3),³⁴ Scanorama (version 1.7.1),³³ DESC (version 2.1.1),³⁷ iMAP (version 1.0.0),³⁶ scVI (version 0.13.0),⁸ scArches (version 1.7.0),³⁵ BBKNN (version 1.7.1).³²

Scanpy is a comprehensive toolkit for analyzing single-cell transcriptome. We first filtered out cells with the number of expressing genes <200 or mitochondrial counts> 30%. We used the function scanpy.pp.highly_variable_genes to selected highly variable genes by setting max_mean to 3 and min_mean to 0.0125, which are the default values. We then applied clustering pipeline and grid-search to perform single-cell clustering on KNN graph. The UMAP is used for visualizing clustering result.

scVI is a deep generative model for mining the single-cell omics data. We filtered out cells with the number of expressing genes <200 or mitochondrial counts> 30%, and selected HVGs with scanpy.pp.highly_variable_genes by setting max_mean to 3 and min_mean to 0.0125. We used the default parameter of scVI to extract the 10 latent features. These latent features were used to construct KNN graphs for community detection by Leiden algorithm.⁵⁰

Pegasus is complete single-cell analysis pipline that is efficient on large datasets. We used the recommended parameters: min_genes of 500, max_genes of 6000, and percent_mito of 10. We identified the robust genes with the default percent_cells of 0.05. Single-cell clustering was performed on KNN graph followed by Leiden algorithm⁵⁰ for community detection.

Quantification and statistical analysis

Clustering and batch-effect metrics

We used Adjusted Rand Index (ARI), Normalized Mutual information (NMI) and Fowlkes-Mallows Index (FMI) to measure clustering performance. We used the kBET acceptance rate³⁸ as a measurement of batch-effect. The clustering metrics of ARI, NMI and FMI were calculated with sklearn (version 0.21.2). kBET acceptance rate is computed with Pegasus (version 1.4.3).

ARI is calculated based on the contingency table summarizing the truth labels and clustering, and the rows and columns represent truth and clustering labels in the contingency table, respectively. The formula is as follows:

$$ARI=\frac{{\sum }_{ij}\left(\begin{array}{l}{n}_{ij}\\ 2\end{array}\right)-\left[{\sum }_i\left(\begin{array}{l}{a}_i\\ 2\end{array}\right){\sum }_j\left(\begin{array}{l}{a}_j\\ 2\end{array}\right)\right]/\left(\begin{array}{l}n\\ 2\end{array}\right)}{\frac{1}{2}\left[{\sum }_i\left(\begin{array}{l}{a}_i\\ 2\end{array}\right)+{\sum }_j\left(\begin{array}{l}{a}_j\\ 2\end{array}\right)\right]-\left[{\sum }_i\left(\begin{array}{l}{a}_i\\ 2\end{array}\right){\sum }_j\left(\begin{array}{l}{a}_j\\ 2\end{array}\right)\right]/\left(\begin{array}{l}n\\ 2\end{array}\right)}$$

where $n_{ij}$ denoted the numbers of cell in common between clustering labels and truth labels, $a_i$ the sum of $i^{th}$ row and $a_j$ the sum of $j^{th}$ column of the contingency table.

NMI is also used to measure the similarity between the clustering labels and actual labels. We assumed that the clustering labels and actual labels of N cells are U and V, and the entropy of U and V is as the following formula:

$$H\left(U\right)=-\sum _{i=1}^{\left|U\right|}P\left(i\right)\log \left(P\left(i\right)\right)$$

$$H\left(V\right)=-\sum _{j=1}^{\left|V\right|}{P}^{\prime }\left(j\right)\log \left(P^{\prime }\left(j\right)\right)$$

where $p\left(i\right)=\left|U_i\right|/N$ is the probability that a cell picked at random from U falls into $U_i$, $p^{\prime }\left(j\right)=\left|V_j\right|/N$ is the probability that a cell picked at random from V falls into $V_j$. We then calculated the mutual information (MI) between U and V, and normalized the mutual information:

$$MI\left(U,V\right)=\sum _{i=1}^{\left|U\right|}\sum _{j=1}^{\left|V\right|}P\left(i,j\right)\log \left(\frac{P\left(i,j\right)}{P\left(i\right)P^{\prime }\left(j\right)}\right)$$

$$NMI\left(U,V\right)=\frac{MI\left(U,V\right)}{\sqrt{H\left(U\right)H\left(V\right)}}$$

where $p\left(i,j\right)=\left|U_i\cap V_j\right|/N$ is the probability that a cell picked at random falls into classes $U_i$ and $V_j$.

Fowlkes-Mallows Index (FMI) is used to measure the consistency between clustering results and real category, and the range of index is from 0 to 1. The FMI metric is denfined as the geometric mean between of the precision and recall:

$$FMI=\frac{TP}{\sqrt{\left(TP+FP\right)\left(TP+FN\right)}}$$

where TP is true positive, FP false positive, FN false negiative.

kBET acceptance rate is a measurement of batch effect. We assumed that the dataset of cells with batches of m, and there are $n_j$ cells in batch j. The batch mixing frequency denotes as:

$$f=\left(f_1,\cdots ,f_m\right)$$

where $f_j=\frac{n_j}{N}$. Then, we calculated the number of neighbors of cell-i belonging to batch j is $n_{ji}^k$. Its ${\chi }^2$ test statistic and p-value with degrees of (m-1) are defined as follows:

$$k_i^k=\sum _{j=1}^m\frac{{\left(n_{ji}^k-f_j·k\right)}^2}{f_j·k}$$

$$p_i^k=1-F_{m-1}\left(k_i^k\right)$$

where $F_{m-1}\left(x\right)$ represents the cumulated density function. The kBET acceptance rate is defined as the percentage of cells that accept the null hypothesis at significance level α as follows:

$$kBET-rate=\frac{{\sum }_{i=1}^{N}I\left(p_i^k\ge \alpha \right)}{N}\times 100\%\text{,}$$

I(x) is the indicator function where I(x) = 1 if x> 0 otherwise I(x) = 0. We used Pegasus (v1.4.3) to calculate the kBET acceptance rate by setting K and α to 5 and 0.01, respectively.

Bilingual Evaluation Understudy (BLEU) is an algorithm for evaluating match variable length phrases between output and the reference sequence.⁵⁹ The basic metric requires the calculation of a brevity penalty $P_B$ as:

$$P_B=\{\begin{array}{l}1,c>r\\ e^{\left(1-r/c\right)},c\le r\end{array}$$

Where r is the length of the reference sequence, and the length of predicted sentence is c.

BLEU score is calculated as:

$$\text{BLEU}=P_B\exp \left(\sum _{n=0}^N{w}_n\log p_n\right)$$

$w_n$ are the positive weights summing to one. $p_n$ is the n-gram precision and it is calculated using n-grames with a maximum length of N.

Additional resources

This study did not generate additional data.

Published: April 20, 2023

Data and code availability

• •All the gene expression matrices were downloaded from public databases. The source list of these datasets was provided in the key resources table and Table S1. Source code is available at https://github.com/deeplearningplus/tGPT.
• •Any additional information required to reanalyze the data reported in this paper is available from the lead contact upon request.