Dissection of tumoral niches using spatial transcriptomics and deep learning

Summary

This study introduces TG-ME, an innovative computational framework that integrates transformer with graph variational autoencoder (GraphVAE) models for dissection of tumoral niches using spatial transcriptomics data and morphological images. TG-ME effectively identifies and characterizes niches in bench datasets and a high resolution NSCLC dataset. The pipeline consists in different stages that include normalization, spatial information integration, morphological feature extraction, gene expression quantification, single cell expression characterization, and tumor niche characterization. For this, TG-ME leverages advanced deep learning techniques that achieve robust clustering and profiling of niches across cancer stages. TG-ME can potentially provide insights into the spatial organization of tumor microenvironments (TME), highlighting specific niche compositions and their molecular changes along cancer progression. TG-ME is a promising tool for guiding personalized treatment strategies by uncovering microenvironmental signatures associated with disease prognosis and therapeutic outcomes.

Graphical abstract

Highlights

• •TG-ME integrates transformer and GraphVAE for cancer niche analysis
• •Achieves superior clustering and profiling tumor microenvironments
• •Leverages multimodal data for comprehensive TME analysis
• •Enables precise identification of tumor microenvironment composition

Microenvironment; Cancer; Transcriptomics; Artificial intelligence

Introduction

In the realm of cancer research, dissecting the interplay between tumor cells and their surrounding tissue microenvironment (TME) stands as a pivotal endeavor for unraveling the mechanisms governing progression, metastasis, and responses to treatment.^1,2 TME is interpreted as a spatial region of cancer tissue that comprises tumor cells, multiple immune cells, fibroblasts, endothelial cells, and other cell types together with the extracellular matrix. Serving as a repository for both upstream stimuli to resident cells and downstream cellular products, the TME facilitates closed-loop inter- and intra-cellular regulations³ and influences tumor progression,³ metastasis [1], tumorigenesis, and treatment resistance, underscoring its importance in cancer research. Previous studies have shown that understanding the TME in cancer holds great promise for identifying therapeutic targets and developing personalized treatment strategies,⁴ underscoring a better understanding of TME for cancer prognosis and treatment. Given its heterogeneous nature, unraveling the cellular and niche composition, spatial cellular localization, and gene expression profiles within the TME becomes a crucial yet historically challenging task.⁵ To address this challenge, spatial transcriptomics (ST) technologies have attracted significant research efforts to enable the visualization and analysis of gene expression profiles in their spatial context in tissues.^{6,7,8,9,10,11,12} Common experimental methods for ST include immunohistochemistry (IHC)^13,14 and fluorescent in situ hybridization.^{10,15,16,17,18,19,20,21} For this last technology, 10X Genome Visium adopts in situ capturing, offering whole transcriptome data with a resolution of 55 μm of diameter known as spot-based technology, accommodating 1–10 cells per spot based on the tissue type and cell size.^6,9,11,12 In the other hand, NanoString CosMx spatial molecular imager (SMI) uses in situ hybridization technology, resolving 50 nm, the typical size of a cell. NanoString first SMI generation provides the spatial locations of about 980 genes at a sub-cellular resolution, and includes the use of morphological images, enabling the characterization of transcription profiles within individual cells.¹⁰ NanoString SMI anticipates unveiling an upgraded technology capable of whole-genome examination at single-cell level in the year 2025, while company 10X with their Visium HD technology aims to profile the whole transcriptome with single-cell resolution in the next few years. These new ST technologies impose the need for innovative tools capable of analysis and interpretation of high-dimensional multimodal datasets. Given the complexity of the spatial transcriptomics data and morphological images involved in the study of tumoral niches, the use of advanced AI techniques like the one proposed in this framework is a reasonable approach that can be accomplished through the implementation of a pipeline using the best tools for every stage. These stages include cell typing prediction,^22,23,24 niche prediction,^{2,10,25,26,27,28,29,30} and identification of cross-talks between specific cell types and the TME.^30,31

In the past several deep learning (DL)^{27,32,33,34,35} and statistical methods^{2,10,12,25,26,28,29,30} have been developed to better understand the TME, revealing spatial niches and heterogeneity in cancer. Some existing methods focus on identifying spatial niches based on transcription patterns alone,^26,28,29,30 while others consider the transcription and the spatial location of cells or spots,^{2,10,12,25,32,35} without considering tissue morphology. For example, StSME includes a normalization method that incorporates spatial locations, morphology images, and gene expressions through imputation to adjust gene expression values between spots.³⁶ First, they calculate a spatial distance matrix between spots. Then, gene expression correlation is calculated, and the top 3 correlated spots are selected for imputation. A morphological similarity across a pair of spots is determined by image features obtained through a convolutional neural network. A final weight is calculated by multiplying the gene expressions and morphological similarities which are further normalized by the total number of selected spots. The normalized gene expressions consider the raw gene expressions of the spot of interest plus the summation of the raw gene expressions of the neighboring spots multiplied by the similarity weights. Louvain clustering is applied to the normalized gene expression levels to identify the spatial niches. It reaches an adjusted rand index (ARI) value of 0.43 in the analysis of the a dorsolateral prefrontal cortex (DLPFC) benchmark dataset.²⁰

SpaGCN deploys a graph convolutional network (GCN) approach where each node in the graph is a spot and each weighted edge between two nodes represents the closeness of two spots based on the location of a spot and similarity of RGB color from the histology image of a spot.³⁴ The SpaGCN uses top principal components from gene expression profiles as input, aggregates inputs with weighted edges in the graph, and identifies spatial niches in the tissue. It reaches an ARI value of 0.52 in the analysis of DLPFC benchmark dataset.²⁰

DeepST first implements spatial data augmentation by calculating similarity weights across a spot of interest and its neighbors with gene expression levels, morphological similarity, and spatial location of spots.²⁷ DeepST further implements a denoising linear autoencoder and a graph autoencoder to identify spatial niches, where the graph was built from the spatial locations of the spots. It reaches an ARI value of 0.59 in the analysis of the DLPFC benchmark dataset.²⁰

STAGATE integrates spatial information and gene expression profiles through a graph attention auto-encoder that learns the low-dimensional latent embeddings to identify spatial niches.³³ The input to STAGATE includes an adjacency matrix computed from the spatial locations of spots, and a pre-clustering of the gene expression conducted by the Louvain algorithm on the principal-component analysis (PCA) embeddings of the gene expression. It reaches an ARI value of 0.6 in the analysis of the DLPFC benchmark dataset.²⁰

SEDR utilizes a deep autoencoder coupled with a masked self-supervised mechanism to build the low-dimensional representations of gene expressions, which are simultaneously embedded with the spatial information through a variational graph autoencoder (VGA), while not integrating morphological information.³⁵ The model is trained to reconstruct the gene expression matrix from the latent representation, obtained by concatenating the embeddings from the VGA and the deep autoencoder. The input to SEDR includes the gene expression matrix and the spatial coordinates of the spots, while the output is the latent representation, further clustered to identify spatial niches in the tissue. It reaches an ARI value of 0.68 in the analysis of the DLPFC benchmark dataset.²⁰ Also Liu et al. proposed a graph deep learning spatial clustering approach to integrate gene expression profiles and spatial positions, followed by a Bayesian Gaussian mixture model to identify spatial niches.^20,32

Studying the cellular and niche composition of the TME has been challenging due to the limitations of the existing methods in processing complex multi-modal datasets. To address this challenge, we propose a TG-ME a transformer and a graph autoencoder to identify microenvironments. Our pipeline identifies tumoral niches in a tissue slide using a transformer model to process single cell gene expression profiles, a convolutional neural network (CNN) to process cell characteristics like size using morphological images, and a graph autoencoder to identify tumoral niches. We tested TG-ME using two benchmark datasets, the DLPFC and 10X Visium benchmark dataset of breast cancer (invasive ductal carcinoma). Then we applied TG-ME to a non-small cell lung cancer (NSCLC) dataset of samples at different stages of cancer.

Results

TG-ME pipeline demonstrates competitive performance in benchmark datasets

In order to validate TG-ME, we processed a single slice of healthy tissue of human dorsolateral prefrontal cortex (DLPFC) and two breast cancer (BC) benchmark datasets.^20,37 In the case of DLPFC we observed that TG-ME identified 7 niches on the healthy tissue, including 5 layers, the white matter (WM) and a region called NA (not a layer or WM) (Figure 1B). We compared TG-ME results with the annotation by a pathologist and found that seven out of eight layers were also identified by TG-ME (Figure 1A). The result demonstrates TG-ME consistency for predicting tissue niches (Figures 1A and 1B). To provide a quantification of TG-ME performance, we calculated the ARI and normalized mutual information (NMI) scores and found that TG-ME has the highest ARI score with 0.54 and NMI score with 0.66, compared to other methods including STAGATE, SEDR, SpaGCN, and Leiden algorithms (Figures 1C–1G). We processed the remaining 11 samples and included their scores in Table S2 (Figure S1). The results, based on all 12 samples, reveal an average ARI score of 0.57 and an NMI score of 0.60. Overall, the results demonstrate that TG-ME has a higher consistency compared to the pathologist’s annotations.

Figure 1.

Quantitative evaluation of multiple methods in the DLPFC dataset

(A) Pathologist annotation.

(B) TG-ME prediction.

(C) STAGATE prediction.

(D) SEDR prediction.

(E) SpaGCN prediction.

(F) Leiden algorithm prediction.

(G) Comparison of ARI and NMI scores (ranging from 0 to 1) across different methods.

To further test TG-ME pipeline, we processed a public 10X Visium benchmark dataset of BC (invasive ductal carcinoma) and compared it to other methods (Figure 2). We calculated the ARI and NMI scores, and we observed again that TG-ME has the highest ARI score with 0.54 and NMI score with 0.44, compared to STAGATE, SEDR, SpaGCN, and Leiden algorithms (Figure 2). To investigate the performance of TG-ME on recognizing tumor niches, we calculated the precision, recall, and F1 scores, for each of the layers predicted by TG-ME. We observe that the higher precision scores were 0.79, and 0.83 for tumor and invasive niches (Table S3), with higher values compared to other methods,^27,35 showing that TG-ME excelled in predicting tumor niches. TG-ME successfully captured the four different niches previously annotated by a pathologist, while the rest of the methods only identified three out of four niches. These findings demonstrate TG-ME’s ability to provide a more detailed analysis of the tumoral niches, compared to previous algorithms.^27,33,34,35

Figure 2.

Quantitative evaluation of multiple methods in the breast cancer dataset

(A) Pathologist annotation.

(B) TG-ME prediction.

(C) STAGATE prediction.

(D) SEDR prediction.

(E) SpaGCN prediction.

(F) Leiden algorithm prediction.

(G) Comparison of ARI and NMI scores (ranging from 0 to 1) across different methods.

We applied TG-ME to a third benchmark dataset, specifically the 10X Visium breast cancer ductal carcinoma in situ (DCIS) dataset from.³⁷ To assess TG-ME’s performance, we once again calculated the ARI and NMI scores, achieving values of 0.54 and 0.44, respectively (Figure 3B). As before, TG-ME outperformed other methods in these metrics (Figure 3). To further evaluate TG-ME’s ability to detect tissue niches, we calculated precision, recall, and F1 scores for the layers predicted by the algorithm. TG-ME demonstrated high precision in identifying the DCIS #1 niche with a score of 0.94, and the stromal niche with a precision of 0.96 (Table S4), outperforming other methods.^27,35 While precision for DCIS #2 was slightly lower at 0.78, TG-ME successfully identified all other niches previously annotated by a pathologist, with the exception of the myoepithelial/stromal/immune niche, which none of the evaluated methods captured. Overall, these results underscore TG-ME’s capability to offer a more detailed and accurate analysis of tumoral niches than existing algorithms.^27,33,34,35

Figure 3.

Quantitative evaluation of multiple methods in the breast cancer dataset

(A) Pathologist annotation.

(B) TG-ME prediction.

(C) STAGATE prediction.

(D) SEDR prediction.

(E) SpaGCN prediction.

(F) Leiden algorithm prediction.

(G) Comparison of ARI and NMI scores (ranging from 0 to 1) across different methods.

Ablation study highlights the importance of TG-ME normalization

One of the key contributions of TG-ME is its normalization process, which integrates multiple data types: gene expression, morphological images, spatial location, and cell composition (cell type). To assess the importance of each component, we conducted an ablation test. This involved systematically removing one data type at a time to observe its effect on ARI and NMI scores. For our analysis, we used the benchmark dataset from.³⁷

First, we implemented the TG-ME pipeline using the full dataset, achieving an ARI score of 0.55 and an NMI score of 0.67 (Figure 4B). Then, we removed the gene expression data and computed the TG-ME pipeline. We observed a significant drop in performance with the ARI score decreasing to 0.21 and the NMI score to 0.36 (Figure 4C). This underscores the crucial role of gene expression in accurately identifying tissue microenvironments, as its removal resulted in poorly defined and low-resolution niches (Figure 4C).

Figure 4.

Ablation scores of the multimodal data for TG-ME normalization

(A) Pathologist annotation.

(B) TG-ME prediction with all the data.

(C) Ablation of the gene expression.

(D) Ablation of the morphology images.

(E) Ablation of the spatial location.

(F) Ablation score of the cell composition.

Next, we evaluated the impact of removing morphological image data. The ARI and NMI scores decreased to 0.47 and 0.63, respectively (Figure 4D). Without this information, the algorithm struggled to differentiate stromal niches, indicating the importance of morphological images for niche identification (Figure 4D).

We then excluded spatial location data from the TG-ME normalization and again observed a decline in performance, with ARI and NMI scores both dropping to 0.47 and 0.63 (Figure 4E). The absence of spatial data led to the misclassification of stromal niches as adipocytes and mixed niches as DCIS #2 (Figure 4E).

Finally, we removed the cell composition data. While the ARI and NMI scores decreased to 0.49 and 0.65, respectively (Figure 4F), the drop was less pronounced compared to the other data types. However, this still resulted in a loss of resolution, as the algorithm failed to identify the left mixed niche and struggled with stromal niche identification.

Overall, these results highlight the importance of TG-ME’s multimodal normalization approach, demonstrating that integrating all available data leads to more precise identification of tissue microenvironments.

Ablation study highlights the critical role of transformer and GraphVAE modules

The TG-ME model consists of two key components: the transformer model and graph variational autoencoder (GraphVAE) (Figures 5 and 6, STAR Methods). The transformer model captures gene-gene interactions within individual cells, while GraphVAE integrates spatial interactions across cells. To assess the contribution of each module, we conducted an ablation study using the benchmark dataset from.³⁷

Figure 5.

Major steps of the TG-ME pipeline

(A) Pre-processing of gene expression.

(B) Division into cells.

(C) Location based adjacency matrix.

(D) Gene expression and cell type composition.

(E) Morphological feature extraction.

(F) Normalized expression.

(G) TG-ME model.

(H) Identification of niches from clustering.

(I) Interpretation and spatial mapping.

(J) TME composition across regions.

(K) Severity signatures across regions.

Figure 6.

TG-ME model overview

(A) The transformer model.

(B) The graph variational autoencoder model.

First, we evaluated the full TG-ME model, incorporating both the transformer and GraphVAE modules (Figure 7B). By comparing the predicted niches with pathologist annotations (Figure 7A), we observed an ARI score of 0.55 and an NMI score of 0.67.

Figure 7.

Ablation scores of the TG-ME model

(A) Pathologist annotation.

(B) TG-ME prediction with both transformer and GraphVAE modules.

(C) Ablation of the GraphVAE module.

(D) Ablation of the transformer module.

(E) Ablation of both GraphVAE and transformer modules, replaced by a fully connected network.

Next, we removed the GraphVAE module, retaining only the transformer module. Upon re-training and evaluation, the ARI and NMI scores dropped to 0.47 and 0.63, respectively (Figure 7C), underscoring the significance of spatial interactions in identifying tissue niches.

We then ablated the transformer module, leaving only GraphVAE. The retrained model showed a decrease in both ARI and NMI scores to 0.49 and 0.63 (Figure 7D), respectively, highlighting the importance of gene-gene interactions for niche identification in spatial transcriptomics data.

Finally, we replaced both modules with a fully connected network. This resulted in further reductions, with ARI and NMI scores declining to 0.46 and 0.61, respectively (Figure 7E). These findings emphasize the complementary roles of the transformer and GraphVAE modules, demonstrating that both are crucial for accurately identifying tissue niches in spatial transcriptomics data.

Application of TG-ME to a high-resolution spatial transcriptomics non-small cell lung cancer dataset

The main target of TG-ME is the identification of tumor niches defined as regions within the TME that contain a distinctive cell-type composition with specific spatial distribution. The first stage of TG-ME pipeline (TG-ME) consists on quality control (QC) (Figure S2).^38,39 At this stage, low-quality cells are filtered out using as criteria the number of features (genes coverage), the number of counts (counts depth), and the percentage of mitochondrial counts per cell (cell activity). Our goal was to ensure that only high-quality cells were included in the analysis while minimizing the risk of introducing artifacts or noise from low-quality cells. We selected the 10% mitochondrial gene expression threshold based on established practices in single-cell RNA sequencing analysis.⁴⁰ High mitochondrial content is often a marker of stressed or dying cells, which can negatively impact the reliability of downstream analyses by introducing biological noise that does not reflect true cellular states. This threshold is widely used in the field to filter out such cells.

Additionally, we adopted a strategy to filter as few cells as possible to maintain the spatial integrity of the transcriptomics data. Given that spatial transcriptomics aims to map gene expression across a tissue, removing too many cells can create gaps or “holes” in the spatial data, which could distort the overall biological interpretation of the TME. Therefore, we carefully balanced the need to exclude poor-quality cells while preserving the structure of the spatial transcriptomic sample.

To evaluate TG-ME in a high resolution ST datasets we processed four samples from the NSCLC dataset¹⁰ (Table 1). Using these data, we filtered and obtained a total of 385,187 cells and 980 genes (Table S5), with 98,002 cells belonging to sample 1, 138,095 cells to sample 2, 71,153 cells to sample 3, and 77,937 to sample 4. Supplement Table S5 shows the results after filtering, and Table S6 the quality of the removed cells.

Table 1.

Patient’s demographic information

Tissue	Sex	Age	Histological diagnosis	Stage	Metastases	TNM	Percentage of tumor content in sample
Sample 1	F	63	Adenocarcinoma	IIIA	2/19 lymph nodes	T2aN2M0	19%
Sample 2	F	65	Adenocarcinoma	IIIA	3/9 lymph nodes	T3N1M0	70%
Sample 3	F	62	Adenocarcinoma	IIIA	0/14 lymph nodes	T4N0M0	26%
Sample 4	M	77	Adenocarcinoma	IIB	3/12 lymph nodes	T3N0M0	34%

TG-ME integration stage highlights consistency in clustering across samples

We evaluate the consistency of clustering across samples after integration (see STAR Methods). For this we visualize the distribution of cells using Uniform Manifold Approximation Projection (UMAP) (Figure 8). We observed that tumor cells colocalize in the same cluster after processing them and exhibit a clear separation from the rest of the clusters (Figure 8) suggesting consistency in the integration stage of TG-ME. To quantify the clustering performance, we calculated the Calinski and Harabasz, and Silhouette scores (Table 2). We obtained a score of 61500.48 that indicates that the clusters are dense and well separated.⁴¹ For Silhouette, we obtained a score of 0.148 on the integrated data. This is important because sample 3 exhibited a negative Silhouette score before integration, suggesting that the method was able to correct and integrate this sample (Table 2). We also observed a correspondence between clustering scores and TNM classification,⁴² suggesting that an increase in heterogeneity is related to a decrease in clustering reflected in the score metrics (Table 1, Figure S3).

Figure 8.

Examination of cell typing of each patient in UMAP and selected gene markers

(A) Integrated UMAP visualization of four samples.

(B) UMAP visualization of the individual samples.

(C) Cell type marker genes.

Table 2.

Clustering performance metrics

Metric	Sample 1	Sample 3	Sample 2	Sample 4	Integrated samples
Silhoutte score	0.294	−0.055	0.245	0.177	0.148
Calinski and Harabasz score	41821.02	5721.4	11824.92	22141.89	61500.48

In order to confirm the consistency of the clusters after integration, we perform differential gene expression analysis to identify gene markers (STAR Methods) that can be used to assign cell types. We observe that the identified markers correspond to the previously claimed cell types (STAR Methods, Figure 8C). To verify that each marker gene belongs to the identified cell type, we underwent a thorough literature review to validate its association. Overall, we observe clear marker genes for each cell type, despite some expected overlap between different T cell and macrophage-related subtypes.

Finally, to assess the effectives of our integration in removing batch and technical effects while conserving biological variance, we computed the scIB batch correction metrics, including the average silhouette width (ASW) and principal component regression.⁴³ An ASW score of 1 indicates perfect batch mixing, whereas a score of 0 reflects strongly separated batches. We obtained a score of 0.91, demonstrating successful batch mixing in our integrated data. This suggests that our integration method effectively mitigated batch effects while maintaining the underlying biological signal.

Integration of spatial information with input data improves clustering performance

The proposed integration approach involves processing one cell at a time, referred to as the “cell of interest” (CoI). The cells in proximity to the CoI are designated as adjacent cells (CAs). In order to identify the CAs, we computed an adjacency matrix (STAR Methods, Figure 5C). On average, approximately 14 neighbors per cell were obtained. Next, a similarity weight value for each gene is calculated between the CoI and its CAs based on gene expression, cell type, and morphology features (STAR Methods). The similarity values are then used to normalize the gene expression of the CoI (STAR Methods). Next the pipeline extracts the morphological images for each of the cells under the assumption that cells of diverse types and functions often display distinct morphological characteristics⁴⁴ (STAR Methods, Figure 5B). These images were generated by staining the cells with a nuclear dye (DAPI) and morphology markers such as membrane (CD298), epithelial cell marker pancytokeratin (PanCK), and T cells marker (CD3).¹⁰ We obtained one image for each cell with a total of 98,002 for sample 1; 138,095 for sample 2; 71,153 for sample 3; and 77,937 from sample 4. Images are uniformly sized at 224 by 224 pixels. To extract meaningful features from these images, we used a pre-trained CNN, specifically ResNet50.⁴⁵ This choice was motivated by the algorithm’s capability to derive representations from images by transforming them into feature vectors (Figure 5E, STAR Methods). The resulting features, initially comprising 2,048 dimensions, were subsequently reduced to 50 dimensions through (PCA).

Next, we visualize the clustering of the four NSCLC samples. We observed a higher cluster separation for all samples after applying TG-ME integration (Figure 9). For example, in sample 1, prior to normalization (Figure 9A) the clusters of T CD4 naive, endothelial, fibroblast, and B cell were initially grouped together at the center of the sample. After TG-ME integration (Figure 9B), these clusters have now distinctively separated, resulting in unique clusters per cell type.

Figure 9.

TG-ME improves the separation of cell types

(A) Sample 1 with gene expression only.

(B) Sample 1 with TG-ME integration.

(C) Sample 4 with gene expression only.

(D) Sample 4 with TG-ME integration.

(E) Sample 3 before TG-ME integration.

(F) Sample 3 after TG-ME integration, where the tumor cells separate into two clusters, and the rest of the cluster becomes well defined.

(G) Sample 2 before TG-ME integration.

(H) Sample 2 after TG-ME integration, where the tumor cells separate into two clusters, and the rest become well-defined.

In the case of sample 4, we observed that the macrophage and epithelial clusters were connected to the fibroblast, Treg, and T memory cells (CD4 and CD8) clusters (Figure 9C). Again, after TG-ME integration these clusters completely separate (Figure 9D).

In the case of sample 3, which is more severe (Table 1), before TG-ME integration, most of the cell types overlap in the center of the UMAP (Figure 9E) and then after integration, there is a clear separation of fibroblast and endothelial clusters (Figure 9F). We also observed that tumor cells separate into two clusters (Figure 9F). One of those clusters is made of tumor cells and is completely separated from the rest of the clusters, while the other, smaller, is mixed with epithelial cells, which may represent epithelial cells becoming dysplastic and transforming into malignant cells as previously described.⁴⁶

For sample 2, before TG-ME integration (Figure 9G), tumor cells overlap with neutrophils and then after, the cluster splits into two different groups. One is entirely separated from the rest of the clusters, while the other is located next to the neutrophil cluster. We also found that a cluster containing fibroblasts and T CD4 memory cells completely separated from the rest of the clusters. This agrees with previous studies that have shown that tumor-associated fibroblasts correlate to T cells in the TME.^47,48

Additionally, we observed appreciable differences in the immune clusters. For low severity samples, the immune clusters are clearly separated, but as severity increases, the immune cells tend to cluster together. For example, in sample 1 (Figure 9B), there are separated clusters of B cells and neutrophils. In sample 4 (Figure 9D), there is a separate cluster of macrophages, and we observe a cluster of Treg and T CD8 memory cells connected to the T CD4 memory cluster, where all of them are well defined. In these two samples, we observe the mast cells cluster separated from the rest of the clusters, which for samples 2 and 3 are overlapping with other cell types including tumor cells. Interestingly, several studies have identified that mast cells play roles in normal and abnormal processes in NSCLC, where they can promote tumor growth in solid tumors.^49,50

To quantify the improvement in clustering after TG-ME integration, we computed the Silhouette⁵¹ and Calinski and Harabasz scores⁴¹ (Table 3) and compare before and after TG-ME integration. Silhouette score ranges from -1 to 1, where -1 means samples have been assigned a wrong cluster, and 1 means perfect clustering.⁵¹ Calinski and Harabasz score is a variance ratio criterion that measures cluster quality by evaluating the separation between clusters and compactness within a cluster, a higher score indicates a better-quality clustering.⁴¹ For all 4 cases, we observe that both the Silhouette and Calinski and Harabasz scores increased after TG-ME processing (Table 3). For example, for sample 1, Silhouette score with gene expression was 0.244, and increased to 0.698 after integration. The Calinski and Harabasz score was 25,389.93 and increased to 211,032.91 after TG-ME integration. Overall TG-ME effectively integrates gene expression, morphology features, cell type, and neighborhood similarity.

Table 3.

Clustering metrics

Sample	Gene expression silhouette score	TG-ME silhouette score	Fold change	Gene expression Calinski and Harabasz score	TG-ME Calinski and Harabasz score	Fold change
1	0.244	0.698	2.86	25389.93	211032.91	8.31
2	0.137	0.293	2.13	3658.87	13560.01	3.70
3	−0.018	0.14	−7.77	5316.66	17807.32	3.35
4	0.133	0.533	4.01	18716.93	118918.82	6.35

The TG-ME transformer model can successfully contextualize gene-gene interactions of individual cells

In order to model the internal expression context within a cell, we implement a transformer model that contextualizes a single cell (Figures 5 and 6). The transformer model in TG-ME is designed to capture the gene-gene interactions within individual cells by focusing on the gene expression profiles. It employs attention mechanisms to calculate the importance or relevance of specific genes within a cell, helping to highlight key gene interactions involved in the microenvironment. For this, we use the cell type provided for each cell as a label. Then we train the transformer model to predict cell types with the intention of determining the gene-gene interactions within a cell. For this, we divided the dataset by cell types and then we extracted 70% of the samples as the training set, 10% as the validation set, and 20% as the testing set. Next, we used the TG-ME integrated gene expression of the 980 genes as input features (STAR Methods). Subsequently, we evaluated the performance by calculating the precision, recall, F1-score, ARI, and NMI scores (Table 4). The precision, recall, and F1-score were greater than 90% for all the samples (Table 4), suggesting that the TG-ME model can accurately determine the gene-gene interactions. We observe that as the severity of the samples increases (Table 1), the precision, recall, and F1-score decrease, suggesting that gene interactions within a cell are dysregulated. To quantify this, we calculated the ARI score, and we observed that for the least severe samples (samples 1 and 4) (Table 1), the ARI score was ∼0.97 (Table 4). Previous studies categorize ARI scores where ARI ≥0.90 as excellent recovery.^52,53,54 For sample 3 the ARI score was 0.92 (Table 4), still considered as excellent classification. While for sample 3, the most severe sample (Table 1), the ARI score was 0.89 (Table 4), considered good recovery, using previously defined criteria.^52,53,54 We again observe a decrease in the ARI score as severity increases.

Table 4.

Performance evaluation of TG-ME transformer model on 4 NSCLC samples

Sample	Precision	Recall	F1-score	ARI	NMI
1	98.4%	98.4%	98.2%	0.978	0.962
2	91.9%	92.2%	90.5%	0.925	0.762
3	92.2%	92.4%	91.6%	0.894	0.830
4	97.1%	97.3%	97.0%	0.971	0.936

Finally, we computed the NMI score, a common metric for evaluating the alignment between predicted cell types and real cell types, which is often used in clustering but applicable for comparing label consistency,⁵⁵ where a 0 value means no mutual information between the predicted and the real cell type labels, and a value of 1 means perfect correlation. The least severe samples have a score greater than 0.90 (Table 4), while for the most severe samples, the score is 0.76. Suggesting again that as severity increases, identifying the gene-gene interactions becomes more complex, since gene expression can be modified by cancer cells. To further study the source of the misclassifications, we computed the confusion matrix for each sample (Tables S7–S10). Overall, we observe that as cancer severity increases, the number of tumor cells also increases, and hence the number of cells belonging to the rest of the cell types decreases. Structural cells including tumor cells, fibroblasts, and endothelial cells; myeloid cells such as neutrophils and macrophages; and plasmablasts presented a low number of mispredictions (Tables S7–S10). While the mispredicted cells were mostly immune cells (Tables S7–S10).

TG-ME can dissect different microenvironments related to tumor severity

To comprehensively analyze TMEs across various stages of NSCLC, we applied the TG-ME pipeline to four samples, each representing a different TNM classification stage. The TG-ME model consists of two key components: a transformer model for detecting gene-gene interactions and a GraphVAE for integrating spatial information (Figures 5 and 6, STAR Methods). The transformer model identifies similarities between genes expressed within individual cells, while the GraphVAE takes these gene similarities further by incorporating spatial neighborhood information through an adjacency matrix. This adjacency matrix is built based on the assumption that cells within the same tumor niche will not only express similar genes but will also be close in space.

The integration of these two modules allows TG-ME to dissect tumor niches with a high level of granularity. For example, TG-ME identified 17 distinct niches in sample 1, 10 niches in sample 4, and 8 niches in both samples 2 and 3 (Figure 10A–10D). The number of detected niches varies according to the cell composition and stage of each tumor. For subsequent analysis, we decided to focus on investigating the identified tumor-specific niches to gain insights into their biological relevance.

Detailed characterization of tumoral niches

In sample 1, TG-ME detected five distinct tumor niches and a tumor-stroma border (Figure 10A). To further understand the differences between these tumoral niches, we performed differential gene expression (DGE) analysis (STAR Methods) followed by enrichment analysis using the Hallmark database, which catalogs gene sets representing specific biological states.^56,57 Each niche exhibited enrichment in distinct biological pathways, underscoring the molecular heterogeneity within the tumor. Detailed p values for each pathway enrichment are provided in Table 4. For instance.

• (1)Tumor-hypoxia niche: Characterized by enrichment in hypoxia-related pathways (Table 5). This niche forms when tumor cells develop faster than the vasculature, leading to oxygen deficit.⁵⁸ Hypoxia is a well-known driver of tumor progression and therapeutic resistance.^58,59,60 The presence of this niche is associated with poor clinical prognosis.
• (2)Tumor-EMT niche: Enriched in pathways related to epithelial-mesenchymal transition (EMT) (Table 5). This niche promotes cancer cell invasion and metastasis.⁶¹
• (3)Tumor-apoptosis niche: This niche is enriched for apoptosis-related genes (Table 5), reflecting programmed cell death mechanisms to control cell proliferation.⁶² Interestingly, apoptosis within tumors can sometimes enhance tumor growth.^63,64
• (4)Tumor-interferon alpha niche: Enriched in pathways related to the interferon alpha response (IFN-alpha) (Table 5), this niche highlights a cytokine pathway that mediates immune responses to tumors.⁶⁵
• (5)Tumor-TNF niche: This niche is enriched in TNF-alpha signaling pathway via the nuclear factor kappa B (NF-kB) pathway (Table 5). The tumor necrosis factor (TNF) is a cytokine implicated in immune homeostasis, inflammation, and host defense, capable of inducing various cellular responses including apoptosis, necrosis, angiogenesis, immune cell activation, differentiation, and cell migration. These processes are important in immune surveillance and tumor progression.⁶⁶

Figure 10.

Tumor microenvironment identified by TG-ME

(A) TME (left) and TME cellular composition (right) for sample 1.

(B) TME (left) and TME cellular composition (right) for sample 4.

(C) TME (left) and TME cellular composition (right) for sample 3.

(D) TME (left) and TME cellular composition (right) for sample 2.

Table 5.

Tumor niches identified by TG-ME

Sample	Tumor niche	Pathway	p value
1	1	Hypoxia	4.97E-07
1	2	Epithelial mesenchymal transition	5.29E-17
1	3	Apoptosis	1.12E-08
1	4	Interferon alpha response	4.02E-07
1	5	TNF-alpha signaling pathways	1.77E-05
4	1	Interferon gamma response	1.00E-02
4	2	Hypoxia	2.64E-06
3	1	Apoptosis	3.80E-03
2	1	E2F targets pathway	1.00E-02
2	2	Unidentified	NA

Additionally, at the tumor-stroma border, we observed enrichment in both the TNF-alpha signaling pathway via the NF-kB pathway (p value 4.5E-5), and the antimicrobial humoral response (GO:0019730), the latter representing an antigen-specific adaptive immune response that directly destroys antigen-expressing target cells.⁶⁷

Sample 4 displayed two tumor niches and a tumor-stroma border (Figure 10B), including.

• (1)Tumor-IFN gamma niche: This niche is enriched in the interferon gamma response (IFN-gamma) pathway (Table 5), suggesting a robust anti-tumor immune response involving inflammation and apoptosis.⁶⁸
• (2)Tumor-hypoxia niche: This niche, is enriched in the hypoxia pathway, also identified in sample 1 (Table 5), suggesting similarities between these two niches.

Sample 3, a more severe case, revealed only one tumor niche (Figure 10C): Tumor-apoptosis (p value 3.8E-3), also identified for one of the niches in sample 1 (Table 5). We also identified a tumor-stroma border that displayed enrichment in the coagulation pathway and the MAPK cascade pathway (GO:0045657), both pivotal in promoting tumor growth, invasiveness, and metastasis.^69,70,71

Sample 2, the most severe sample, contained two tumor niches (Figure 10D).

• (1)Tumor-E2F: This niche is enriched in the E2F targets pathway, which is associated with cell cycle regulation, proliferation, differentiation, and stress responses (Table 5).⁷²
• (2)Tumor-unidentified: The second niche, while not presenting any pathway enrichment, contained both tumor and neutrophil cells (Figure 6D), indicating potential interactions between the tumor and the immune system. We called this niche tumor-neutrophil.

Additionally, the tumor-stroma border in sample 2, demonstrated pathways similar to those in sample 1, including the TNF-alpha signaling via NF-kB pathway (p value 1.8E-), and the antimicrobial humoral response (GO:0019730) (p value 1.6E-4).

Across all four samples, we observed that as the severity of the tumor increases, the percentage of certain niches, such as tumor-apoptosis and tumor-E2F also increases. For example, the tumor-apoptosis niche increased from 2.03% in sample 1 (least severe) to 8.18% in sample 2 (second most severe) (Table 6). Additionally, the Tumor-E2F niche dominated in sample 3, accounting for 34.65% of the total tumor composition, suggesting a highly proliferative and aggressive tumor state (Table 6).

Table 6.

TG-ME identified tumoral niches

Sample 1	Niche	Percentage of cells from niche
Sample 1	Tumor-hypoxia	6.70%
Sample 1	Tumor-EMT	2.85%
Sample 1	Tumor-Apoptosis	2.03%
Sample 1	Tumor-IFN alpha	0.93%
Sample 1	Tumor-TNF	0.12%
Sample 4	Tumor-hypoxia	6.50%
Sample 4	Tumor-IFN Gamma	10.60%
Sample 2	Tumor-Apoptosis	8.18%
Sample 3	Tumor-E2F	34.65%
Sample 3	Tumor-neutrophil	21.14%

In conclusion, TG-ME provides a detailed dissection of tumoral niches, revealing diverse molecular signatures that reflect both severity and heterogeneity of NSCLC. The attention scores obtained from the transformer model are crucial for understanding how specific genes and pathways contribute to the TME dynamics and disease progression. By identifying these key interactions, the model can disclose biological mechanisms underlying cancer, such as hypoxia-related gene activity, immune response modulation, or changes associated with EMT. These findings highlight the value of niche-specific insights for understanding cancer progression and offer potential pathways for therapeutic intervention.

Biological implications of tumoral niches in NSCLC

Tumor-hypoxia niche

TG-ME identified two niches enriched in the hypoxia pathway, one in sample 1 and the other in sample 4. Hypoxia occurs when there is a rapid grow of tumor that outpaces its vascular supply, leading to an oxygen deficit. This activates hypoxia-inducible factors (HIFs), that promote angiogenesis, metabolic reprogramming, and immune suppression.^58,59 In NSCLC, hypoxia is known to increase tumor aggressiveness, metastasis, resistance to therapy, and tumor recurrence.^59,73

Additionally, non-immune cells, such as endothelial cells, play a crucial role in responding to hypoxia by promoting the formation of new blood vessels (angiogenesis) to improve oxygen supply to the tumor.⁷⁴ This process that is mediated by vascular endothelial growth factor (VEGF), not only supports tumor growth but also facilitates tumor metastasis.⁷⁵ In other hand, cancer-associated fibroblasts (CAFs) can also be activated under hypoxic conditions and contribute to extracellular matrix remodeling, further enhancing tumor invasiveness.⁷⁶

In our findings, the presence of the tumor-hypoxia niches across multiple NSCLC samples suggests that hypoxia-related pathways may be involved in tumor progression. This highlights potential therapeutic opportunities, for example targeting HIF pathways or VEGF signaling to interrupt the pro-tumorigenic effects of hypoxia in NSCLC patients.

Tumor-EMT niche

EMT, is a key process in cancer metastasis.⁷⁷ During EMT, epithelial cells lose their adhesion properties and apical-basal polarity gaining mesenchymal characteristics that allows them to migrate and invade tissues.⁷⁸ Factors such as growth factor stimulation and adhesion to type I collagen (Col-I) induce EMT in cancer cells which are often activated in response to external stimuli in the TME.⁷⁸ In NSCLC, EMT has been linked to poor prognosis and resistance to therapy, particularly in advanced stages.⁷⁹

Tumor-apoptosis niche

The tumor-apoptosis niche is characterized by the enrichment of genes related to programmed cell death or apoptosis. While apoptosis pathway is typically a tumor-suppressive mechanism that eliminates tumoral cells, certain conditions within the TME may also allow cancer cells to evade apoptosis, promoting survival and growth.⁸⁰ Additionally, apoptosis can enhance tumor progression by releasing inflammatory signals that remodel the TME to foster tumor development.⁶⁴

Specifically, in NSCLC, apoptosis can promote cancer by enhancing proliferation. Apoptosis provides support of proliferating cells by removing tumor cells that respond to treatment leading to tumor resistance. In NSCLC therapy, it is crucial to consider other regulatory mechanisms apart from apoptosis.⁸¹

Tumor-IFN alpha and gamma niches

The tumor-IFN alpha and gamma niches are enriched in pathways related to interferon signaling, which plays a crucial role in anti-tumor immunity.⁸² IFN-alpha and gamma have the ability to induce the expression of genes that promote immune responses against tumor cells, inhibiting tumor proliferation and enhancing the presentation of tumor antigens to the immune system.⁸³

In NSCLC, interferon signaling is essential for immune response within the TME, particularly in the interaction between tumor cells and immune cells. The presence of these niches in our analysis reflects the ongoing battle between the tumor cells and the immune system, with interferon pathways playing a key role in this interaction.

Discussion

In this work, we have designed, implemented and applied TG-ME, a method for automatic tumor niche classification. The proposed approach resolves gene expression at subcellular resolution and uses a network to infer gene expression changes as a result of cell interactions and then categorized each cell with a specific niche. TG-ME achieves significantly higher ARI and NMI scores compared to previous methods.

We validated the strengths of this approach on human dorsolateral prefrontal cortex and a layer of breast cancer benchmark datasets. The recent advancements in high-resolution spatial transcriptomics technologies have significantly increased the demand for sophisticated computational frameworks like TG-ME to analyze and extract meaningful insights from these complex datasets. TG-ME approach consists in the normalization of gene expression for each cell using two main assumptions. First, TG-ME considers that the internal gene-gene interactions within a cell are reflected in its gene expression profile. Second, TG-ME considers that the cell-to-cell interactions within a tumor neighborhood are indeed reflected in the gene expression profile of that spatial neighborhood. These are crucial aspects that TG-ME framework leverages to gain insights into the complex cellular dynamics within the tumor niches.

One of the main advantages of normalizing the gene expression is the ability to comprehensively profile individual cells. The combination of the gene expression data provides insights into the molecular activities of the cells, mainly into the genes that are actively transcribed and potentially contributing to tumor progression.^84,85 Simultaneously, the inclusion of morphology features offers a structural perspective, elucidating the shape, size, and, in some instances, the type of cells.^10,86 This comprehensive profiling provides a more detailed and context-rich representation of the cellular behavior within the TME. Cell type information, in the other hand, incorporated into our multimodal data representation, helps to understand the distinct contributions of different populations to the TME.⁸⁷ It allows to identify cell type specific gene expression changes and their potential implications in the TME.

Also the implemented model inspired by Zhang T.-H. et al.⁸⁸ has the characteristic of one-to-one representations. This means it exhibits an input-output consistency, aiming to preserve the biological context. Therefore, for every gene in a cell, the model will generate a representation that considers its relationship with the remaining genes. These representations are called attention values, and each attention layer will generate an attention value for each gene, as a function of the output of the previous attention layer. Having not only the gene expression information but also integrating the information extracted from the morphology images to determine how similar cells are in morphology terms, improves the robustness of discerning the similarity between two cells, and identifying cells transformed by cancer. Future studies can add over TG-ME model due to its modularity.

Limitations of the study

We observed that the precision, recall, and F1-score decrease as the severity of the sample increases which highlights the challenges in distinguishing between cell types as cancer progresses.

Resource availability

Lead contact

Further information and requests for resources and reagents should be directed to the lead contact, Mario Flores (mario.flores@utsa.edu).

Materials availability

This study did not generate new unique reagents.

Data and code availability

• •This paper analyzes existing, publicly available data, accessible at https://doi.org/10.1038/s41593-020-00787-0, https://support.10xgenomics.com/spatial-gene-expression/datasets/1.0.0/V1_Breast_Cancer_Block_A_Section_1, Gene Expression Omnibus (GEO) through accession number GSE243275, and https://doi.org/10.1038/s41587-022-01483-z, and are publicly available as of the date of publication.
• •All original code has been deposited at GitHub: https://github.com/Karladanielap/TG-ME/ and is publicly available as of the date of publication.
• •Any additional information required to reanalyze the data reported in this paper is available from the lead contact upon request

Acknowledgments

This article’s publication costs were supported partially by the National Institutes of Health (3R01CA124332-13A1S1 to M.A.F.), the National Science Foundation (2051113 to Y.J.), the National Institutes of Health (P30 CA054174 to Y.C.), Cancer Prevention and Research Institute of Texas Core (RP220662 to Y.C.), National Institutes of Health (R01 CA284554 to S.G.).

Author contributions

K.P. and M.F. was responsible for data acquisition, analysis, interpretation, and drafted and critically revised the manuscript. Y.-F.J. and M.F. contributed to the conceptualization, data acquisition, analysis, and interpretation, and also drafted and critically revised the manuscript. Y.C., S.G., and Y.H. provided resources, contributed to writing, and approved the final version of the manuscript. All authors have read and approved the final manuscript.

Declaration of interests

The authors declare no competing interests.

STAR★Methods

Key resources table

REAGENT or RESOURCE	SOURCE	IDENTIFIER
Deposited data
Visium 10X DLPFC Dataset	Maynard et al.20	https://www.nature.com/articles/s41593-020-00787-0
Visium 10X Human Breast Cancer	10X Genomics	https://support.10xgenomics.com/spatial-gene-expression/datasets/1.0.0/V1_Breast_Cancer_Block_A_Section_1
Visiun 10X Human Breast Cancer	Janesick et al.37	GSE243275
Nanostring CosMX Non-Small Cell Lung Cancer	Shanshan et al.10	https://nanostring.com/products/cosmx-spatial-molecular-imager/ffpe-dataset/nsclc-ffpe-dataset/
Software and algorithms
R package Seurat v4.1.0	Hafemeister, & Satija89	https://genomebiology.biomedcentral.com/articles/10.1186/s13059-019-1874-1
R package enrichR 3.2	Kuleshov et al.56	https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4987924/
Python package Scanpy 1.10.1	Wolf, Angerer, & Theis90	https://genomebiology.biomedcentral.com/articles/10.1186/s13059-017-1382-0
Python package Pytorch 1.13.1+cu116	Paszke et al.91	https://proceedings.neurips.cc/paper/2019/hash/bdbca288fee7f92f2bfa9f7012727740-Abstract.h
Python package Pandas 2.2.2	The pandas development team	https://zenodo.org/records/10957263
Python package Numpy 1.26.0	Harris et al.92	https://doi.org/10.1038/s41586-020-2649-2
Python package anndata 0.10.7	Virshup et al.93	https://www.biorxiv.org/content/10.1101/2021.12.16.473007v1
TG-ME code	This paper	https://github.com/Karladanielap/TG-ME/

Method details

Please provide precise details of all the procedures in the paper (behavioral task, generation of reagents, biological assays, modeling, etc.) such that it is clear how, when, where, and why procedures were performed. We encourage authors to provide information related to the experimental design as suggested by NIH and ARRIVE guidelines (e.g., information about replicates, randomization, blinding, sample size estimation, and the criteria for inclusion and exclusion of any data or subjects).

Pre-processing of gene expressions of the NSCLC dataset

The raw gene expression matrix was processed using Seurat v4,⁹⁴ where cells with a mitochondrial percentage higher than 10% were removed, keeping the cells with the highest quality.⁴⁰ All 980 genes included in the dataset were used since the number of features and the total counts illustrated good quality control of the data as shown in Figure S2. More information about the quality control process can be found at Methods S1. Details of the removed cells are shown at Table S6.

Normalization, scaling, and selection of highly variable genes were performed with SCTransform, a regularized negative binomial regression-based model available in Seurat.⁸⁹ Principal component analysis (PCA) was conducted on the normalized data for dimensionality reduction, resulting in the top 30 components. The data was visualized with the uniform manifold approximation projection (UMAP) computed over the first 30 PCAs, with neighboring points of 5 and a minimum distance of 0.001. The nearest neighbors were identified over the 30 principal components, and clusters were computed using the Leiden algorithm with a resolution of 0.8.

Integration of different samples

The gene expression matrices of the 4 samples were integrated by Seurat v4 using the SCTransformed counts (Figure 4A). PCA was conducted on integrated data, where the top 30 Principal Components (PC) were selected for UMAP visualization.

Image stitching

The Morphology images were further stitched into one image in Python v3.9.7, with the package stitch2d v1.1,⁹⁵ through the function StructuredMosaic which arranges the images into a grid. The number of images to be stitched together in each line is determined by the size of the samples. The origin was always set to the lower left corner, the stitching direction was defined to be horizontal, and the selected stitching pattern was raster.

Extraction of cell morphology image

The dataset also provides the center (X-Y coordinates), the width, and the height of each cell. The width or height of the largest cell is used as the size template for image cropping, so each cell has a squared image cropped while the center of the cell is the center of the image. For a smaller cell, its image can include more than one cell, where the cell of interest is always located at the center of the image, providing information not only of the cell of interest but also of its context.

Extraction of morphology features

The morphology images of each cell are augmented, including normalization, rotation, sharpness, and adjusting. The adjusted morphology images were processed by a pre-trained CNN, ResNet50 to extract the features.⁴⁵ The extracted features in a cell were represented as a 2,048-dimension vector. PCA analysis is performed over the vector of each cell to further reduce the dimensions, and the top 50 PCAs are extracted as the morphological features ($M$) of each cell.

Processing of cell location-adjacency matrix

A ball tree graph is built to calculate the distances between spots by constructing a cell-cell spatial relationship graph, which is used to construct the adjacency matrix (γ). The cell-cell adjacency matrix is built over the top 12 nearest neighbors, where, when two cells n and j are neighbors, it is represented by a γ_nj = γ_jn = 1, and γ_nj = γ_jn = 0 otherwise. The adjacency matrix contains self-loops. The resulting matrix is a squared matrix of size N by N, where N represents the number of cells. N varies across samples (Table 2).

The ball tree graph is a hierarchical data structure used for organizing points in a multi-dimensional space. The graph-structured representation contains the information of the normalized adjacency matrix, the adjacency labels, and the normalization term.

Selection of similar cells in the neighborhood using morphological and location information

Cellular morphological features ($M$), cell types ($P$), and neighborhood ($W$) of a cell are used to screen similar cells in a region based on cosine similarity. The number of cell types in the dataset is used for one-hot encoding of the cell type feature ($P$). The adjacency matrix previously computed illustrates if two cells are neighbors ($W=1$) or not ($W=0$). Then, the similarity weights (K) between a cell n and its neighboring cell j are calculated according to the vectors $V_n$ and $V_j$ as Equation 1

$$K=\cos \theta =\frac{\overrightarrow{V_n}·\overrightarrow{V_j}}{\Vert \overrightarrow{V_n}\Vert \Vert \overrightarrow{V_j}\Vert }$$ (Equation 1)

Where,

$$V_n=\left[\begin{array}{c}{M}_n\\ W_{nj}\\ P_n\end{array}\right],V_j=\left[\begin{array}{c}{M}_j\\ W_{jn}\\ P_j\end{array}\right]$$ (Equation 2)

TG-ME normalization

If cells are similar in morphology, type, and locations, cosine similarity of expression levels of 980 genes between cell n, Gn, and Gj from its neighboring cell j, $G_{nj}$, is calculated.

TG-ME pipeline normalizes the gene expression levels in a cell if the cell share similar expression, morphology, location, and type with its neighbors, suggesting a logical and $(\bigwedge )$ operation between the similarity weights $G_{nj}$ and $K$, obtaining a final similarity weight $S$ (Equation 4).

$$S=\left[G_{nj}\bigwedge K\right]$$ (Equation 3)

The gene expression of a cell of interest (n) will be normalized accounting for the similarity weights (S) as follows:

$${\tilde{G}}_n=\frac{G_n+\sum _{j=1}^m{G}_{j}S}{m+1}$$ (Equation 4)

Where ${\tilde{G}}_n$ is the normalized gene expression of cell n, S is the similarity weight obtained before, and m is the total number of neighbors.

TG-ME normalization characterizes the similarity of cells deploying morphology, cell type, gene expression, and spatial location information.

TG-ME model for TME dissection

The normalized gene expression matrix ${\tilde{G}}_{ij},i\in \left[1,C\right]\text{,}$ $j\in \left[1,N\right],C$ representing the total number of cells, and N is the number of genes in the dataset, serves as the input to the TG-ME model including a Transformer model, followed by a Graph Variational Autoencoder (GraphVAE), as illustrated in Figure 8.

Transformer

The transformer model consists of three attention layers, each featuring three attention heads (Figure 8A). Each attention layer generates an attention vector $\left(A_{i_b}\right)$, where b represents each layer, encapsulating the TG-ME normalized gene expression vector $\left({\tilde{G}}_i\right)$.

The self-attention module, the cornerstone of the transformer architecture, generates attention values for each element in the input sequence. It conducts two primary operations: computing the query $Q_g$, key $K_g$ and value $V_g$ vectors for each element, where the corresponding weights ($W_{Q_g},W_{K_g,}{W}_{v_g}$) are learned during training. These weights are initially set to random values before training commences and are then updated iteratively during the training process. The optimizer then adjusts these weights to minimize the loss. This process repeats for each epoch until the model achieves satisfactory performance, with the weights being fine-tuned to capture intricate patterns and relationships within the data.

For gene $j$ in cell $i$, the query of this gene in the first attention layer is computed by multiplying the query weight of the gene ($W_{Q_{g_j}}$) by its gene expression $\left({\tilde{G}}_{i_j}\right)$ in cell $i$, as

$$Q_{g_j}=W_{Q_{g_j}}{\tilde{G}}_{i_j}\text{,}$$ (Equation 5)

$$K_{g_j}=W_{K_{g_j}}{\tilde{G}}_{i_j}\text{,}$$ (Equation 6)

$$V_{g_j}=W_{V_{g_j}}{\tilde{G}}_{i_j}$$ (Equation 7)

Each gene in a cell will have its own query, key, and value.

The second set of operations includes the calculation of the attention weight vector (${\alpha }_{j_l}$) as a probability distribution representing the similarity between $Q_{g_j}$ and $K_{g_l},j\ne l$, to obtain the relationship between gene j and the rest of genes:

$${\alpha }_{j_l}=softmax\left(Q_{g_j}{K}_{g_l}\right),j\ne l\text{,}$$ (Equation 8)

Where ${\alpha }_{j_l}$ represents the attention weight between the gene j and gene l. Softmax calculates the probability distribution of the similarity between $Q_{g_j}$ and $K_{g_l}$.

Subsequently, a representation of the Query gene ($z_{j_l}$) is obtained for each self-attention module by multiplying the transpose of ${\alpha }_{j_l}$ by the vector of values, also excluding the gene of interest $V_{g_l},j\ne l$.

$$z_{j_l}={{\alpha }_{j_l}}^T{V}_{g_l}$$ (Equation 9)

For each cell $i$, the output of the attention layer is a vector $Z_i$ of length 980, containing all the representations for the query genes $z_{j_l}$.

The proposed model contains 3 attention heads $H$. The representations $Z_i$ from each head are linearly combined to obtain one final set of representations $A_{i_b}$, where $b$ represents each layer.

$$A_{i_b}=x+layernorm\left(W_H^T{E}_i\right)$$ (Equation 10)

The summation represents the residual connection, $W_H$ the weights of head H, and $E_i$ an $h$ by $N$ matrix, where $h$ is the number of heads, and $N$ is the number of genes that containsthe $Z_i$ representations for each head. This computation will be repeated for each of the layers, wherein the first layer $x$ is the TG-ME normalized gene expression for cell $i$, ${\tilde{G}}_i$, while in the 2nd and 3rd layers $x$ is the output of the previous layer $A_{i_b}$. Layernorm is the layer normalization function,⁹⁶ which calculates normalization statistics directly from the combined inputs to the neurons within a hidden layer.

Finally, a classification layer is included to classify between different cell types using the Softmax function. The model is further trained with a dropout rate of 0.7 and no activation function, and the final set of representations $A_3$ for each cell generated by the last attention layer is extracted because it represents the relationship among genes which are further used as an input for the GraphVAE, as shown in Figures 7 and 8.

Graph Variational Autoencoder (GraphVAE)

GraphVAE integrates Graph Convolutional Networks (GCNs) with a Variational Autoencoder (VAE). GraphVAE aims to acquire latent features embedded in location information denoted by graph γ and the attention output from the transformer $A_3$.

The encoder contains 3 GCN layers, as presented in Figure 8B. The first GCN layer convolutes $A_3$ and $\gamma$, with a rectified linear unit (ReLU) activation function,

$$\overline{A}=GCN\left(A_3,\gamma \right)=ReLU\left(\tilde{\gamma }\ast A_3\ast W_1\right)$$ (Equation 11)

$$\tilde{\gamma }=D^{-\frac{1}{2}}\gamma D^{-\frac{1}{2}}\text{,}$$ (Equation 12)

where $\tilde{\gamma }$ is the symmetrically normalized adjacency matrix, and D is the degree matrix.

The second layer computes the mean μ of the latent representation, and the third layer computes the log variance log σ2,

$$\mu ={GCN}_{\mu }\left(A_3,\gamma \right)=\tilde{\gamma }\ast \overline{A}\ast W_2$$ (Equation 13)

$$\log {\sigma }^2={{GCN}_{\sigma }\left(A_3,\gamma \right)=W}_3\ast \tilde{\gamma }\ast \mu$$ (Equation 14)

where $W_1\in R^{980x64}$, $W_2\in R^{64x32}$, and $W_3\in R^{64x32}$ are the weights for each layer updated during the training process with Adam optimizer. The output of the encoder consists of the reparameterizations, where noise $\epsilon$ is introduced, and the final latent representation $\mathrm{\beta }$ is computed as

$$\mathrm{\beta }=\mu +\log {\sigma }^2\ast \epsilon$$ (Equation 15)

with $\in \left(0,1\right)$ .

The GraphVAE model also has a decoder that reconstructs the input $A_3$, denoted as ξ. It is made of a fully connected layer $d$ with the ReLU activation function, followed by a second fully connected layer. The convolutional hidden layers have a size of.^16,32

$$d=ReLU\left(w_1·\beta +b_1\right)$$ (Equation 16)

$$\xi =w_2·d+b_2$$ (Equation 17)

where $b_1,b_2$ are the bias vectors and $w_1\in R^{996x128},w_2$ $\in R^{128x980}$ are the weights updated during training.

The model is designed to consider two different angles, the first one is the transformer model, which learns the relationships across genes $A_3$, the second one is the GraphVAE, where $\mathrm{\beta }$ contains location-based signatures integrating gene expression and morphological information. $A_3$ and $\mathrm{\beta }$ are concatenated and used for downstream analysis.

Hyperparameters of the TG-ME model

The model was trained for 100 epochs (with early stopping) with the batch size of 128. Back propagation and the Adam optimizer with a learning rate of 0.01 and a weight decay of 0.01 were applied. A validation set was used during training. The transformer model was trained with a dropout rate of 0.7, while GraphVAE was trained with a dropout rate of 0.1.

The negative log likelihood loss was calculated for the class prediction using the Transformer model, and a mean squared error loss between the decoded features $\mathrm{\xi }$ from the GraphVAE and the input to the GraphVAE ($A_3$, γ) was calculated. Both losses were weightily added (0.8 for the Transformer and 0.2 for the GCN autoencoder) to calculate a final loss.

To address potential overfitting, we incorporated several strategies in TG-ME’s architecture, including early stopping, dropout layers (0.7 for the Transformer model and 0.1 for the GraphVAE), and regularization techniques like weight decay. These mechanisms help to prevent overfitting by reducing the model’s complexity and improving generalization. Additionally, we used cross-validation, and a separate validation set to fine-tune hyperparameters and evaluate performances across diverse samples.

Spatial Transcriptomics data can have high dimensionality; to further avoid overfitting, we reduced feature space using dimensionality reduction techniques (PCA) before feeding the data into the model, which helps to mitigate the curse of dimensionality and limits the risk of overfitting.

The model takes 134.13 s to train for 100 epochs on an NVIDIA GeForce RTX 3070 Ti GPU, which has 8 GB of total memory. The GPU utilizes NVIDIA driver version 560.94 and supports CUDA version 12.6. In contrast, the model trains in 30.09 s for 100 epochs in a high-performance computing (HPC) environment using a Tesla V100S-PCIE-32GB GPU, equipped with 32 GB of total memory. This GPU is running NVIDIA driver version 545.23.08 and supports CUDA version 12.3.

Tumor microenvironment identification from clustering

The concatenated location-based signatures, $\mathrm{\beta }\text{,}$ and $A_3$ are further clustered with Louvain algorithm in Scanpy python package⁹⁰ to identify cells with similar representations. If both, $\mathrm{\beta }$ and $A_3$ are similar, cells will be assigned the same cluster. At the end, each cell is assigned one cluster, which is considered a niche of the TME.

Each of the identified clusters are assigned a name according to its cell type composition and spatial location for interpretation. The niches are mapped back to single-cell resolution by assigning the corresponding niche to each of the cells.

Evaluation

Two benchmark public datasets, a LIBD human dorsolateral prefrontal cortex²⁰ and a public 10X Visium benchmark dataset of Breast Cancer (Invasive Ductal Carcinoma) were processed by the proposed TG-ME pipeline. The adjusted rand index (ARI) and normalized mutual information (NMI) scores were calculated to evaluate the performance of the proposed pipeline in identifying microenvironment niches. The results were compared with the niches identified by the pathologist (annotation), STlearn, and Leiden clustering.

Severity signatures across tumor regions NSCLC

With the validated TG-ME model, the NSCLC dataset was further analyzed to examine the severity signatures across tumor regions.

The TG-ME normalized gene expression for cells in the identified tumor and Tumor-Stroma Border niches across samples are integrated by batch normalization using Seurat V4.⁸⁹ Differential Gene Expression (DGE) analysis is further computed using the MAST method, with a log fold change threshold of 0.25 and a minimum percentage of expression in 0.1 cells (10%). Enrichment analysis was performed for the identify differentially expressed genes with enrichr package⁵⁶ in R and the Hallmark⁵⁷ database to get an insight into the severity pathways.

Quantification and statistical analysis

Pathway enrichment analysis was conducted using R package enrichR 3.2 and significance was assessed based on adjusted p-values derived from the Fisher’s exact test. The statistical details are provided in Table 5 and results section TG-ME can dissect different microenvironments related to tumor severity. Statistical significance was defined as p-values < 0.05.

Published: March 13, 2025