A composite scaling network of EfficientNet for improving spatial domain identification performance

Abstract

Spatial Transcriptomics leverages gene expression profiling while preserving spatial location and histological images. However, processing the vast and noisy image data in spatial transcriptomics (ST) for precise recognition of spatial domains remains a challenge. In this study, we propose a method of EfNST for recognizing spatial domains, which employs an efficient composite scaling network of EfficientNet to learn multi-scale image features. Compared with other relevant algorithms on six data sets from three sequencing platforms, EfNST exhibits higher accuracy in discerning fine tissue structures, highlighting its strong scalability to data and operational efficiency. Under limited computing resources, the testing results on multiple data sets show that the EfNST algorithm runs faster while maintaining accuracy. The ablation studies of EfNST model demonstrate the significant effectiveness of the EfficientNet. Within the annotated data sets, EfNST showcases the ability to finely identify subregions within tissue structure and discover corresponding marker genes. In the unannotated data sets, EfNST successfully identifies minute regions within complex tissues and elucidated their spatial expression patterns in biological processes. In summary, EfNST presents a novel approach to inferring cellular spatial organization from discrete data spots with significant implications for the exploration of tissue structure and function.

EfNST improves the identification of spatial domains using EfficientNet for multi-scale image features. It effectively reveals tissue structures and marker genes, aiding in the understanding of spatial gene expression patterns.

Introduction

The single-cell transcriptomic¹ has refined the research accuracy from the organizational level to the single cell, allowing researchers to analyze processes such as embryonic development, cell reprogramming, and disease occurrence with unprecedented resolution^2–5. However, cell subpopulations act in an intertwined manner based on their specific spatial positions within a given tissue⁶. The spatial heterogeneity is a crucial characteristic for understanding organ function, cell fate regulation mechanism, and cell lineage generation^7–9. The single-cell transcriptomics technologies inevitably lose spatial information in the process of dissociating solid tissues into individual cells. Therefore, in order to better understand the spatial heterogeneity of different cells, it is necessary to simultaneously understand their transcriptional heterogeneity and spatial location information.

In recent years, the ST technologies such as 10x Visium¹⁰, Slide-seq¹¹, and Stereo-seq¹² has become a powerful tool in studying molecular biological mechanisms, which can obtain the spatial information along with transcriptomic expression profiling information of cells to analyze and characterize the expression profiles of specific cell types on a spatial scale¹³. However, the low sensitivity, high dimensionality, multiple sparsity, high noise and multimodality of spatial transcriptomics data make spatial domain recognition limiting. These challenges have spurred the development of algorithms designed for accurate and robust spatial domain recognition. Currently, spatial transcriptomic algorithms are primarily categorized into probability-based models and deep learning models. Probability-based algorithms (e.g., BayesSpace¹⁴, Giotto¹⁵, SC-MEB¹⁶, Scanpy¹⁷, Seurat¹⁸) often have longer runtime, lower flexibility in handling large-scale ST data, struggle with complex nonlinear relationships, and lack autonomous feature learning capabilities. On the other hand, deep learning models (e.g., STAGATE¹⁹, SEDR²⁰, CCST²¹) are suitable for large-scale, high-dimensional data, allowing nested model usage for automatic and fast learning of complex features, thus alleviating analysis complexity and significantly advancing spatial domain recognition technology. Despite their outstanding feature learning, these models have complex training designs, slower training speeds, and features learned lack clustering orientation.

Spatial transcriptomics (ST) data encompasses image information obtained through techniques like organization, slicing, staining, and other imaging modalities. To derive meaningful insights from these data, it is essential to employ image segmentation and feature extraction techniques. The subsequent analysis of the image data involves integration with transcriptomic data to elucidate the spatial distribution and functions of cells within tissues. Due to the unique morphological features provided by image information, intuitively depicting the spatial distribution of cells, incorporating ST histological image information to aid spatial domain recognition is of crucial importance. However, due to the significant noise and large size of ST data, most current algorithms (e.g., stLearn²², SpaGCN²³, TIST²⁴) still face challenges in efficiently integrating image information, exhibiting high model complexity, slow processing speeds, and inadequate image feature extraction. Thus, efficiently handling the vast image data in ST using an effective network to obtain image features and increase computational speed remains a challenge.

To address the above issues, we present EfNST, a model rooted in the EfficientNet²⁵ convolutional neural network architecture. EfNST is specifically designed to efficiently integrate image information and enhance spatial domain recognition performance. For image feature extraction, it utilizes the robust architecture of EfficientNet, a convolutional neural network framework developed by Google. This framework is adept at learning multiscale image features, thereby boosting computational efficiency. In addition, EfNST incorporates a Variational Graph Autoencoder (VGAE)²⁶ to capture latent representations of nodes in graph-structured data. Furthermore, a Denoising Autoencoder (DAE)²⁷ is implemented to mitigate complex noise and interference within the data, facilitating the extraction of more resilient latent features. To validate the efficacy of the algorithm, we conducted tests on six data sets from three sequencing platforms. The results indicated that EfNST outperforms seven classic benchmark methods in terms of spatial domain recognition. Specifically, EfNST demonstrated high accuracy in identifying subregions of tumors within the human breast cancer data set, providing a more detailed delineation of tumor areas. Moreover, in the case of four other data sets, EfNST exhibited superior proficiency in recognizing intricate spatial patterns within tissues. In conclusion, EfNST provides a novel approach to spatial domain identification, facilitating more accurate analysis of tissue structures and advancing the exploration of spatial gene expression patterns.

Results

The EfNST framework

EfNST accurately identifies spatial domains by integrating gene expression profiling, spatial location, and potential characterization of histological image information to elucidate heterogeneity in tissue structure (Fig. 1a–d). First, for the ST data of 10x Visium with histological images, EfNST preprocesses gene expression profiling, spatial location, and hematoxylin and eosin (H&E) image information in the ST data. Then, to characterize the neighborhood information, EfNST utilizes the K-Nearest Neighbors similarity measurement method to construct a normalized adjacency matrix from spatial location information and capture the neighboring spots for each spot. For histological image data, we first segment the original image (H&E staining tiles) based on the coordinates of each spot to obtain 50 × 50 pixel sub-images. The torchvision.transforms function from the PyTorch library was then used to transform and enhance these sub-images into high-quality 224 × 224 pixel blocks. Then, the EfficientNet-B0 model from EfficientNet network is employed for feature extraction on the image blocks. The extracted features are dimensionally reduced to 50 using Principal Component Analysis (PCA), resulting in an underlying feature representation of the image blocks. These representations form the image feature matrix, where each row corresponds to a feature vector of an image block, offering a comprehensive description and representation of the entire image data set. Compared to conventional CNN models (e.g., Resnet50²⁸, Resnet152²⁸, VGG19²⁹, DenseNet121³⁰, and Inception_v3³¹), EfficientNet employs network structure search and a multiscale compound method, reducing parameters and computational complexity while maintaining accuracy. Furthermore, it showcases enhanced generalization capabilities, increased accuracy, and faster convergence speed when handling input data of varying sizes. Subsequently, data augmentation is applied to the gene expression matrix, adjacency matrix and image feature matrix to obtain an augmented gene expression matrix.

Fig. 1. The workflow of the EfNST.

a The input ST data are Gene Expression, Histological Images and Spatial Location. b EfNST processed the H&E Images and Spatial Locations to obtain Image Patches, which were processed using a pre-trained EfficientNet network to obtain Image Feature Matrix. Data Augmentation is performed for each spot based on the similarity of spots in spatial combined with the gene expression weights and the spatial location weights. c Final latent embedding is achieved using VGAE and DAE, H1 represents Hidden layer, Z represents the graph embedding, H2 represents Low-dimensional representation. d Latent representations can be used to perform Downstream Analysis.

For the osmFISH and STARmap data without histological images, the gene expression matrix and adjacency matrix were directly used to obtain the augmented gene expression matrix, and the adjacency matrix used in the calculation has a weight of 0.2. Then, principal component analysis (PCA) was applied to the augmented gene expression data to extract the first 50 dimensions of principal components (optional) as latent characteristics. Following this, the augmented gene expression data is trained using a linear layer and a convolutional layer. Specifically, the linear encoder comprises hidden layers of sizes 50 and 20, while the linear decoder consists of hidden layers of sizes 50 and 60. The convolutional layer includes hidden units numbered at 32 and 8. Afterward, a global attention mechanism is introduced into the encoder component of the Variational Graph Autoencoder (VGAE) to capture patterns among intermediate nodes, enabling effective extraction of latent embeddings by VGAE and Deep Autoencoder (Fig. 1c). This combination helps in managing the noise and sparsity in the data, enhancing the model’s ability to learn and represent data features. Finally, the low-dimensional latent representations can be utilized for downstream analyses, including the identification of spatial domains, marker genes, and visualization purposes (Fig. 1d).

EfNST clearly depicts laminar structures in the human dorsolateral prefrontal cortex

To evaluate the fundamental performance of the EfNST algorithm, we utilized the Dorsolateral Prefrontal Cortex (DLPFC)³² data set consisting of 12 slices from spatialLIBD³³. This data set serves as a benchmark due to its distinct and well-established morphological boundaries. The obvious hierarchical structure and labeling information of the sliced data set allow for a visualization of the similarity between the algorithm’s clustering effect and the manually labeled regions. This comparison enables a quantitative evaluation of the algorithm’s efficacy in identifying spatial domains. Within the manually annotated DLPFC region, eight slices exhibit a seven-layer organization. These slices span six cortical layers and white matter (WM), and the order of these layers is layer 1 to layer 6 to the WM layer. The remaining four DLPFC slices feature a five-layer tissue structure, spanning four cortical layers and the WM layer in the sequence of layer 3, layer 4, layer 5, layer 6, and the WM layer.

In the study, we mainly conducted performance testing on two selected brain slices: one with a five-layer structure and the other with a seven-layer structure. To assess the effectiveness of EfNST, we compared its results with those from seven classical methods: Seurat, CCST, Scanpy, SEDR, stLearn, conST, and SpaGCN. In the DLPFC data set, slice 151670 was one of the few slices with five layers of organization (Fig. 2a). Our analysis showed that EfNST successfully identified these five layers with obvious stratification and clearer boundaries, aligning closely with manually labeled structures. There were almost no scattered spots, especially in the consistent positions of WM, Layer 3 and Layer 5. Conversely, the results of Scanpy displayed a lack of hierarchy with most spots in a mixed state. Seurat revealed two hierarchical structures but failed to clearly define the remaining regions. stLearn accurately identified the WM layer but falls short in delineating the other spatial domains. Although the results of CCST had no scattered spots, the location of the identified regions was not accurate. The conST identifies five layered structures but without clear hierarchy or distinct boundaries. SpaGCN identified four layered structures, but its identified regions lack accuracy. Although BASS, ADEPT, and STAGATE identified 5 layers of layered tissue, the specific location of the layer structure was not accurately identified, such as the WM layer. Compared with the ground truth, GraphST and SpatialPCA had accurate position recognition and better spatial domain recognition performance (Supplementary Fig. 1a).

Fig. 2. EfNST improves the ability to recognize the DLPFC layer structures.

a The comparison between the results of EfNST and that of other seven algorithms on slice 151670. b The comparison between the results of EfNST and that of other seven algorithms on slice 151507. c The comparison of four evaluation indicators between the results of EfNST and that of other 7 algorithms.

With a seven-layer hierarchical structure, the slice 151507 was the largest slice (Fig. 2b). The manually annotated layers served as the ground truth for comparison purposes. The EfNST was compared with seven other classic algorithms. The results revealed that the layer structures identified by EfNST exhibited clear boundaries, accurate positions, and no scattered spots. In contrast, the regions identified by Scanpy were chaotic, with most of the structural areas not accurately divided. Although Layer 1 was identified, it also contained spots from other spatial domains. Seurat roughly identified Layer 1, Layer 3 and WM layer, with spots of other layers being randomly distributed. Compared to the ground truth, the identified three layers had a wide positional range and were not very accurate. The clustering results of SEDR also exhibited the same phenomenon. CCST and stLearn had clear boundaries and fewer scattered spots, overall performing better than the previous three methods. However, when compared to the ground truth, their identified positions were not as accurate. For instance, CCST identified both Layer 4 and Layer 3 as Layer 5 with a wide range. The spots in Layer 5 and Layer 6 identified by SpaGCN and conST were mixed, and the boundaries were not well distinguished. BASS and GraphST identified less clear domain boundaries, but both recognized seven-layer organizational structures. The structure layers identified by STAGATE, ADEPT, and SpatialPCA were clear and accurate, demonstrating good clustering performance on slices of seven-layered organizational structures (Supplementary Fig. 1b).

In order to comprehensively evaluate the ability of EfNST, four clustered evaluation indicators were utilized to assess the similarity between the recognized domains of the above two slices and the ground truth benchmark. It is evident that EfNST outperformed the seven previous algorithms in ARI, NMI, FMI, and AMI scores on slice 151670. While on slice 151507, the algorithms stLearn, CCST, SpaGCN, and EfNST demonstrated strong performance (Fig. 2c and Supplementary Fig. 1a, b). The results of F1 scores also demonstrated the superior performance of EfNST compared to these algorithms (Supplementary Table 1). We further expanded the comparison to include more algorithms, and the results also demonstrated the superior performance of EfNST in detecting spatial domains (Supplementary Table 2). Moreover, random seeds were introduced in the image processing process, and the results verified the stability of EfNST (Supplementary Table 3).

EfNST enables finer characterization of human breast cancer tissue spatial domains

The spatial structure of breast cancer plays a pivotal role in understanding the development and progression of the disease. Spatial domain recognition in breast cancer data sets contributes to validating the algorithm’s ability to accurately capture spatial information and handle disease heterogeneity³⁴. Based on extensive previous research, tissue slices are meticulously categorized into 21 regions through manual annotation, elucidating the differences in different regions at a finer scale within breast cancer tissue³⁵. These 21 regions are further classified into four distinct types: Ductal Carcinoma in Situ/Lobular Carcinoma In Situ (DCIS/LCIS), healthy tissue, Invasive Ductal Carcinoma (IDC), and Tumor Edge (Fig. 3a).

Fig. 3. EfNST can recognize the spatial domain of human breast cancer with a finer level of granularity.

a The ground truth and the comparison of four evaluation indicators between the results of EfNST and that of other seven algorithms. b The comparison between results of EfNST and that of other seven algorithms on breast cancer tissue. c The spatial expression distribution of marker genes in breast cancer.

The seven algorithms were compared and analyzed on the breast cancer data set. As shown in Fig. 3b, Seurat identified 20 spatial domains with an ARI of 0.46, but the regions appeared more chaotic and had unclear boundaries compared to other algorithms. Scanpy performed worse than Seurat, with chaotic spot distributions, many clustered and discontinuous spots, and poorly defined region boundaries. It struggled to distinguish between DCIS/LCIS and Tumor_edge regions, resulting in confusing spots within these areas. The stLearn was able to clearly identify the tumor edge regions, including the smaller Tumor_edge_4 in it, and the Healthy regions were identified more excellently. However, the IDC regions in it is poorly recognized, the spots in the region are confusing and boundaries were not clear. SEDR identified the same number of clusters as the ground truth, but its accuracy in identifying the positions of independent regions was low, especially in subdividing multiple tumor edge regions into fragmented areas. CCST accurately identified multiple regions, including IDC_3, IDC_4, Tumor_edge_6, DCIS/LCIS_1, and DCIS/LCIS_5, but other tissue areas were fragmented. The conST performed poorly in recognizing small regions. SpaGCN had excellent clustering identification results but unclear boundaries at different structural domain borders. In comparison, the results of EfNST showed that it could generate more independent clusters, identifying the most independent clusters with the clearest boundaries. The positions of identified spatial domains by EfNST closely matched the ground truth, demonstrating its advantages in consistently detecting the spatial distribution of tumor tissues (Fig. 3a). The four clustering metrics were also higher than the other seven methods (Supplementary Fig. 2). EfNST had the highest ARI value of 0.61, while SEDR and Scanpy had lower scores, and conST and SpaGCN scored between 0.4 and 0.5. CCST, stLearn, and SEDR scored higher between 0.52 and 0.59 (Fig. 3b).

To evaluate the effectiveness of EfNST in detecting disease status or prognosis of breast cancer, we analyzed the spatial expression distribution of well-known breast cancer marker genes. For example, the breast cancer marker FOXA1, which is responsive to hormone therapy and associated with a good prognosis³⁶, displays concentrated expression in the IDC region. As a crucial factor in assessing patient prognosis and links to the invasive characteristics of cancer cells, CCND1³⁷ also shows concentrated expression in the IDC region. Furthermore, the spatial expression distributions of TP53, BIRC5, TOP2A, and MYB were examined (Fig. 3c). Among them, TP53 mutation as a prognostic marker and a predictor for therapy response³⁸. BIRC5 is a marker that indicates a poor prognosis in stage II/III breast cancer and shows no response to neoadjuvant chemotherapy³⁹. TOP2A and MYB are molecular targets for breast cancer treatment, and several therapeutic strategies have been developed to target TOP2A, extending the survival of cancer patients^40,41. Overall, our analysis highlights the potential of EfNST to provide valuable insights into the spatial expression patterns of breast cancer markers. In addition, it underscores the necessity of future studies with larger patient cohorts to validate these findings and further investigate their clinical implications.

EfNST can accurately describe the complex organization of the mouse brain

The mouse brain data set enables researchers to analyze specific brain regions, subregions, and cell types, permitting the observation and analysis of the spatial distribution patterns of gene expression across different anatomical structures. This facilitates the validation of the algorithm in identifying fine tissue structures⁴². To examine the recognition performance of the EfNST algorithm on mouse brain tissues, we used the mouse brain sagittal anterior and CytAssist fresh frozen mouse brain data sets from the 10x Visium platform for testing.

We used eight algorithms to evaluate the clustering effect on the mouse brain sagittal anterior data set with baseline truth (Supplementary Fig. 3a, b). The results showed that the tissue regions identified by conST are the most chaotic, and the boundaries between different regions cannot be clearly observed. The clustering effect of Seurat is slightly better than that of conST, but it also has the problem of unclear boundary hierarchy. The performance of Scanpy and SEDR was comparable, with both methods enabling the visualization of tissue areas, albeit with some chaotic spot patterns in certain regions. SpaGCN and stLearn exhibited equivalent clustering capabilities in the mouse brain sagittal anterior, achieving an ARI score of 0.31. Among the seven algorithms evaluated, EfNST stood out with the highest ARI score of 0.39, demonstrating a clear spatial domain structure and distinct stratification. In contrast, the ARI scores for the remaining algorithms did not exceed 0.36.

To further reveal the ability of EfNST to infer the biological relevance of domains, we examined the expression of relevant marker genes in different domains of the mouse brain sagittal anterior. In Supplementary Fig. 3c, we found that the genes of Enpp2⁴³, Nwd2⁴⁴, Cpne4⁴⁵, C1ql2⁴⁶, Gpr151⁴⁷, and Wsf1⁴⁸ were highly expressed in the tissue slices. This suggests that the mouse brain sagittal anterior is closely associated with cognitive functions, and the role of Wsf1 may influence cognitive functions such as learning and memory in mice. The abnormal expression of C1ql2 may be related to the pathogenesis of neurodegenerative diseases.

To evaluate the ability of the EfNST algorithm in precisely identifying microstructural tissue regions within a data set lacking ground truth, cluster analysis of CytAssist fresh frozen mouse brain was performed. Due to the lack of a ground truth, there is no explicit definition for the number of clusters, so we implemented a comparative analysis for these algorithms with a consistent clustering resolution. The aim of this study is to reveal the differences between the algorithms in capturing the fine structure of brain regions and distinguishing ambiguous regions at the boundaries. Owing to the computational limitations of other algorithms, this paper only compares the Seurat, SEDR, stLearn, Scanpy, and conST methods.

Considering the complexity of brain tissue, we subdivided the data set into 47 regions to achieve a more precise identification of regions in the mouse brain (Fig. 4a). The results showed that conST performed poorly in recognizing mouse brain regions outside the hippocampus, where the spots were disorganized. Within the hippocampus, only the Cornu Ammonis 1 (CA1) and the Dentate Gyrus (DG) regions were identified. In contrast, Scanpy and stLearn identified three regions in the hippocampus: CA1, CA3, and DG, with clearer boundaries. SEDR, Seurat and EfNST achieved the best recognition, displaying clear, symmetrical brain regions and more accurate, less cluttered spot locations in the hippocampus. Seurat and SEDR were clear in recognizing the layering between different structures, but there are still some stray spots in certain areas that mix with spots from other regions. In contrast, the tissue regions identified by EfNST exhibit high symmetry without scattered spots between the structures. This is especially notable in mouse hippocampal tissue, where EfNST can identify fine regions that are difficult for other algorithms to recognize, such as Cornu Ammonis 2 (CA2), the periventricular zone (PZ), and visual area 3 (V3) (Fig. 4b).

Fig. 4. EfNST can recognize complex spatial domains in the CytAssist fresh frozen mouse brain.

a The identification results of spatial domains of six algorithms. b The visualization of the identification results of selected regions. c The annotation map of mouse hippocampus tissue and the visualization results of marker gene expression associated with selected regions.

The CA2 region was highly correlated with the expression of the marker gene Amigo2⁴⁹ and Map3k15 (Fig. 4c). Understanding that Amigo2 can influence cognitive functions such as learning and memory, Map3k15 expression affects social and spatial memory, we speculate that the CA2 region may play a crucial role in social memory and be involved in processing information related to social interactions. In addition, according to literature, Gpr151 and Nwd2 co-locate in the PZ region, and both genes can regulate neurotransmitter systems, such as dopamine, affecting reward-related behaviors. Therefore, we infer that the PZ region may be associated with the regulation of emotion, reward systems, and behaviors related to motivation. Finally, the identified Visual Area 3 (V3) region in the mouse brain, a visual cortex area, may be influenced by Enpp2, as validated by the accuracy of its position under this gene’s marker. The V3 region might participate in visual learning and memory functions in mice. In the annotation regions of the mouse hippocampus (https://atlas.brain-map.org/) (Supplementary Fig. 4), it is observed that the identified relevant regions align well with the original annotations, with high consistency between CA1, CA3, DG regions, and the expression of related marker genes⁵⁰.

Overall, the clustering results of EfNST on mouse brain data demonstrated the advantages of EfNST in identifying spatial domains. EfNST not only accurately identified complex regions of tissues in data with ground truth but also accurately located fine tissues not identified by other methods on data without ground truth. This validated the crucial role of EfNST in deciphering spatial expression patterns and inferring biological functions in biological processes.

EfNST can identify all spatial domains of the mouse somatosensory cortex

To test the scalability of the EfNST algorithm on different sequencing platform, we further utilized the mouse somatosensory cortex data set from the osmFISH technology platform. With 5328 cells and 33 genes, the data set covers the tissue structure of 11 regions including the Hippocampus, Internal Capsule Caudoputamen, Layer 2-3 lateral, Layer 2-3 medial, Layer 3-4, Layer 4, Layer 5, Layer 6, Pia Layer 1, Ventricle and White Matter.

Data incompatibility hindered the utilization of the Seurat method, we selected the remaining 6 methods for comparative analysis (Supplementary Fig. 5a). The results revealed that Scanpy and conST successfully identified all tissue regions but showed poor clustering. Compared to the ground truth, only the White Matter layer was distinct, while the other layers appeared chaotic and indistinguishable. SEDR performed slightly better, revealing the White Matter and the ventricle layer more clearly. CCST provided satisfactory results, especially at the Pia Layer 1 and Hippocampus interface, though the boundaries between layers remained unclear. stLearn identified all regions accurately and showed clear stratification, especially for Layer 1, Internal Capsule Caudoputamen, and White Matter. SpaGCN exhibited poor identification results, with Layer 5 spots dispersed among other structural domains. Notably, EfNST outperformed other algorithms by accurately identifying all regions annotated in the ground truth, including some crucial layers (e.g., Pia Layer 1, Layer 4, and White Matter).

UMAP visualization provides a way to assess the performance of algorithms in capturing data structure. High-density clusters with clear boundaries indicate better performance, and comparing UMAP results from different algorithms highlights their relative effectiveness. As shown in Supplementary Fig. 5b, only the EfNST method clearly separates the 11 spatial domains, while other methods display disorganized groupings. Specifically, EfNST effectively distinguished clusters for White Matter, Layer 6, Layer 5, Layer 4, Pia Layer 1, and Internal Capsule Caudoputamen. All of the structures played fundamental roles in sensory perception, motor control and cognitive functions. Understanding their spatial distribution is essential for investigating neural system functions and abnormalities.

EfNST can accurately identify tissue layers in the mouse visual cortex

The mouse visual cortex data set from the STARmap platform encompasses rich and detailed tissue information, allowing us to evaluate the accuracy and reliability of EfNST. The data set covers a tissue region with 7 hierarchical structures containing the hippocampus (HPC), corpus callosum (CC), Layer 6 (L6), Layer 5 (L5), Layer 4 (L4), Layer 2/3 (L2/3) and Layer 1 (L1).

We compared the performance of EfNST with six other algorithms on the mouse visual cortex data set (Fig. 5). The results indicated that stLearn failed to recognize a clear hierarchical structure in the mouse visual cortex, with scattered spots from different layers mixed together and unclear boundaries (Fig. 5a). While Scanpy and conST also displayed many misclassifications and unclear layer boundaries, they correctly assigned spots to certain structures, particularly in L2/3. SEDR produced similarly scattered results, with the identified L1 layer being too large and inaccurate, and no clear boundaries between L1 and HPC. SpaGCN showed better performance but still had scattered spots across three areas. CCST identified more tissue structures and outperformed Scanpy, SEDR, stLearn, conST, and SpaGCN, but its results lacked clarity, especially in the boundaries between CC and L4. In contrast, EfNST clearly identified the seven-layer structure with well-defined layer boundaries, and the identified regions were more accurate compared to the ground truth.

Fig. 5. The EfNST algorithm can clearly identified entire layers of organizational structure in mouse visual cortex.

a The comparison between the results of EfNST and that of other six classic algorithms on mouse visual cortex. b The UMAP visualization between EfNST and other six algorithms.

Next, UMAP was used to visualize and analyze the embeddings of the seven methods. As shown in Fig. 5b, CCST and SEDR presented many scattered spots, making it difficult to effectively separate mapping clusters. While Scanpy and stLearn could differentiate the mapping clusters of the seven layers, the boundaries between these structures remained unclear, and the clusters appeared somewhat cluttered., However, EfNST demonstrated outstanding performance in identifying the seven-layered structures of the mouse visual cortex data set. It successfully differentiated the various layers, including the challenging hippocampal layer, corpus callosum, and the mapping clusters of L1 and L2/3.

Running speed and ablation studies

To evaluate the efficiency of EfNST, we compared its running speed with several other algorithms across various data sets, including Mouse Brain, Breast Cancer, and DLPFC (Table 1). This comparison highlights EfNST’s ability to significantly improve processing speed under constrained resources without compromising accuracy. These results demonstrated EfNST’s robustness in handling diverse data sets while maintaining its computational efficiency, further underscoring its potential for broader applications in spatial transcriptomics analysis.

Table 1.

The comparison of running speed(s) between EfNST and five classic algorithms on four data sets

	EfNST	stLearn22	SEDR20	Scanpy17	Seurat18	CCST21
Mouse Brain	145	225	264	40	51	1800
Breast Cancer	67	341	491	20	72	5047
DLPFC151507	71	840	735	17	88	1480
DLPFC151670	60	502	736	30	74	1219

Table 1 presents the execution times (in seconds) of six algorithms using a i5 laptop configuration. The other two algorithms of SpaGCN and conST necessitate more advanced settings and are solely executable on a server cluster.

Moreover, to evaluate the key role of several critical components of EfNST (EfficientNet, VGAE, and DAE), a series of ablation studies were performed. For EfficientNet section, five variations of the EfNST model were developed, each utilizes a classic neural network architecture ResNet50, ResNet152, VGG19, DenseNet121, and Inception_v3 to replace EfficientNet to extract image features. The results indicated noticeable decreased in ARI values for the five variants of the EfNST model as compared to the original EfNST model (Supplementary Fig. 6). This further underline the considerable effectiveness of EfficientNet in identifying spatial domains. For VGAE and DAE section, we also conducted a similar ablation study to assess their impact on clustering performance, The results showed that clustering performance was significantly enhanced only when both VGAE and DAE were present (Supplementary Table 4 and Supplementary Fig. 6b).

Discussion

Currently, identifying spatial domains of tissue structure is crucial for mapping cellular heterogeneity, understanding tissue organization, and gaining insights into disease progression and tissue function. Here, we propose a spatial domain recognition algorithm called EfNST, which is based on a composite scaling network. EfNST facilitates multimodal integration of spatial transcriptomics data by extracting image features to accurately identify tissue structural domains. A key component of the algorithm is the use of the EfficientNet network to extract features from spatial transcriptomics images, improving accuracy while reducing parameter complexity and enhancing model speed under resource constraints. Additionally, the integration of VGAE and DAE further enhances the algorithm’s ability to handle potential features.

EfNST demonstrated strong performance across six data sets from three sequencing platforms, including large-scale (e.g., human breast cancer) and small-scale data sets (e.g., mouse visual cortex), showcasing its excellent data generalization and robustness. Compared with seven classic algorithms using four evaluation metrics, the EfNST algorithm achieved better results. In the data set of human Dorsolateral Prefrontal Cortex, the EfNST algorithm exhibits the better results on 12 slices, and identifies laminar structures more clearly with sharper boundaries. The results on data sets of human breast cancer and mouse brain reveal its ability to dissect spatial domains of complex tissues structure at a finer scale. It can also identify the areas that contrast algorithms fail to recognize. The highly consistent expression of relevant marker genes in literature validates the precise localization of unlabeled regions.

The superior performance of EfNST is largely attributed to its foundation on EfficientNet. EfficientNet’s strength lies in its innovative scaling method, rigorous experimental validation, optimized baseline design, and broad applicability. These factors collectively underscore the rationale and effectiveness of using EfficientNet for spatial domain recognition. Random seeds were introduced in the image processing process, and the results verified the stability of EfNST. Moreover, incorporating the Variational Graph Autoencoder (VGAE) for learning latent representations and the Denoising Autoencoder (DAE) for noise reduction has provided significant benefits. Related ablation experiments have further highlighted the critical role of these components.

Our work on the EfNST algorithm framework has achieved significant results in image feature extraction by incorporating the EfficientNet module to handle the large amount of image data in ST. It has not only improved model accuracy but also significantly enhanced efficiency. However, EfficientNet still has some limitations. For instance, its performance is dependent on the quality and quantity of training data, and the required computational resources increase as the model size grows (e.g., from EfficientNet-B0 to EfficientNet-B7). Furthermore, due to its complex network structure and nonlinear feature extraction capabilities, understanding and interpreting the model’s decision-making process becomes challenging. Additionally, EfNST is currently unable to be applied to multiple regions simultaneously and identify aligned spatial domains across tissues. Referring to the latest researches, on the basis of optimizing algorithms for individual slice, appropriately increasing multi-slice-related reference data from different platforms can enhance the recognition performance of the spatial domain on a single slice. In future work, we can use the EfNST method to learn shared latent spot embeddings after joint training on multiple slices. Alternatively, we can align H&E images from different slices to a common spatial reference, creating a unified neighborhood graph, and then utilize the EfNST method for comprehensive training and precise spatial domain recognition.

Methods

Data sets and preprocessing

To evaluate the clustering performance of EfNST, we used six data sets from three sequencing platforms of 10x Visium, osmFISH and STARmap. The three 10x Visium data sets includes the human Dorsolateral Prefrontal Cortex (DLPFC) (http://research.libd.org/spatialLIBD/), the human breast cancer, the mouse brain sagittal anterior and CytAssist fresh frozen mouse brain (https://www.10xgenomics.com/). Specifically, the DLPFC data set consists of 12 slices sampled from three experimental individuals. The number of spots per slice varies from 3498 to 4789. The data sets of human breast cancer and mouse brain sagittal anterior contains 2696 and 3798 spots, respectively. The data set of mouse somatosensory cortex from the osmFISH of subcellular level contains 5328 cells and 33 genes (http://linnarssonlab.org/osmFISH/). The data set of mouse primary visual cortex from the STARmap of subcellular level contains 1207 cells and 1020 genes (https://kangaroo-goby.squarespace.com/data). All of the data sets have benchmark truths to quantitatively evaluate the performance of the various algorithms. The mouse whole brain data set has 4298 spots with no baseline truth to verify the performance of different algorithms in recognizing unknown complex regions. In the data preprocessing stage for all data sets, we performed Quality Control, Log transformation and Normalization on the gene expression data, and then reduced the dimensionality of the gene expression profiling in the ST data set using Principal Component Analysis (PCA).

EfficientNet

EfficientNet is a principle-based way of augmenting convolutional networks. Unlike traditional methods that scale only one dimension, EfficientNet introduced an innovative compound scaling method that systematically balances the network’s depth, width, and resolution. Empirical evidence demonstrated that uniformly scaling all three dimensions ensured higher accuracy and efficiency within a given computational budget. Additionally, the baseline network of EfficientNet was designed through neural architecture search, optimizing both model accuracy and computational efficiency. Moreover, EfficientNet has shown outstanding performance in multiple transfer learning tasks, indicating its excellent generalization capabilities and suitability for various computer vision applications. In summary, the rationale for selecting the EfficientNet network structure lies in its innovative scaling method, successful experimental validation, optimized baseline design, and broad applicability. These factors collectively demonstrate the rationale and effectiveness of using EfficientNet for spatial domain recognition.

The workflow diagram of EfficientNet is shown in Supplementary Fig. 7. In the workflow diagram, Mobile Inverted Residual Bottleneck Convolution (MBConv)^51,52 module is a specific building block used in convolutional neural network (CNN) architectures, which combines Depth-wise Separable Convolution and Residual Connection. MBConv contains key components such as Inverted Residual Block, Bottleneck Design, Activation and Regularization. It is designed to provide efficient yet effective feature extraction, particularly suited for mobile and embedded devices where computational resources are limited. As the basic module in EfficientNet, MBConv modules are scaled up to create a family of models optimized for both accuracy and efficiency across different scales and achieve state-of-the-art performance across different resource constraints. Firstly, the histological image is transformed through the first Conv3x3 layer to the input dimension required by the MBConv module. Then, the image goes through a series of MBConv modules to extract feature maps; the parameters of each MBConv module are finely adjusted to adapt to the current operating environment. A combined scale optimization method can enable the network to achieve a better receptive field. Next, an adaptive connection method based on the Fully Convolutional Neural Network feature map is utilized. A Conv1x1 network is used to adapt to feature maps of various sizes and unify them to the dimension required. Finally, the image’s classification recognition detection is completed through the output feature map. The corresponding mathematical formulas are as follows:

$${\max }_{d,w,r}Accuracy\left(\mathcal{N}\left(d,w,r\right)\right)$$

$$s.t.\mathcal{N}\left(d,w,r\right)={\odot }_{i=1\dots s}{F}_i^{d.L_i}\left(X_{⟨r.H_i,r.W_i,r.C_i⟩}\right)$$ 1

$$Memory\left(\mathcal{N}\right)\le {target}_{memory}$$

$$FLOPS\left(\mathcal{N}\right)\mathcal{\backslash le}{target}_{flops}$$

Where $d$, $w$ and $r$ are coefficients used to scale the depth, width and resolution of the network. $F_i$, $L_i$, 〈H_i, W_i〉 and $C_i$ represent Operator, Layers, the resolution of the input and Channels, where i represents the Stage.

Tan and Le founded that the optimized $d$, $w$ and $r$ were interdependent, so they proposed a new uniform scaling method for $d$, $w$ and $r$, which was a necessary requirement for achieving better accuracy and efficiency²⁵. The method selects the composite coefficient ϕ scale the network width, depth and resolution uniformly in a principled manner. The corresponding mathematical formula are as follows:

$$\mathrm{depth}:d={\alpha }^{\varphi }$$

$$width:w={\beta }^{\varphi }$$

$$resolution:r={\gamma }^{\varphi }$$ 2

$$s.t.\alpha \mathrm{\cdot }{\beta }^2\mathrm{\cdot }{\gamma }^2\approx 2$$

$$\alpha \ge 1,\beta \ge 1,\gamma \ge 1$$

where $d,w$ and $r$ are dynamic coefficients used to control depth, width and resolution, and ϕ is a constant used to control model complexity.

The series of models obtained after expanding the EfficientNet baseline network were called EfficientNets. Compared to previous ConvNets, EfficientNets have higher accuracy and efficiency. In comparison with other popular deep learning frameworks, such as ResNet²⁸, Inception⁵³, and MobileNet⁵⁴, EfficientNets can achieve better computational performance. In addition, EfficientNet also show the best performance in image processing compared to other image classification benchmarks, such as ImageNet⁵⁵ and CIFAR-100⁵⁶.

At the same time, EfficientNets can better adapt to robust to data expansion and noise while improving accuracy and efficiency. Thus, it is more suitable for practical applications has in resource-constrained environments. Currently, the EfficientNets are popularly used in biomedical aspects, such as breast cancer detection⁵⁷, retinal disease screening⁵⁸, and brain tumor classification⁵⁹.

Data augmentation

Data augmentation involves applying a series of transformations or operations to the original data to enhance its quality. These transformations are typically reversible and do not alter the labels of the samples. In this paper, we use spatial location information corresponding to gene expression data and histological image information to augment the gene expression of neighboring spots. With reference to the previous work⁶⁰, the process aims to enhance the quality of the data set for subsequent analysis.

(1) The spatial gene expression weight values between spot ${\mathit{S}}_{\mathit{i}}$ and spot ${\mathit{S}}_{\mathit{j}}$ are calculated using the gene expression correlation, and the corresponding mathematical formula is as follows:

$$\mathit{G}{\mathit{C}}_{\mathit{ij}}=1-\left({\mathit{S}}_{\mathit{i}}-\overline{{\mathit{S}}_{\mathit{i}}}\right)\cdot \left({\mathit{S}}_{\mathit{j}}-\overline{{\mathit{S}}_{\mathit{j}}}\right)/\parallel \left({\mathit{S}}_{\mathit{i}}-\overline{{\mathit{S}}_{\mathit{i}}}\right){\parallel }_{\mathit{2}}\parallel \left({\mathit{S}}_{\mathit{j}}-\overline{{\mathit{S}}_{\mathit{j}}}\right){\parallel }_{\mathit{2}}$$ 3

(2) The high-dimensional image features of each spot tile are extracted from a pre-trained efficientnet-b0 model. To better represent image features, 50 principal components after PCA dimension reduction were used as latent features. Finally, cosine similarity is used to calculate the image similarity weight ${\mathit{IS}}_{\mathit{ij}}$ between spots. The specific formula is as follows:

$$\mathit{I}{\mathit{S}}_{\mathit{ij}}=\mathit{1}-{\mathit{S}}_{\mathit{i}}\cdot {\mathit{S}}_{\mathit{j}}/\parallel {\mathit{S}}_{\mathit{i}}{\parallel }_{\mathit{2}}\parallel {\mathit{S}}_{\mathit{j}}{\parallel }_{\mathit{2}}$$ 4

(3) Spatial adjacency mainly uses the spatial location information to determine the distance between each spot and all other spots, and we select the four (optional) nearest neighbors based on the distance from spatial coordinates and calculate the radius as the sum of their mean and variance. For a given spot ${\mathit{S}}_{\mathit{i}}$, a spot ${\mathit{S}}_{\mathit{j}}$ can be viewed as one of the neighbors, if and only if the distance between spot ${\mathit{S}}_{\mathit{i}}$ and spot ${\mathit{S}}_{\mathit{j}}$ is less than the radius, $\mathit{S}{\mathit{W}}_{\mathit{ij}}=\mathit{0}$, otherwise, $\mathit{S}{\mathit{W}}_{\mathit{ij}}=\mathit{1}$.

Finally, the enhances gene expression of each spot ${\mathit{S}}_{\mathit{i}}$ is expanded to ${\overline{\mathit{GA}}}_i$, and the formula for the 10x Visium platform is as follows:

$${\widetilde{\mathit{GA}}}_{\mathit{i}}={\mathit{GE}}_{\mathit{i}}+\frac{{\sum }_{\mathit{j=1}}^{\mathit{n}}\mathit{G}{\mathit{E}}_{\mathit{j}}\cdot \mathit{I}{\mathit{S}}_{\mathit{ij}}\cdot \mathit{G}{\mathit{C}}_{\mathit{ij}}\cdot \mathit{S}{\mathit{W}}_{\mathit{ij}}}{\mathit{n}}$$ 5

The formula used for the platforms of osmFISH and STARmap is as follows:

$${\widetilde{\mathit{GA}}}_{\mathit{i}}=\mathit{G}{\mathit{E}}_{\mathit{i}}+\frac{{\sum }_{\mathit{j=1}}^{\mathit{n}}\mathit{G}{\mathit{E}}_{\mathit{j}}\cdot \mathit{G}{\mathit{C}}_{\mathit{ij}}\cdot \mathit{S}{\mathit{W}}_{\mathit{ij}}}{\mathit{n}}$$ 6

Where $\mathit{G}{\mathit{E}}_{\mathit{i}}$ and $\mathit{G}{\mathit{E}}_{\mathit{j}}$ represent the original gene expression of spot ${\mathit{S}}_{\mathit{i}}$ and n adjacent sites ${\mathit{S}}_{\mathit{j}}$.

Variational graph autoencoder

The VGAE²⁶ was proposed by Kipf and Welling utilized latent variables to learn latent representations in graph-structured data based on variational autoencoder (VAE). According to the given unweighted undirected graph G and its adjacency matrix A, it uses the latent variable $Z_i$ to reconstruct the Adjacency matrix:

$$q\left(Z\mid X,A\right)={\prod }_{i=1}^{N}q\left(z_i\mid X,A\right)$$ 7

X is the feature matrix of the nodes; A is the adjacency matrix:

$$q\left(Z_i\mid X,A\right)=N\left(z_i\mid {\mu }_i,diag\left({\sigma }_i^2\right)\right)$$ 8

where $\mu =GCN\mu \left(X,A\right)$ is the mean of the feature vector; $\log {\sigma }^2=GCN\sigma \left(X,A\right)$ is the variance of the node vector.

The defined two-layer neural network is:

$$GCN\left(X,A\right)=\tilde{A}ReLU\left(\tilde{A}X{W}_0\right)W_1$$ 9

$\tilde{A}=D^{-\frac{1}{2}}A{D}^{-\frac{1}{2}}$ is the symmetric normalized adjacency matrix.

The last-layer neural network generates μ and logσ² share the second level parameter $W_1$;

After this, the decoder reconstructs the adjacency matrix using the inner product of the latent variables.

$$p\left(A\mid Z\right)={\prod }_{i=1}^N{\prod }_{j=1}^{N}p\left(A_{ij}\mid z_i,z_j\right)$$ 10

Among them $p\left(A_{ij}=1\mid z_I,z_j\right)=\sigma \left(z_i^T{z}_j\right)$.

The KL scatter is added to the loss function to improve the generalization ability and robustness of the model:

$$L=E_{-}\left(q\left(Z\mid X,A\right)\right)\left[logp\left(\right)A\mid Z\right]-KL\left[q\left(Z\mid X,A\right)\parallel p\left(Z\right)\right]$$ 11

Where $p\left(Z\right)={\prod }_{i}N\left(z_i\mid 0,I\right)$.

Denoising autoencoder

Denoising Autoencoder (DAE)²⁷ is a powerful tool in the realm of unsupervised learning and data preprocessing. It is effective in learning useful representations and features from noisy data, which can improve the performance of downstream tasks such as classification or clustering. The augmented gene expression data in this paper includes various noises, such as background noise in histological image, dropout events of gene expression, spatial noise in sampling process. Thereby, we used a DAE to learn a nonlinear mapping from the integrated feature space to a low-dimensional representation space. By learning to remove noise from data, the model also learns more robust features and representations of the data.

In our work, the DAE is a variation of the traditional autoencoder which is a type of neural network designed to learn efficient representations of data, typically by compressing the input into a latent-space representation and then reconstructing the output from this representation. The encoder is composed of multiple fully connected stacked linear layers, which send the integrated gene expression X into the encoder. Each layer is linear and nonlinear activated to achieve the purpose of extracting deep features. The final coder transforms the gene expression data into a low-dimensional representation ${\mathit{Z}}_{\mathit{g}}$, with the specific formula as follows:

$$E\left(\mathit{X}\right)={\mathit{Z}}_{\mathit{g}},\mathit{X}\in R^{N\times G},{\mathit{Z}}_{\mathit{g}}\in R^{N\times R}$$ 12

Where N is the total number of spots, G is the number of genes, and R is the dimension of the last-layer encoder. In contrast to this, decoder reconstructed the original input to obtain ${\mathit{X}}^{\prime }$, expressed as:

$${\mathit{Z}}_{{\mathit{g}}^{\prime }}={\mathit{Z}}_{\mathit{g}}+\mathit{Z}$$

$$D\left({\mathit{Z}}_{{\mathit{g}}^{\prime }}\right)={\mathit{X}}^{\prime },{\mathit{Z}}_{{\mathit{g}}^{\prime }}\in R^{N\times \left(R+R^{\prime }\right)},{\mathit{X}}^{\prime }\in R^{N\times G}$$ 13

Dimensionality reduction and clustering

As a nonlinear technique, uniform manifold approximation and projection (UMAP)⁶¹ maps high-dimensional data to a lower-dimensional space while preserving both local and global data structures^62,63, revealing clustering patterns of gene expression at different sites. The clear boundaries and high-density clusters in UMAP visualizations indicate the effectiveness of this algorithm in capturing structural information in the data, thereby allowing for indirect comparison of different algorithms’ performance.

EfNST uses Leiden⁶⁴ for clustering. Regarding the number of clusters, EfNST, and the algorithms compared in this paper follow the same strategy: For data sets with ground truth, we directly adjust the number of clusters to match the ground truth. For data sets without ground truth, we made a comparison at the same level of granularity.

Evaluation indicators

The Adjusted Rand Index (ARI)⁶⁵, Adjusted Mutual Information (AMI)⁶⁶, Fowlkes-Mallows Index (FMI)⁶⁷, and Normalized Mutual Information (NMI)⁶⁸ provide a more comprehensive and robust assessment of clustering performance than the metrics Specificity and Sensitivity^69,70, which are used to assess the performance of a classification task, and are better suited to evaluate the degree of consistency between clustering results generated by different algorithms and the benchmark truth. Therefore, four clustering external evaluation metrics in the Scikit-learn package⁷¹ are used to assess the clustering performance of the algorithms, among which ARI is usually used as the main metric to measure the clustering ability in spatial domain identification because it integrates the completeness, consistency, and randomness of the clustering results.

The formula for the evaluation metrics is as follows:

$$ARI=\frac{2*\left(TP*TN-FN*FP\right)}{\left(TP+FN\right)\left(FN+TN\right)+\left(TP+FP\right)\left(FP+TN\right)}$$ 14

Where TP, TN, FP, and FN represent true positive, true negative, false positive and false negative. The value range of ARI is [0,1], a higher ARI score indicates a stronger consistency between the clustering results and the ground truth.

$$AMI=\frac{\left(MI-E\left[MI\right]\right)}{\left(\max \left(H\left(U\right)\right),H\left(V\right)\right)-E\left[MI\right]\left)\right)}$$ 15

MI denotes mutual information, H(U) and H(V) represent the entropy of the clustering result and the ground truth, respectively. Meanwhile, E[MI] denotes the expected mutual information with random assignment. A higher AMI score indicates a stronger clustering ability of the algorithm.

$$NMI\left(g,L\right)=\frac{2*I\left(g;L\right)}{H\left(g\right)+H\left(L\right)}$$ 16

The variable $g$, $L$, and $I\left(g;L\right)$ represent the ground truth, the result of clustering, the Cross Entropy, and the mutual information. The mutual information is calculated using the formula $I\left(g;L\right)=H\left(g\right)-H\left(g\mid L\right)$.

Fowlkes–Mallows Index (FMI) is commonly used to measure the similarity between clustering and the ground truth in clustering, its score range is [0,1].

$$\mathit{FMI}=\frac{TP}{\sqrt{\left(TP+FP\right)\left(TP+FN\right)}}$$ 17

Where $TP$, $FP$ and $FP$ respectively represent the number of true positives, the number of false positives and false negative. A higher FMI score indicates a good similarity between the clusters and the ground truth.

Ablation study

The EfNST algorithm utilizes the EfficientNet framework for image feature extraction, and integrate the gene expression information and the spatial location information to establish a comprehensive feature system. To evaluate the effectiveness of EfficientNet in identifying spatial domains, a series of ablation studies were performed on three main data sets of DLPFC, human breast cancer data sets and mouse brain with known ground truth. Five variations of the EfNST model were developed, each utilizes a classic neural network architecture ResNet50, ResNet152, VGG19, DenseNet121, and Inception_v3 to replace EfficientNet to extract image features. To ensure the fairness of the experimental results, all other parameters settings of model were kept consistent.

An ablation study on VGAE and DAE in the EfNST algorithm was conducted to assess their impact on clustering performance. The results showed that clustering performance is significantly enhanced only when both VGAE and DAE are present (Supplementary Fig. 6b).

Comparison of EfNST with other algorithms

For the detailed parameter settings of EfNST and other algorithms (Seurat, Scanpy, SEDR, conST, stLearn, CCST, SpaGCN, ADEPT⁷², BASS⁷³, SpatialPCA⁷⁴, GraphST⁷⁵, STAGATE) used for the comparison can be found in the Supplementary Materials.

Statistics and reproducibility

Data processing and analysis were conducted with Python (version 3.9.16) along with the following libraries: Pandas (version 1.5.2), numpy (version 1.22.4), scipy (version 1.8.0), and torch (version 2.0.1). For data set preprocessing, we utilized scanpy (version 1.9.1), anndata (version 0.8.0), and matplotlib (version 3.6.3). The visual representations in our research were created using scanpy (version 1.9.1) and OriginPro software.

Reporting summary

Further information on research design is available in the Nature Portfolio Reporting Summary linked to this article.

Author contributions

Z.L., Z.F., and Y.Z. (Yongchun Zuo) conceived and designed the study. Y.Z. (Yanan Zhao) and C.L. were responsible for data collection and preprocessing, developed the algorithm, performed all data analysis, conducted comparative evaluations, and contributed to the drafting and revision of the manuscript. W.S. and Z.S. participated in the data analysis and helped to revise the manuscript. All authors read and approved the manuscript.

Peer review

Peer review information

Communications Biology thanks the anonymous reviewers for their contribution to the peer review of this work. Primary Handling Editors: Aylin Bircan and Mengtan Xing.

Data availability

This study utilizes six data sets from three sequencing platforms. The 10x Visium platform includes four data sets: the human dorsolateral prefrontal cortex (DLPFC) data set (http://research.libd.org/spatialLIBD/), along with data sets for human breast cancer, mouse brain sagittal anterior, and CytAssist fresh frozen mouse brain (https://www.10xgenomics.com/). The osmFISH platform provides the mouse somatosensory cortex data set (http://linnarssonlab.org/osmFISH/). The STARmap platform includes the mouse primary visual cortex data set (https://kangaroo-goby.squarespace.com/data).

Code availability

The code of this study is available at https://github.com/Zaoyanan/EfNST/ and 10.5281/zenodo.14059229⁷⁶.

Competing interests

The authors declare no competing interests.

Supplementary information

The online version contains supplementary material available at 10.1038/s42003-024-07286-z.