Spatialsmooth: a spatially-aware convolutional autoencoder framework for enhanced deconvolution of spatial transcriptomics data

Abstract

Spatial transcriptomics technologies have revolutionized our understanding of tissue heterogeneity by providing gene expression data with spatial context. However, due to the limited resolution of some of these techniques, multiple cell types are often captured at each spatial location. Several spatial deconvolution methods have been developed in recent years to infer cell type compositions from these mixed spots, but their results can be noisy and spatially inconsistent. In this study, we introduce a spatial smoothing method, Spatialsmooth based on convolutional autoencoder, in which several spatial deconvolution tools are integrated to obtain the deconvolution results, the spatial location information is fully utilized using positional encoding, and the deconvolved cell type compositions are smoothed using a convolutional autoencoder to optimize the distribution of different cell type compositions in space. This method integrates multiple spatial deconvolutions tools and fully utilizes spatial location information and cell type composition inferred by multiple deconvolution algorithms to produce smooth, biologically plausible cell type distributions. The smoothing effect was measured by using three spatial metrics, namely, Moran's I, Geary's C, and Total Variation, on multiple pairs of datasets. pancreatic ductal adenocarcinomas (PDAC) dataset, Spatialsmooth reached 0.52 on the Moran's I metric (92% higher than the Redeconve method), the Geary's C metric was reduced to 0.47 (a 45% improvement compared to RCTD), and Total Variation was reduced by 22%. Applied to benchmark data, multiple cell types and molecular markers with different spatial localizations were identified, closer to the spatial projection of marker genes than other deconvolution tools. (https://github.com/njjyxl/Spatialsmooth).

Supplementary Information

The online version contains supplementary material available at 10.1186/s12864-025-11959-2.

Introduction

Spatial transcriptomics techniques (e.g., 10 × Visium [1] and Slide-seq [2]) enable gene expression profiling while preserving spatial information. These techniques provide valuable information for understanding the spatial organization and interactions of different cell types within a tissue. However, due to the limited resolution of these techniques, each spatial location often contains a mixture of multiple cell types [3]. Deconvolution algorithms have been developed, such as CARD [4], RCTD [5], SPOTlight [6] and SpatialDWLS [7], and the numerous deconvolution tools are quite independent of each other and can only be run independently of each other, which is a great inconvenience to use; on the other hand, deconvolution is used to estimate the proportions of the different cell types within each spatial location, and although these methods can provide useful estimates, the resulting proportions can be noisy and spatially inconsistent with high heterogeneity. Therefore, there is significant value in integrating multiple spatial deconvolution tools to optimize the prediction of cellular fractions in order to determine the transcriptomic landscape on tissues [8–16] to improve the reliability of studies of biological mechanisms. To this end, we developed an R package called Spatialsmooth, which integrates multiple spatial deconvolutions tools and utilizes convolutional autoencoders to optimize the prediction of cell composition in spots. Spatialsmooth's application on various published datasets has shown that it has a better facilitation in smoothing the deconvolutionally inferred cell-type composition. In the context of pancreatic cancer and human thymus cells, Spatialsmooth better enhances the deconvolution inference results in close agreement with the spatial projection of marker genes. In addition, Spatialsmooth proved to be effective in elucidating the negative correlations in cTEC and mTEC of human thymocytes. The Spatialsmooth framework integrates a variety of deconvolution tools including CARD, RCTD, SPOTlight, SpatialDWLS, Seurat [17], AdRoit [18], Redeconve [19] and SpatialDecon [20] which contribute to the inferred results of deconvolution tools and bring greater significance to the study of biology in revealed space.

Results

The spatialsmooth workflow

Spatialsmooth is a comprehensive framework that integrates multiple spatial deconvolutions tools and performs spatial smoothing optimization (Fig. 1). Single deconvolution tools can indeed achieve reasonable accuracy in their respective application scenarios, but they have inherent limitations. Different tools produce different results for the same dataset, which means that each tool uses a different model. Based on this, it is necessary to optimize the deconvolution results. In terms of functional architecture, it can be divided into two main modules:

Fig. 1.

Overview of the Spatialsmooth schematic. CAE for spatial smoothing aims at smoothing the cell type matrix obtained by various types of deconvolution methods and is used to ensure that this spatial distribution is smooth and continuous, meaning that changes in cell type do not suddenly and abruptly change spatially, but rather gradually. This is important for both the fidelity and reliability of biological data, as biological tissues are usually structured and continuous. CAE requires a deconvolutionally inferred cell type composition for each spatial location (Fig. 1 A left) as well as localization information for spatial transcriptomics data (Fig. 1B top). With these two inputs, the positional encoding is integrated with the matrix for convolutional self-encoding via CAE (Fig. 1C bottom). Using a custom loss function and the Adam optimization method, a smoothed cell type composition is output (right). a unique feature of CAE is the ability of the TV loss to enhance the structural continuity of the spatial distribution of cells, which is important for the simulation of real biological phenomena

The deconvolution toolbox module (Fig. 1A) integrates eight mainstream spatial transcriptome deconvolution methods: CARD, RCTD, SPOTlight, Seurat, SpatialDWLS, AdRoit, Redeconve, and SpatialDecon, which have their own unique features, and are able to parse spatial transcriptome data from different perspectives, providing more accurate information an out the composition of cell types.

The deconvolution methods are integrated by calling the eight methods uniformly through the main function run_deconvolution_methods. First the input formats are standardized, e.g., single-cell data (H5Seurat) and spatial data paths, cell type annotated column names. Using dynamic parameter allocation, special methods (such as SpatialDWLS) pass additional parameters (Python paths) through conditional judgment. Encapsulate each method independently as a function, retaining native algorithmic properties, managing method name and function mapping through a registry (methods list), supporting hot-pluggable extensions, adding new methods by simply adding a function and updating the registry, either together or as a single method.

The spatial smoothing optimization module (Fig. 1B) employs a three-step processing strategy: first, it converts the 2D coordinate information of each spatial location into a high-dimensional numerical vector by position encoding technique, which can effectively capture the spatial distribution characteristics of the cells in the tissue. Second, it uses a convolutional autoencoder (Fig. 1C) to spatially smooth the deconvolution results, and at the same time ensures that the smoothing results maintain spatial continuity without excessive deviation from the original prediction by means of a well-designed dual loss function (structural similarity index measure SSIM and Total variation TV). Finally, by comprehensively evaluating multiple spatial statistical metrics, the optimal cell type composition prediction is filtered from the prediction results of multiple deconvolution tools to ensure that the final results are both biologically consistent and have good spatial continuity.

Through this strategy of multi-tool integration and spatial optimization, Spatialsmooth is able to overcome the limitations of a single deconvolution method and provide more accurate and reliable predictions of cell type spatial distribution.

Comparison of different deconvolution tools and Spatialsmooth using simulated data

To evaluate the performance of Spatialsmooth, we generated PDAC simulation data according to the simulation strategy in (Fig. 2a), with 428 points and 20 cell types. Based on pancreatic ductal adenocarcinoma (PDAC) spatial transcriptome data, we designed a three-stage screening framework: 1) single-tool pre-screening: calculate the SSIM and correlation (Pearson's r) of each tool (CARD, RCTD, etc.) with the ground truth (scRNA-seq) and rank the weighted scores; 2) stepwise Combination optimization: add tools step by step in descending order of Score to evaluate the combination performance (Fig. 2b); 3) Inflection point determination: stop when Score decreases, determine the optimal combination as CARD + RCTD + SpatialDWLS + AdRoit (Score = 0.768, SSIM = 0.703, r = 0.731) (Supplementary Table 6). Comparisons were made between real cell type composition, CARD and Spatialsmooth. performance evaluation was performed using simulation data (Fig. 2c) using three evaluation metrics Moran's I [21], Geary's C and Total Variation.

Fig. 2.

Proportion matrix simulation strategy and smoothing performance evaluation. a The simulation strategy uses rpois to generate cell counts from existing Spatial and scRNA-seq data, and records the cell counts and proportions at each spot to obtain True proportion. b Add tools step by step in descending order of Score to evaluate the combination performance. c Comparison of the eight deconvolution methods and Spatialsmooth, with True Proportion in two spatial metrics and total variation versus the smoothing metrics. d Comparison of the four organizational metrics for the cell type composition of the cell type composition of the simulated PDAC at the spatial points, PCC, SSIM, RMSE, and JSD values for the cell type composition of the spots simulated from dataset 11, generated by eight integration methods. center line, median; box limits, Center line, median; box limits, upper and lower quartiles; whiskers, 1.5 × interquartile range. e The left part shows the simulated PDAC cell type composition, and in the middle, each spatial location, showing the proportion of cells per point type inferred by deconvolution and the cell type composition after Spatialsmooth treatment. The right part shows that to assess the aggregation of cell types in specific regions, we used Getis-Ord Gi hotspot analysis as a complement to Moran's I and Geary's C to detect the aggregation of cell types in localized areas of high (hotspots) or low (coldspots) concentration and correlated the results with the Spatialsmooth results using ground-truth Correlation of Spatialsmooth results with ground truth was analyzed and correlations with ground truth using Spatialsmooth were found to be as high as 0.95 or higher

Where after smoothing there is a significant increase in Moran's I, reaching above 0.5, Geary's C reached 0.47 shows that the smoothed value is further away from 1 indicating an increase in its spatial autocorrelation, while a decrease in the value of Total Variation reached 682.7 indicates that the smoothing process reduces the overall variability of the data, suggesting that after spatial smoothing, the composition of the cell types is better. In spatial spot analysis (Fig. 2d), each spatial spot represents a physical location on the tissue section, with a focus on the cell type composition at a single location. Evaluating the resolution accuracy of each spatial location reflects the algorithm's performance in handling spatial heterogeneity and aids in understanding the cellular composition of the local microenvironment. Spatialsmooth performed best in terms of PCC (0.857) and SSIM (0.818), while CARD performed similarly to Spatialsmooth in terms of RMSE (0.048) and JSD (0.278) metrics.

By generating simulated cell type compositions as real cell type compositions, the technical details of which are described in the method section. We evaluate the performance of Spatialsmooth through simulation and compare it with eight existing deconvolution methods (CARD, SPOTlight, RCTD, Seurat, SpatialDWLS, AdRoit, Redeconve and SpatialDecon). To further verify the smoothing effect, we showed the spatial projections of Acinar cells, Cancer clone A, and Ductal centroacinar (Fig. 2e), and the simulation results showed that acinar cells were locally distributed in certain regions, and both Spatialsmooth and CARD projections showed a decrease in the contrast of the corresponding regions. but Spatialsmooth results exhibit higher correlation with the ground truth, The correlation coefficient for ductal centroacinar cells reached 0.954.

To assess the concentration distribution of cell types in a specific region, we employed the Getis-Ord Gi hotspot analysis as a complementary method to Moran's I and Geary's C to detect high-concentration (hotspots) or low-concentration (coldspots) aggregation phenomena of cell types within local areas, and conducted a correlation analysis between ground truth and Spatialsmooth results (Fig. 2e). Fig. S3 shows the hotspot analysis results of other methods. The results indicate that Spatialsmooth exhibits the highest correlation.

Comparison of methods on different datasets

Ten pairs of spatial and single-cell RNA sequencing (scRNA-seq) datasets were collected from published studies. The number of genes, spots, and data sources for each dataset are detailed in Supplementary Table 1, which includes eight pairs of real datasets and two pairs of simulated datasets collected from published literature. Detailed information about the datasets used and their cell types is described in Supplementary Table 2. Firstly, in a spatial comparison between the benchmark data and the cell type composition after Spatialsmooth smoothing, Spatialsmooth exhibits a larger Moran's I, close to 1, a smaller Geary's C after smoothing, away from 1, and a significantly smaller value of Total Variation, obviously after smoothed cell type composition has a better spatial representation (Fig. 3a). Comparison of boxplots allows identification of which method performs best or most consistently on a given metric (Fig. 3b). Spatialsmooth's median on the boxplot is significantly different from the rest of the methods, with Moran's I being higher; Geary's C and Total Variation have significantly smaller values, presuming that the method is spatially superior. Spatialsmooth has good spatial performance when i, ii, and iii in (Fig. 3c) compare 10 pairs of datasets using different deconvolution methods with smoothing. Three metrics of changes in PDAC and Mouse Olfactory Bulb (MOB) are shown in Supplementary Table 3. (Fig. S1) primarily demonstrates the degree of similarity between the smoothed cell type composition and the original cell type composition in terms of correlation, structural similarity, and probability distribution. The matrix smoothed using CAE shows significant improvements compared to the other eight methods in terms of spatial autocorrelation, spatial local autocorrelation, and total variance scores. Additionally, based on the values of JSD, PCC SSIM, and MSE, it is demonstrated that the probability distribution of Spatialsmooth is related to the tools used, and the difference between the optimized proportional matrix after smoothing and the original matrix is not significant, with structural similarity typically exceeding 0.8.

Fig. 3.

Evaluation of CAE performance on 10 pairs of datasets. a Evaluation of True & Spatialsmooth's performance on three spatial smoothing metrics. Moran's I's before and after smoothing become larger and close to 1, Geary's C decreases after smoothing and moves away from 1, and the value of Total Variation decreases significantly, indicating that going into CAE reduces the overall variability of the data, suggesting that spatial smoothing was successful. b Comparison of box plots identifies which method performs best or most consistently on a given indicator. The box plot of Moran's I shows that its median is significantly higher than the other methods and has less volatility, so it can be assumed that the method performs better on that indicator. Geary's C, Total Variation, on the other hand, performs better the smaller it is. c Spatial and smoothing performance on all data. By comparing different deconvolution tools with Spatialsmooth, it is clear that the cell scale matrix after Spatialsmooth has a better spatial structure

Memory and time

Using 10 datasets (human heart, MOB, mouse brain, mouse hippocampus, human pancreatic ductal adenocarcinoma, seqFISH +, human breast cancer, human lymph node) and simulated data obtained from other public literature (Supplementary Table 2), we conducted a detailed comparative analysis of the performance of different spatial transcriptomics deconvolution methods. The performance of different spatial transcriptomics deconvolution methods was compared and analyzed primarily from two dimensions: computational time and memory consumption. Several key findings can be seen through the values in (Fig. 4a) and (Fig. 4b): first, among the methods, RCTD and SPOTlight exhibit the highest resource requirements. These two methods not only require longer computation time, but also consume more memory resources. This implies that in practical applications, using these two methods may require computing devices with higher configurations.

Fig. 4.

Runtime and memory consumption. a Memory resource consumption of eight deconvolution tools and Spatialsmooth across 10 datasets. The 10 datasets are as follows: human heart, MOB, mouse brain, mouse hippocampus, human pancreatic ductal adenocarcinoma, seqFISH +, human breast cancer, human lymph node, and two simulated datasets from public literature. b Memory resource consumption of eight deconvolution tools and Spatialsmooth across 10 datasets. The 10 datasets are as follows: human heart, MOB, mouse brain, mouse hippocampus, human pancreatic ductal adenocarcinoma, seqFISH +, human breast cancer, human lymph node, and simulated data from two publicly available literature sources

In contrast, the CARD method demonstrates significant efficiency advantages. It achieves optimal performance in both computation time and memory consumption metrics, indicating that the method achieves better optimization in algorithm design, which enables it to operate efficiently with limited computational resources. This feature makes CARD particularly suitable for use on commonly configured devices, lowering the hardware threshold for researchers. It is worth noting the performance characteristics of the Spatialsmooth method, which we measure in terms of memory consumption and time. As shown in the data for each subgraph in Fig. 4, after applying the parallel strategy, there is no linear relationship between the memory usage and time consumption of Spatialsmooth and the single method. From the results, it can be seen that the parallel time is shorter than the sum of the times of all methods (Supplementary Table 4). This phenomenon primarily stems from several factors: a) Parallel processing eliminates waiting time by enabling sequential execution. When the CPU runs one method, other methods must wait. In contrast, parallel execution allows multiple methods to run simultaneously on different CPU cores without waiting. b) Resource utilization optimization. For I/O-intensive methods, while one method waits for disk/network access, other methods can utilize the CPU. c) Leveraging the advantages of modern CPU architectures. Through hyper-threading technology, physical cores simulate logical cores, and with multi-level caches, different cores can independently access the cache.

The analysis of performance differences is an important guide for choosing an appropriate analysis method. In resource-constrained situations, CARD may be the more practical choice, while Spatialsmooth may be more suitable when comprehensiveness is sought and computational resources are sufficient. This also suggests that we need to strike a proper balance between algorithm performance and computational efficiency when developing new spatial transcriptomics analysis methods.

Validation of the biological significance of spatialsmooth using human thymocytes

Human thymocytes contain cortical, medullary, and medullary cortical boundaries with six cell types: cortical thymic epithelial cells, medullary thymic epithelial cells, vascular smooth muscle cells, fibroblasts, double-positive primitive cells, and double-positive rearranged cells, annotated by histologists based on H&E staining (Fig. 5a). Through deconvolution, located various cell types into different tissue regions (Fig. 5b). To determine the validity of Spatialsmooth in exploring biological significance, the study was first validated by human thymocytes data. It is known that cTEC and mTEC have a negative correlation in human thymocytes through the public information [22], firstly, four deconvolution methods were used to get the inferred cell type compositions at each spatial location, and the compositions of cTEC and mTEC were projected onto the space, and it was found that the effect of Spatialsmooth was more in line with the results of RCTD, the obtained cell type compositions were smoothed according to RCTD, and the results were also projected onto the space (Fig. 5c), and the correlation between the two cell ratios was represented using a scatter plot, and it was found that the cell type composition after Spatialsmooth had a pronounced negative correlation (Fig. 5d). The biological validity of this tool was successfully verified.

Fig. 5.

Analysis of Human Thymocytes data and PDAC co-location. a H&E staining of the Human Thymus. b A spatial scatter pie plot displays inferred cell-type composition on each spatial location from different deconvolution methods. Compared methods include CARD, Seurat, RCTD, SPOTlight, Redeconve, SpatialDecon, and Spatialsmooth. c and d The effectiveness of Spatialsmooth in exploring biological significance was determined by two cell types, cTEC and mTEC, in human thymocytes. e Analysis of co-localization between each cell type of PDAC clearly showed that after Spatilasmooth smoothing some intercellular correlations appeared that could not be obtained by the rest of the methods or were not obvious

Co-localization analysis of pancreatic ductal adenocarcinoma

We used rigorous statistical methods to determine spatial colocalization between cell types, with Pearson's correlation coefficient as a quantitative indicator of spatial colocalization. An adjusted p-value < 0.05 was used as the significance criterion, and methods were applied to correct for multiple comparison problems.

By using our Spatialsmooth smoothing on Human Pancreatic Ductal Adenocarcinomas data the correlation of T cells & NK cells with pDCs, RBCs in the cell ratio matrix was significantly increased (Fig. 5e), which did not appear or was not evident using other methods, an observation that is actually consistent with our knowledge of the immune system and tumor microenvironment. T cells and NK cells, as important effector cells of the immune system, are usually closely functionally related to other immune cells such as pDCs [23]. Spatialsmooth captures this correlation possibly because the method is able to better preserve the cellular spatial distribution's true characteristics and reduce the effect of technical noise. This treatment may have helped reveal patterns of spatial correlation between cells that may have been missed by other methods. In addition, we visualized the spatial cell distribution results for the datasets Human Heart, Mouse Brain, human pancreatic ductal adenocarcinomas, seqFISH +, Mouse hippocampus, Human breast cancer, and Human lymph node used in the paper (Fig. 6a and Fig. S4). To demonstrate that the stronger co-localization analyzed by Spatialsmooth is not purely due to a higher number of cells, we supported true co-localization by calculating biased correlations, and we showed correlations between different cells by CARD, RCTD, AdRoit, SpatialDWLS, and SpatialDecon, and the Spatialsmooth's co-localization analysis after controlling for cell abundance was shown to be consistent with our conclusion (Colocalization of RBCs with T cells & NK cells [24], and pDCs with T cells & NK cells [25]), whereas the co-localization was not found in other deconvolution methods (Fig. 6b).

Fig. 6.

Spatial projection of the used dataset and co-localization controlling for cell abundance. a We projected the spatial cell distribution results of the eight methods used in the text for the datasets used for testing Human Heart, Mouse Brian, human pancreatic ductal adenocarcinomas, seqFISH +, and Mouse hippocampus, as the basis for Spatialsmooth's selection of default parameter choices. b True co-localization was supported by calculating the bias correlation, where the bias correlation coefficients of specific cell type pairs remained significant after controlling for global abundance, demonstrating co-localization of multiple methods, and found to match the co-localization exposed by Spatialsmooth by comparing co-localization controlling for cellular abundance to global co-localization

Single-cell resolution via spatialsmooth enables the identification of pancreatic cancer-clone-specific T cell infiltration

We examined the detailed characterization of tumor-infiltrating T cells according to the Spatialsmooth results, which is important for understanding the tumor immune microenvironment in pancreatic cancer. The results of AdRoit, RCTD and SPOTlight showed that T cells appeared at almost all points (Fig. S2a), which is inconsistent with the nature of PDAC as a cold tumor [26]. Moreover, SpatialDWLS, CARD and Spatialsmooth clearly demonstrated the sparseness of tumor-infiltrating T cells in pancreatic cancer. This is consistent with the spatial distribution of T cell-related genes (Use of CD3 molecules as pan-T cell markers, including CD3E, CD3D, and CD3G, Fig. S2b).

Methods

Preprocessing of datasets

We preprocessed each dataset with the following steps (Supplementary Table 5). (1) Removal of low-quality cells. For scRNA-seq data, we used Seurat with parameters ‘min.features = 200’to remove cells for which fewer than 200 RNAs were captured. (2) Normalization of the expression matrix. (3) Selection of highly variable genes. (4) Cell annotation information was associated with the expression matrix to construct h5seurat objects, ensuring that the spatial topology could be resolved by subsequent tools.

Overview of the spatialsmooth framework

Considering the excellent performance of the Transformer architecture [27] in images in recent years leads us to consider whether the deconvolution results can be optimized using positional encoding in the Transformer strategy so that they conform to the actual distribution? A convolutional autoencoder based approach using positional coding is proposed, which is trained with a neural network through a customized loss function, and finally reconstructs the entire cell scale matrix to smooth the deconvolutional cell type while maintaining spatial consistency. In addition, eight deconvolution methods are integrated in the tool in order to facilitate comparison with the remaining deconvolution methods. The convolutional autoencoder [28] consists of an encoder and a decoder. The encoder applies a series of Convolutional and Maximum Pooling layers to extract high-level features from the input data, and then the decoder uses a Transpose Convolutional layer to upsample and reconstruct the smoothed cell-type proportions. We include a Dropout layer to prevent overfitting and use softmax as an activation function in the final layer to ensure that the output proportions sum to 1 at each spatial location.

Spatialsmooth is a computational framework that integrates multiple deconvolution methods and performs spatial smoothing, and its workflow consists of three key steps: first, it takes single-cell RNA sequencing data as a reference, and uses eight mainstream deconvolution methods (CARD, RCTD, SPOTlight, Seurat, SpatialDWLS, AdRoit, Redeconve, and SpatialDecon) to deconvolve spatial transcriptome data separately to obtain the proportional composition of cell types at each spatial location. Second, the method innovatively combines the positional encoding technique to capture spatial information and designs a dual loss function based on the structural similarity index measure (SSIM) and total variation score (TV), by which spatial smoothing is ensured while maintaining the key features of the original prediction results. Finally, by comprehensively evaluating multiple metrics of correlation, structural similarity, and total variation, Spatialsmooth is able to extract the optimal cell type composition information from the prediction results of multiple deconvolution tools, which maintains the prediction accuracy and ensures the continuity of the spatial distributions and biological rationality. This multi-tool integration and spatial optimization strategy enables Spatialsmooth to produce more accurate cell type distribution predictions than a single deconvolution tool.

Deconvolution analysis

Spatialsmooth integrates CARD, RCTD, SPOTlight, Seurat, SpatialDWLS, AdRoit, and Redeconve, and the deconvolution is based on integrated single-cell and SRT data, and deconvolution analysis is performed to infer the composition of the different cell types in a tissue sample. In Spatialsmooth, gene expression profiles of known cell types obtained from single-cell transcriptome data are used to spatially resolve gene expression profiles by deconvolution from SRT data. Specifically, (spot × celltype) is denoted as a cell type composition matrix that represents the proportion of cell types at each spatial location. The spatial transcriptome data matrix is denoted as (gene × spot) and the cell type-specific score matrix is denoted as (gene × celltype), which can usually be estimated from single-cell data. The three matrices can be unified into a non-negative matrix factorization model (Eq. 1), where is the residual matrix.

1

Spatial positional encoding

In our study, we first collected the cell type composition data inferred by eight mainstream deconvolution methods (CARD, RCTD, SPOTlight, Seurat, SpatialDWLS, AdRoit, Redeconve, and SpatialDecon) and organized these data into the form of a matrix, where the rows represent spatial locations (spots) and the columns represent different cell types. In order to better understand the spatial distribution characteristics of cells in tissues, we introduced the positional encoding technique [29], which was originally derived from the Transformer model and is able to convert two-dimensional spatial position information into continuous numerical vectors [30]. Neighboring location cell types are similar, so the information of location is indispensable, so we also need to encode the location information in one, by combining the location encoding with the cell type proportion data, our model is able to recognize the proportion of cell type composition, each SPOT cell type composition as Token, each spatial localization information as Positional Encoding. Each cell type of each spot is combined with the spatial localization information Positional Encoding to obtain a vector as an input to the encoder. This approach captures the spatial distribution pattern of cells and the relationship between neighboring cells, thus improving the spatial resolution while maintaining the overall scale accuracy.

In Spatialsmooth, positional encoding is an important data processing technique that mathematically transforms the spatial location information of a cell into a numerical representation that can be learned by a model. Specifically, this encoding utilizes the periodicity feature of the sine and cosine functions [31] to convert each spatial position (i.e., x and y coordinates) into a unique set of numerical vectors. In practice, we usually choose 16-dimensional position encoding vectors [32, 33], a dimension that strikes a good balance between computational efficiency and representational power. The combination of the position-encoded vectors and the cell-type data for each spatial location (also referred to as “pixel” or “spot”) allows the model to take into account both spatial position information and cell-type distribution information. This combination allows the model to maintain regional continuity while performing spatial smoothing operations, and to more accurately capture spatial structural features in tissues, leading to a better understanding of the spatial distribution patterns of gene expression. The core advantage of this approach is that it helps the model identify and maintain spatial structure in tissues while considering biological correlations between adjacent regions, ultimately providing more accurate results for spatial transcriptome analysis.

Calculate the positional encoding

Positional encoding uses a series of sine and cosine transformations of frequencies. Assuming the coding dimension is , and the spatial localization information consists of and , the frequency sequence is defined as follows:

2

Let denote the th frequency:

3

The positional encode is:

4

5

Combine the and positional encoding to get the final code:

6

where the positional codes have columns, since 8 sine and 8 cosine codes have been generated for each position ( and ) each.

Network structure

A convolutional autoencoder (CAE) architecture [34] was designed to smooth cell type proportions. The input to the autoencoder is a 4-dimensional tensor (n_samples, n_pixels, n_cell_types, n_features), where n_samples is the number of cell proportion matrices obtained by the deconvolution algorithm, n_pixels is the number of spatial location spot, n_cell_types is the cell type number, and n_features includes cell type proportions and spatial coordinates. By looping, the cell type proportion matrix proportion_matrices[[i]] for each sample are stored in the first feature channel of the x_train.

Then, for each pixel, it copies the jth row of the pos_encoded matrix n_cell_types times, generating a new matrix, which is then stored in the remaining feature channel of x_train. This is done to provide the same positional encoding for each cell type.

We trained the autoencoder using a custom loss function that combines the structural similarity index measure (SSIM) and the total variation (TV) regularization term.TV regularization encourages spatial smoothness by penalizing large differences between adjacent spatial locations. We trained the autoencoder 4000 times with a batch size of 5. An early-stopping strategy was set up to determine the optimal number of training rounds.

Loss functional

In the smoothing treatment of spatial transcriptome data, we use a loss function consisting of two key components to optimize the prediction results. One of them is the total variance regularization term [35], which is responsible for ensuring spatial smoothing, which is achieved by calculating the difference in the probability distributions of cell types between neighboring locations—the higher the difference, the higher the loss value. The other component is the structural similarity index measure(SSIM) [36], which serves to maintain the overall structural similarity between the processed predictions and the original predictions. These two components are balanced by a weighting factor that allows the final prediction to remain spatially smooth without overly deviating from the basic characteristics of the original predicted values. This design allows the algorithm to effectively remove spatial discontinuities while maintaining the fidelity of the data.

The specific steps are: 1). calculate the horizontal difference: calculate the absolute difference between each pixel and its neighboring pixels in the horizontal direction. 2). Calculate vertical variance: Calculate the absolute difference between each pixel and its neighboring pixels in the vertical direction. 3). Calculate the total variance loss: the sum of the mean values of the horizontal and vertical variances. Based on the above steps the loss function can be defined as:

Horizontal variance:

7

Vertical variance:

8

The total variation loss is:

9

This is followed by a combination of structural similarity index and total variance loss.

SSIM aims to measure the structural similarity between two images. Under this paper, it will help to ensure that the overall structure and patterns in the matrix are preserved after smoothing. The specific steps are:

1). Calculate the mean: this will calculate the mean and of the input and prediction matrices respectively. 2). Calculate the variance and covariance: this calculates , and , the variance of the input and prediction matrices and their covariances. 3). Define constants: These are the C1 and C2 values in the SSIM formulae used to stabilize the division.

10

The total loss function is a linear combination of the structural similarity index and the total variance loss in the following steps: the two previously computed loss functions structural similarity index and total variance loss are summed according to a scaling parameter ('lambda'). That is, the combination of the structural similarity index and the total variance loss with an equilibrium parameter :

11

where is a hyperparameter used to balance the effects of the structural similarity index and the loss of total variation error.

The structural similarity index measure (SSIM) is a comparison method that takes into account the relationship between neighboring elements, which evaluates the similarity between matrices by analyzing the local patterns and structural features. Unlike simple element-by-element comparisons, this method better maintains the overall structural features of the data while being tolerant to uniform scaling of the values, which makes it particularly suitable for evaluating the spatial transcriptome data smoothing Effectiveness.

The loss function reasonably combines the structural similarity index and the total variation loss to simultaneously optimize prediction accuracy and spatial smoothing. This is very suitable for spatial data that require smoothing output. However, in practice, the'lambda'parameter may need to be adjusted for optimal performance. With this combination, it can be ensured that the model can generate more accurate and smooth predictions, thus improving the overall performance of the task.

Simulations

Spatial transcriptomics was constructed using single-cell RNA sequencing (scRNA-seq) data from human pancreatic ductal adenocarcinoma, comprising 1,926 single cells and 20 cell types, to obtain the composition of cell types as ground truth. First, common genes between spatial data and single-cell data were identified, and a correlation matrix between spatial spots and single cells was calculated. The number of cells contained in each spot was randomly determined using a Dirichlet distribution, and finally, the most similar single cell was selected for each spot based on the correlation (Fig. 2a).

Spatial comparison metrics

Moran's I is an indicator of spatial autocorrelation. Values closer to 1 indicate a stronger positive correlation, closer to −1 a stronger negative correlation, and closer to 0 a random distribution. Geary's C: another indicator of spatial autocorrelation. A value less than 1 indicates positive correlation, greater than 1 indicates negative correlation, and equal to 1 indicates no correlation. Total Variation: total variation, used to measure the degree of dispersion of the data, in the study indicates the effect of smoothing, the smaller the value the better.

Implementation of methods

In our study, we compared with eight deconvolution (CARD, RCTD, SPOTlight, SpatialDWLS, Seurat, AdRoit, Redeconve and SpatialDecon) methods (Table 1).

Table 1.

Overview of ST deconvolution methods

Method	Inference method	Character	URLs
CARD	Conditional autoregressive model	Explaining the spatial correlation of cell type composition across spatial locations	https://github.com/YingMa0107/CARD
RCTD	Probabilistic, Poisson distribution, maximum likelihood	Supervised learning to decompos e cell type mixture	https://github.com/dmcable/spacexr
SPOTlight	Non-negative matrix factorization (NMF) and non-negative least squares (NNLS)	Adding a priori information to t he model and initializing the ba se and coefficient matrices with cell type marker genes greatly i mproves sensitivity and robustne ss	https://github.com/MarcElosua/SPOTlight
SpatialDWLS	Dampened weighted least squares (DWLS)	Includes an additional filtering s tep to remove irrelevant cell typ es for enhanced specificity	https://rubd.github.io/Giotto_site/articles/tut7_giotto_enrichment.html
AdRoit	Probabilistic, nonnegative least squares regression	Sensitive to gene expression bat ch effects and reliant on high c overage SC reference data	https://github.com/TaoYangdev/AdRoit
Seurat	Analysis pipelines with integrated algorithms	Assign cells from scRNA-seq to spatial location in histological section	https://github.com/satijalab/seurat
Redeconve	Regularization, quadratic programming solution	The ST data is more informative after performing a fill and helps to resolve the cellular state of a particular tissue	https://github.com/ZxZhou4150/Redeconve
SpatialDecon	log-normal regression and modelling background	Give context to gene-level result s, resolving whether a gene’s ex pression pattern reflects different ial expression within a cell type or merely differences in cell-ty pe abundance	https://github.com/Nanostring-Biostats/SpatialDecon.git

CARD [4]: we used the code of CARD v1.0.0 from https://github.com/YingMa0107/CARD. We set minCountGene to 5 and minCountSpot to 5, which are the default parameter settings.

RCTD [5]: we used the code of RCTD from https://github.com/dmcable/spacexr, which is integrated into a tool called spacexr (2.0.0). Spacexr (RCTD) was run with following the configuration: (1) create.RCTD was used with the parameter CELL_MIN_INSTANCE = 1; (2) run.RCTD was used in the doublet mode.

SPOTlight [6]: we used the code of SPOTlight v0.99.0 from https://github.com/MarcElosua/SPOTlight. We set cl_n to 10 and hvg to 2000.

SpatialDWLS [7]: We followed the guidelines on the SpatialDWLS website: https://rubd.github.io/Giotto_site/articles/tut7_giotto_enrichment.html. We set the parameter as n_cell = 20.

Seurat [17]:We followed the instructions on the Seurat 3.2 website: https://satijalab.org/seurat/archive/v3.2/integration.html. We set the parameter dim = 1:30, normalization.method = ‘SCT’.

AdRoit [18]: We followed the instructions on the AdRoit 0.0.2 website: https://github.com/lijxug/AdRoit. We set the parameter nbootsids = 5, minbootsize = 50.

Redeconve [19]: We followed the instructions on the Redeconve 1.1.2 website: https://github.com/ZxZhou4150/Redeconve. We set the parameter genemode ="def", hpmode ="auto", dopar = T, ncores = 8.

SpatialDecon [20]: we used the code of SpatialDecon from https://github.com/Nanostring-Biostats/SpatialDecon.git. SpatialDecon was run with the expected background count bg set to 0.01.

Discussion

Our convolutional autoencoder-based method effectively smooths the deconvolution of cell type composition while maintaining spatial consistency. By utilizing spatial information and cell type composition derived from multiple deconvolution algorithms, our method generates more biologically consistent cell type distributions. Incorporating a total variation regularity term in the loss function improves spatial smoothness and reduces noise and local inconsistency.

The smoothed cell type proportions obtained by our method more reliably reflect the spatial distribution of cell types within a tissue. This information is valuable for understanding the spatial organization and interactions of different cell types in the context of tissue function and disease.

Our method has several advantages over existing smoothing methods. First, it exploits the spatial information inherent in the spatial transcriptome, allowing convolutional autoencoders to learn and utilize spatial patterns. Second, by combining the proportions of multiple deconvolution algorithms, our method captures consensus information and reduces the impact of algorithm-specific biases. Third, the auto-encoder's convolutional architecture is well suited for processing spatial data, effectively capturing local patterns and dependencies. Fourth, the fact that Spatialsmooth does not rely on a priori knowledge of the biology of a particular species, but only requires the cell type matrix after deconvolution, as well as the applicability of the deconvolution results to different species highlights the potential of Spatialsmooth as a general-purpose tool.

Because of the potential risk of decoupling the loss function optimization objective from the deconvolution accuracy assessment, we discuss the design intent of the loss function from the following three aspects, (1) TV Loss: suppresses spatial discontinuities induced by technological noise by constraining the expression gradient between neighboring spots, which is essentially an explicit embedding of the spatial smoothing prior, and SSIM Loss: enforces that the reconstruction result and the input data are globally similar in structure to avoid feature loss caused by excessive smoothing. (2) Potential differences between training objectives and accuracy metrics. Difference in optimization direction: optimization of TV/SSIM may be at the expense of local specificity (e.g., weakening signals of rare cell types), while accuracy metrics (e.g., MSE, JSD) require global consistency with the ground truth; Difference in assessment dimensions: TV/SSIM reflects the regularization effect of the method itself, while accuracy metrics need to rely on independent experimental validation. (3) Validation of ablation experiments (Supplementary Table 7). combination of SSIM + Total Variation (TV) Loss: performance enhancement of 12.5%, 17.1% and 19.04% in correlation, SSIM and Total Variation Score metrics respectively compared to a single SSIM Loss; single TV Loss: prone to smoothing failure due to ignoring spatial structure; MSE Loss: Unable to reconstruct effective cellular landscapes, verifying the necessity of structure-aware loss. It shows that the TV term is irreplaceable for the preservation of real structure.

However, our method has some limitations. The choice of different deconvolution results affects the smoothing performance. To obtain the best results on different datasets, it may be necessary to adjust the input combination of the neural network. In addition, the method assumes that the spatial location of the position is available. In cases where spatial information is not available, the input data and autoencoder architecture may need to be modified.

Future work could explore the integration of other features such as gene expression data or histological images to further improve smoothing performance. Incorporating a priori knowledge about the spatial organization of cell types or using domain-specific constraints could also improve the biological relevance of the smoothing ratio. In addition, extending our approach to deal with three-dimensional spatial data (e.g., data obtained from tissue removal techniques) could provide a more comprehensive understanding of the spatial distribution of cell types in complex tissues.

Conclusion

In this study, we propose to not only integrate eight deconvolution methods and use a parallel strategy to be able to make each method independently runnable; in addition to proposing a convolutional autoencoder-based approach that utilizes a method based on the encoding of the spatial positional of the spot, where the location information of each spot can be captured, and the encoding is unique for the different locations, for smoothing out the deconvolutional cell-type proportion, the while maintaining spatial consistency. Our method fully utilizes spatial information and proportion from multiple deconvolution algorithms to generate smooth, biologically plausible cell type distributions. The smoothed proportion obtained by our method more reliably reflect the spatial organization and interactions of different cell types within a tissue. We believe that the method proposed in the study is expected to enhance the interpretation and analysis of spatial transcriptomics data and contribute to a deeper understanding of tissue heterogeneity and function.

Abbreviations

CAE

Convolution Auto-Encoders

SSIM

Structure Similarity Index Measure

TV

Total Variation

JSD

Jensen-Shannon Divergence

MSE

Mean Square Error

PCC

Pearson Correlation Coefficient

Authors’ contributions

X.Y. conceived and designed the experiments, performed the experiments, analyzed the data, prepared figures and/or tables, authored or reviewed drafts of the article, and approved the final draft. J.J.X. performed the experiments, analyzed the data, prepared figures and/or tables, and approved the final draft. Y.B.F analyzed the data, prepared figures and/or tables, and approved the final draft. Y.F.Z. analyzed the data, prepared figures and/or tables, and approved the final draft. H.M.L. conceived and designed the experiments, authored or reviewed drafts of the article, and approved the final draft. X.Z.C. conceived and designed the experiments, authored or reviewed drafts of the article, and approved the final draft.

Funding

This research was funded by the National Natural Science Foundation of China of XZC, grant number 31460297. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Data availability

Details of each dataset are given in Supplementary Table 1. The gridded human heart data andannotated scRNA-seq reference are publicly available at (https://github.com/JiawenChenn/St-review/tree/main/processed_data/heart/ISS). Mouse olfactory bulb from (https://github.com/JiawenChenn/St-review/tree/main/processed_data/MOB). Mouse Brain: scRNA-seq and ST data. For ST data,we only used the frontal cortex region for this study. Can download the scRNA-seq data here: (https://www.dropbox.com/s/ruseq3necn176c7/brain_sc.rds?dl=0). Can download the ST data here:(https://www.dropbox.com/s/azjysbt7lbpmbew/brain_st_cortex.rds?dl=0). Mouse hippocampus Slide-seqV2 dataset (https://singlecell.broadinstitute.org/single_cell/study/SCP948/robust-decomposition-of-cell-type-mixtures-in-spatial-transcriptomics). seqFISH + : scRNA-seq and spatial transcriptomics data were both obtained from (https://github.com/CaiGroup/seqFISH-PLUS). The processed human breast cancer Visium data are available at zenodo (https://zenodo.org/record/4739739#.Ys0v6jdBy3D). The processed human breast cancer scRNA-seq data used in this study are available in the Gene Expression Omnibus database under accession code GSE176078. The processed human lymph nodes Visium data are available at 10 × Genomics website [https://www.10xgenomics.com/resources/datasets/human-lymph-node-1-standard-1–1-0]. The processed human lymph nodes scRNA-seq data are available from Kleshchevnikov et al. (https://cell2location.cog.sanger.ac.uk/browser.html). Simulations: simulation dataset from CARD (https://github.com/YMa-lab/CARD-Analysis/tree/master). Gridded mouse visual cortex spatial transcriptomic data, annotated scRNA-seq reference, and the results of other algorithms on this dataset are provided by the platform at (https://github.com/QuKunLab/SpatialBenchmarking/tree/main/FigureData/Fig.4/Dataset10_STARmap).

Declarations

Ethics approval and consent to participate

No applicable.

Consent for publication

No applicable.

Competing interests

The authors declare no competing interests.