Randomized Spatial PCA (RASP): a computationally efficient method for dimensionality reduction of high-resolution spatial transcriptomics data

Abstract

Spatial transcriptomics (ST) provides critical insights into the spatial organization of gene expression, enabling researchers to unravel the intricate relationship between cellular environments and biological function. Identifying spatial domains within tissues is key to understanding tissue architecture and mechanisms underlying development and disease progression. Here, we present Randomized Spatial PCA (RASP), a novel spatially-aware dimensionality reduction method for ST data. RASP is designed to be orders-of-magnitude faster than existing techniques, scale to datasets with 100, 000+ locations, support flexible integration of non-transcriptomic covariates, and reconstruct de-noised, spatially-smoothed gene expression values. It employs a randomized two-stage PCA framework with sparse matrix operations and configurable spatial smoothing. RASP was compared to BASS, GraphST, SEDR, SpatialPCA, and STAGATE using diverse ST datasets (10x Visium, Stereo-Seq, MERFISH, 10x Xenium) on human and mouse tissues. RASP demonstrates comparable or superior tissue domain detection with substantial improvements in computational speed, enhancing exploration of high-resolution subcellular datasets.

1. Introduction

Spatial transcriptomics (ST) assays measure gene expression at thousands to hundreds-of-thousands of spatial locations within tissue sections, enabling researchers to characterize transcript abundance in vivo at resolutions ranging from small cell groups to subcellular structures [1, 2]. ST technology facilitates exploration of gene expression’s spatial organization, providing insights into cellular microenvironments and functional states of individual cells [3]. Two primary types of ST technologies exist: high-throughput sequencing technologies (sST) and single-molecule fluorescent in situ hybridization or imaging-based technologies (iST) [4, 5]. High-throughput sequencing technologies, such as 10x Genomics’ Visium, capture mRNA transcripts at multiple sites or spots on a slide. The original 10x Visium platform offers a resolution of 55 μm, with each spot containing between 1 – 10 cells [6]. Emerging technologies, such as Visium HD, promise higher resolution of 2 μm, approaching single-cell levels, while alternatives like Slide-seq V2 capture transcripts with a resolution of 10 μm, also nearing single-cell resolution [7, 8]. iST technologies, which reach single cell resolution, include platforms such as MERFISH/MERSCOPE (Vizgen), seqFISH, osmFISH, COsMx, and 10x Xenium [9-13]. A notable advancement in recent years has been the introduction of iST platforms that support the use of formalin-fixed, paraffin-embedded (FFPE) tissue samples, allowing for use of archival tissue banks, increasing the use of iST technologies [9, 12, 13]. Additionally, probe sets used in iST technologies have expanded, with systems like 10x Xenium developing probe sets with up to 5,000 individual probes [14], helping to bridge the trade-off between sequencing depth and resolution, allowing for more comprehensive profiling of gene expression. As costs decrease and technology advances, ST studies will transition from analyzing small sample sizes to extensive investigations [2, 13, 15-19], generating multi-slide, multi-timepoint ST datasets covering millions of cells across tissues [2]. Larger datasets necessitate development of novel computational tools that leverage both spatial and transcriptomic data efficiently.

ST’s rich data allows tissue investigation at multiple biological scales, and computational approaches must adapt. On a cellular level, cell type assignment facilitates spatial distribution visualization, enhancing understanding of cellular structure and function in native tissue [20, 21]. At larger scales, identifying tissue domains aids comprehension of complex tissue microenvironments. Identifying cell types and spatial domains is achievable with recently developed computational tools. Popular single-cell RNA sequencing (scRNA-seq) tools like Seurat and Scanpy perform principal component analysis (PCA) and clustering on gene expression data while disregarding spatial context [22, 23]. However, methods like BASS and FICT now incorporate spatial context [24, 25]. For spatial domain detection, methodologies like BASS, SEDR, GraphST, STAGATE, and SpatialPCA exist [24, 26-29], utilizing different model architectures to embed spatial and gene expression data into low-dimensionality spaces for clustering and domain identification. ST data is often sparse, so de-noising and imputing poorly captured transcripts is a desirable feature, and many spatially-aware methods provide this functionality [30-33].

Yet, current methods have notable limitations. While effective on a single ST slice with few profiled locations, they are computationally intensive, exhibiting slow processing locally, particularly on iST datasets exceeding 100, 000 cells. With iST dataset generation increasing, efficient, spatially-aware dimensionality reduction algorithms are necessary. Many existing methods are also inflexible and complex, hindering customization of spatial smoothing or integration of non-transcriptomic covariates, such as those derived from hematoxylin and eosin (H&E) images or protein abundance metrics. Addressing these limitations is crucial for deriving meaningful insights from large-scale ST datasets.

In this paper, we present Randomized Spatial PCA (RASP), a computational method designed to be much faster than existing techniques, scalable to ST data with hundreds-of-thousands of locations, and supportive of non-transcriptomic covariates, while remaining easy to understand and implement. RASP employs randomized linear algebra techniques for efficient dimensionality reduction of large datasets [34]. It performs spatial smoothing using the coordinates returned from the ST platform and supports optional integration of covariates like cellular density or sequencing depth, providing greater analytical flexibility. Tested on four real-world and two simulated datasets (including 10x Genomics Visium and subcellular imaging technologies like Stereo-Seq, MERFISH, and 10x Xenium), RASP demonstrates tissue domain detection performance comparable to or superior to existing methods, with orders-of-magnitude faster runtimes.

2. Results

2.1. RASP overview

RASP performs dimensionality reduction and spatial smoothing on normalized ST data, with the option to incorporate location-specific covariates. The randomized two-stage PCA optimizes computational efficiency while enabling the flexible and efficient analysis of high-resolution ST datasets. RASP utilizes a sparse matrix representation of the data, conducts dimensionality reduction via randomized PCA, and generates spatially smoothed principal components (PCs) using an inverse distance matrix sparsified by a k-nearest neighbors (kNN) threshold (Fig. 1A). Covariates integration into the latent variables is achieved through an optional second-stage randomized PCA on a matrix with covariates appended to first-stage smoothed PCs. RASP’s spatially smoothed PCs are useful for a variety of downstream analyses, including clustering and reduced rank reconstruction of gene-level expression. See Algorithm 1 and Section 4.1 for details. Note that by default, RASP utilizes 20 PCs, although this is a user tunable parameter. See Extended data Fig 15. for effects of variable PC number on RASP performance.

Figure 1: RASP overview.

A: Method workflow. RASP takes as input a cell-by-gene expression matrix and spatial coordinates from a ST experiment. The expression matrix is reduced via randomized PCA and spatially smoothed by a sparse inverse distance matrix. Non-transcriptomic covariates can be added at this stage (not pictured). The output of RASP can be used for downsteam analyses including cell type annotation and spatial domain identification. B: Evaluation pipeline. RASP was tested on four ST datasets each generated using a different platform (MERFISH, Stereo-seq, Xenium, Visium) and two synthetic datasets (100 replicates each). RASP results were compared against standard PCA and alternative spatially aware dimensionality reduction models (SEDR, GraphST, SpatialPCA, BASS, and STAGATE) for both accuracy and speed.

2.2. RASP evaluation on real data

We applied RASP to four publicly available ST datasets generated using diverse techniques and resolutions on human and mouse tissues including the mouse ovary (Vizgen MERFISH technology), mouse olfactory bulb (STOmics Stereo-seq technology), human breast cancer tumor (10x Genomics, Xenium technology), and human DLPFC (10x Genomics, Visium technology)(Fig. 1B) and SI Table 1. For details regarding field standard preprocessing steps used, such as filtering criteria and normalization, please refer to Sections 4.4.1-4.4.3, and SI Sections 1.2-1.3.

We evaluated RASP by clustering locations using the spatially smoothed principal components (PCs) output by the algorithm. Cluster quality was assessed through spatial continuity and compactness via CHAOS score (SI 1.3), spatial autocorrelation by Moran’s I (SI 1.4) and the Adjusted Rand Index (ARI) (SI 1.2) compared to ground truth annotations. We conducted a sensitivity analysis on key RASP parameters, focusing on the kNN distance threshold for sparse matrix construction and the β power parameter (see Figure 1A). This analysis used a one-parameter-at-a-time strategy to investigate parameter impacts on clustering outcomes.

Performance comparisons were made between RASP and five other spatially informed dimensionality reduction or spatial domain detection methods: BASS [24], GraphST [26], SEDR [27], SpatialPCA [28], and STAGATE [29] as well as performing standard PCA. For each comparison, we adhered to the respective GitHub tutorials to run these methods for spatial domain and cell type detection. Clustering of latent dimensions found by each method was performed using Mclust [35], Louvain [36], Leiden [37], and Walktrap [38] algorithms, where appropriate. The size of the human breast cancer dataset posed significant computational challenges, requiring allocations of arrays exceeding 1TB, leading to unsuccessful runs for some methods.

2.3. RASP identifies biologically relevant structures in heterogeneous mouse ovary MERFISH data

We selected this MERFISH dataset for its complex tissue structure and cellular resolution, featuring spatially distinct and heterogeneous cell types. Obtained from Huang et al., [15] it includes ground truth cell type annotations based on histology and differential gene expression (Fig. 2A). Annotations include luteal, luteinizing mural, granulosa, theca, and stromal cells, with clear spatial organization, alongside endothelial, epithelial, and immune cells, which are heterogeneously distributed across the tissue [39], making automatic annotation challenging.

Figure 2: Mouse ovary analysis (Vizgen MERFISH).

A: Ground truth cell type annotations (top left) and corresponding predictions for RASP, normal PCA, and other methods B: Quantification of runtime for all methods compared to normal PCA. C: Quantification of ARI for all methods, different colors indicate the clustering algorithm used to assign labels. The interquartile range and median ARI values at default RASP parameters are shown (kNN = 3-10, β = 2), as well as the maximum ARI value achieved by RASP. D: ARI values for RASP with inverse distance raised to the β of 0 (left), 0.5 (middle), and 2 (right), plotted against kNN. Colors indicate clustering algorithm. E: Normalized Lhcgr expression (left), reduced rank reconstructed Lhcgr expression (center), and spatially smoothed reduced rank reconstructed Lhcgr expression (right).

For this dataset, RASP using Louvain clustering achieved the highest ARI of 0.69, followed by standard PCA (ARI of 0.58) and SEDR (ARI of 0.57) (see Fig. 2A and 2C). We employed three different clustering methods to assign labels to cells in the dataset. The Louvain and Leiden algorithms performed best, while Mclust the worst. Note that BASS utilizes Mclust as a prior in its Bayesian framework but does not support a user-defined clustering algorithm.

Relative runtimes show RASP is one to three orders-of-magnitude faster than competing methods on this dataset (Fig. 2B). SEDR is an order-of-magnitude slower than RASP, while BASS is almost four orders-of-magnitude slower. RASP performance diminishes with increasing kNN thresholds for PC smoothing; highlighted by decreasing ARI scores at larger kNN values (Fig. 2D). This result is not unexpected for cell type annotation, but it is noteworthy that the local smoothing employed by RASP in this task, essentially borrowing information from just a few neighboring cells, has a marked increase in performance over normal PCA for clustering-based label assignment. We found β = 2 the best performing inverse distance power value. At higher β kNN threshold’s impact on ARI was blunted, with comparable performance across the entire range of kNN thresholds sampled. Conversely, at β = 0, the kNN threshold dramatically affected RASP performance (Fig. 2D, Extended Data Fig.1).

Using RASP for reduced rank reconstruction of gene expression profiles, we analyzed the spatial signature of the luteinizing hormone/chorionic gonadotropin receptor (Lhcgr) given its essential role in ovarian biology and ovulation (Fig. 2E). Lhcgr is a G-protein coupled receptor that binds luteinizing hormone and chorionic gonadotropins, is expressed on granulosa cells, theca cells, and luteal cells [40-42]. Originally, gene expression showed minimal spatial patterning (Fig. 2E, left plot). After RASP reconstruction, expression is localized to luteinizing mural and luteal cells (Fig. 2E, middle and right plots). For this example, we utilized the distances to only 3 kNN, which results in very similar signatures between normal PCA-based reduced rank reconstruction and RASP.

2.4. Incorporation of covariates improves RASP annotation of mouse ovary ST data

A key feature of RASP is support for integrating non-transcriptomic covariates into reduced dimensions. The two-stage PCA approach used by RASP offers benefits relative to a single PCA on merged transcriptomic and non-transcriptomic variables: It prevents covariate signal dilution within the high-dimensional transcriptomic data, and decouples spatial smoothing of expression data from covariates. To RASP covariate integration, we explored three covariates on analysis of the mouse ovary data: local cell density, transcriptomic library size, and cell volume. For each covariate, we performed parameter sweeps over the kNN parameter using three RASP configurations: 1) standard RASP without covariates (i.e., the configuration used to generate the results in Figure 2) 2), single-stage RASP executed on the merge of ST expression data and covariates, and 3) two-stage RASP with the covariates added to the output from the first stage (see Fig. 3A (bottom)). For these experiments, the β parameter was held constant at β = 2,the optimal value for this dataset (see Fig. 2D, and Extended Data Fig 1). For local density, spatial smoothing was applied to the PC matrix, but not the local density variable, as local density inherently contains spatial information. Library size and cell volume covariates where smoothed by the same inverse distance weighting as the PCs matrix, prior the second round of dimensionality reduction.

Figure 3: Covariate analysis (Mouse ovary dataset).

A: Schematic depicting RASP model architecture with no covariates (top left), single-stage RASP architecture with an additional covariate (top right), and two-stage RASP model architecture with an additional covariate. B: ARI values for no covariate, one-stage RASP, and two-stage RASP plotted against kNN. Columns show different clustering algorithms, and rows capture the type of covariate: local cell density (top), cell library size (middle), and cell volume (bottom).

All three covariates impacted ARI values, with clustering algorithm determining the magnitude and direction of the ARI change. The ARI using Mclust improved regardless of covariate added, with single-stage RASP equal to or outperforming the two-stage version in most cases (see Fig. 3B (left column)). Mclust ARI without covariates was 0.39 improving to 0.45 and 0.5 with covariates. ARI with Leiden was highest for two-stage RASP in most cases, while the single-stage version was covariate dependent, with the addition of local density decreasing ARI values, and addition of library size and cell volume having mixed effects (see Fig. 3B (right column)). Notably, Leiden clustering with cell volume achieved a top ARI of 0.71, the highest attained for this dataset. Louvain-classification was hindered by local density additions, with ARI values dropping across kNN values, while affects of library size and cell volume fluctuated with kNN value. Overall, these results indicate that integration of covariates can enhance RASP performance, but covariate selection, clustering algorithm, spatial smoothing require careful consideration. Importantly, two-stage RASP avoids covariate dilution offering more control over spatial smoothing.

2.5. RASP shows comparable performance to alternative methods for cortical layer identifi-cation using human DLPFC Visium data

The dataset utilized for this analysis was generated using the 10x Visium platform by the LIBD human Dorsolateral Prefrontal Cortex (DLPFC) project [43] and includes manual cortical layer annotations. This dataset serves as a gold standard benchmark for spatial domain detection algorithms due to its clear neuroanatomical boundaries providing reliable ground truth and a small number of profiled tissue locations facilitating evaluation of computationally complex algorithms. For benchmarking, we selected slice #151,673 from the LIBD dataset, which consists of seven spatial domains including six cortical layers and white matter [43].

For this dataset, RASP and SpatialPCA emerged as the best-performers, achieving ARIs of 0.62 and 0.61 respectively. SEDR and BASS followed closely, with ARIs of 0.60, while GraphST scored of 0.57 (Fig. 4A and 4D). Importantly, RASP has comparable computational cost to standard PCA but with substantially higher ARI. RASP was one to three orders-of-magnitude faster than other methods, highlighting its efficiency and accuracy (Fig. 4B). For optimal RASP performance, a kNN ranging from 3 – 10 was effective; larger kNN thresholds reduced scores, as they smooth across cortical layer boundaries, creating overly large domain identification (Fig. 4D). Interestingly,all β values generated similar ARI vs. kNN threshold trends. The Mclust algorithm was relatively invariant to both the β and kNN threshold parameters, showing consistent performance across parameter space. Louvain and Leiden algorithms dropped in performance as the kNN threshold increased, regardless of the β value, with β = 2 performing marginally better (Fig. 4, Extended Data Fig. 3A).

Figure 4: Human DLPFC analysis (10X Visium).

A: Ground truth cortical layer annotations (left) and corresponding spatial domains identified by RASP, PCA, and competing methods. B: Quantification of runtime for all methods compared to normal PCA. C: Quantification of ARI for all methods, different colors indicate the clustering algorithm used to assign labels. Interquartile range and median ARI values at default RASP parameters (kNN =3-10, β = 2) is shown, along with maximum ARI values achieved by RASP. D: ARI values for RASP with inverse distance raised to the β of 0 (left), 1 (middle), and 2 (right), plotted against kNN. Colors indicate clustering algorithm. E: Normalize TMSB10 expression (left), reduced rank reconstructed TMSB10 expression (center), and spatially smoothed reduced rank reconstructed TMSB10 expression (right) plotted on tissue.

To demonstrate the use of RASP for reduced rank reconstruction, the TMSB10 (thymosin beta 10) was reconstructed [40] (Fig. 4E). The spatially smoothed reconstruction of TMSB10 is isolated to layer 4 with noise substantially relative to normalized expression and non-smoothed reconstruction, highlighting RASP’s utility for spatially informed gene-level analyses.

2.6. RASP effectively characterizes tissue structures in mouse olfactory bulb Stereo-seq data

The mouse olfactory bulb analysis used a dataset generated with Stereo-seq technology, capturing subcellular tissue resolution, providing a valuable benchmark. The olfactory bulb consists of distinct cortical layers: meninges, olfactory nerve layer (ONL), glomerular layer (GL), external plexifrom layer (EPL), mitral cell layer (MCL), internal plexiform layer (IPL), granule cell layer (GCL), and the rostral migratory stream (RMS) [44, 45]. RASP, SEDR, and GraphST produced clusters that effectively align with olfactory bulb layers (Fig. 2A). In contrast, STAGATE, BASS, standard PCA, and SpatialPCA were unable to characterize tissue structures in the core of the olfactory bulb.

RASP demonstrated computational costs comparable to standard PCA and two to three orders-of-magnitude faster than the other methods (Fig. 5B). Due to lack of ground truth annotations cluster quality was assessed using Moran’s I and CHAOS score (Fig. 5C,D, SI Algorithm 7,8). RASP exhibited the highest Moran’s I and lowest CHAOS scores among evaluated methods, with Louvain clustering producing favorable outcomes. Analysis of different kNN thresholds revealed increased kNN values resulted in higher Moran’s I values and lower CHAOS scores, though beyond a kNN of 50, clusters deviated from anatomical layers, producing large homogeneous structures that, while rewarded by the CHAOS score, lack biological plausibility (Fig. 5D). RASP achieved the highest Moran’s I value and lowest CHAOS score with low β values ranging from 0 – 0.5. As the β parameter increased past this point, Moran’s I decreased and the CHAOS score increased, indicating less optimal clustering solutions (Fig. 5, Extended Data Fig.5).

Figure 5: Mouse olfactory bulb analysis (STOmics Stereo-seq).

A: Cortical laminar structure (top left) and corresponding spatial domains identified by RASP, PCA, and other methods. B: Quantification of runtime for all methods compared to normal PCA. C: Quantification of Moran’s I score (top), and CHAOS score (bottom) for all methods, different colors indicate the clustering algorithm used to assign labels. Interquartile range and median values shown for kNN between 50-100, and β = 0, along with maximum and minimum scores achived by RASP. D: Moran’s I score (top) and CHAOS score (bottom) with inverse distance raised to the β of 0 (left), 1 (middle), and 2 (right), plotted against kNN. Colors indicate clustering algorithm. E: Normalize Doc2g expression (left), reduced rank reconstructed Doc2g expression (center), and spatially smoothed reduced rank reconstructed Doc2g expression (right).

Reduced rank reconstruction performance was explored using the Doc2g gene, which acts upstream of neuro-transmitter secretion and locates in membrane and presynapses of neurons [46, 47]. Within the olfactory bulb, Doc2g is localized to the EPL and IPL. While normalized expression of Doc2g in the original dataset appears sparse and noisy, RASP’s spatially smoothed expression is clearly localized to the EPL and IPL, showing greater spatial organization than standard PCA reconstruction 5E).

2.7. RASP efficiently scales to large human breast cancer Xenium data while maintaining accurate tissue domain detection

To assess RASP on large, high-resolution data, we downloaded the 10x Xenium human breast cancer dataset and annotations from the R package SubcellularSpatialData [48]. This dataset originates from a publicly available 10x Genomics dataset of re-sectioned FFPE human breast tissue infiltrating ductal carcinoma in situ. Annotations are based on H&E stained tissue accompanying the Xenium data. Due to it’s size, over half a million cells, competing domain detection methods encounter out-of-memory errors, requiring over 1 TB of RAM- unrealistic for many researchers, even with institutional high-performance computing environments. Given these limitations, we compared RASP against standard PCA. RASP substantially outscored PCA (ARI 0.63 vs ARI 0.25 ) (Fig. 6A,B).

Figure 6: Human breast cancer analysis (10x Xenium).

A: Ground truth spatial domain annotations (top left) and corresponding predictions for RASP, and normal PCA. B: Quantification of ARI for all methods, different colors indicate the clustering algorithm used to assign labels. Interquartile range and median ARI scores at default RASP parameters (kNN = 50-100, β = 0.0.25, 0.5), and maximum ARI values achieve by RASP. C: Normalize SFRP1 and LUM expression (left column), reduced rank reconstructed SFRP1 and LUM expression (center column), and spatially smoothed reduced rank reconstructed SFRP1 and LUM expression (right column) plotted on tissue. D: ARI values for RASP plotted against smoothing distance (top) and kNN (right). Colors indicate clustering algorithm.

Optimal kNN smoothing thresholds for RASP revealed a larger kNN threshold range of 50 – 100 is needed to identify the tissue-spanning domains in this dataset (Fig.6D). For this dataset, Leiden and Louvain clustering were optimal, followed by Mclust. Consider the computational burden of clustering algorithms, especially given the lengthy runtimes associated with large datasets. Mclust can be three to four times as more computationally expensive as Louvain or Leiden, but if testing multiple resolutions to achieve a desired number of clusters, the three algorithms are comparable. Similar to the olfactory bulb dataset, smaller β values (0 – 0.5) produce better results, with ARI values dropping considerably for all clustering algorithms and all kNN thresholds at higher β values (Fig. 6D, Extended Data Fig. 6A).

To explore RASP’s reduced rank reconstruction feature, we compared normalized and reconstructed expression signatures for two genes: SFRP1 and LUM. SFRP1, part of the secreted frizzled-related protein family, and modulating Wnt signaling, is dysregulated in several human tumor types [40, 49-51]. RASP’s spatially smoothed reduced rank reconstruction of SFRP1 shows expression in normal ducts and stromal cells, highlighting areas where the normalized expression was very sparse (Fig. 6C). LUM, a small proteoglycan family protein, binds collagen, and exhibits pathway dysregulation in many cancers [40, 52, 53]. Reduced rank reconstruction of LUM indicates high expression in both stromal and tumorous regions of the tissue.

2.8. Simulation analysis

In addition to evaluating RASP on real data, we applied RASP and comparison techniques to simulated ST datasets generated using the R package SRTsim[54] for two different tissue models with 100 replicates each. The first model, the Stripes model, represents a simple tissue with a laminar structure of eight tissue domains (see Extended Data Fig. 8A). The second model, the Dots model, represents a complex tissue with heterogeneous circular regions scattered across the tissue, and a background cell type (Extended Data Fig. 11A).

RASP outperformed all other methods in computational speed and accuracy using default parameters. For the Stripes model, a larger kNN range of 15 – 50 is needed to correctly identify the laminar tissue-spanning domains (Extended Data Fig. 8D). For clustering Stripes data, Leiden and Louvain were found to be optimal, followed by Walktrap and Mclust. Similar to the olfactory bulb and breast cancer datasets, smaller β values (0 – 0.5) produced better results, with ARI values dropping for all clustering algorithms and kNN thresholds at higher β values (Extended Data Fig. 8D).

For the Dots model, a smaller kNN range of 2 – 20 is needed to identify the regions in this dataset (Extended Data Fig. 11D). For clustering Dots data, Walktrap was optimal, followed by Mclust, Leiden and Louvain. Similar to the Ovary dataset, larger β values between 1.5 – 2.0 were found to produce better results, with ARI values dropping considerably for all clustering algorithms at lower β values (Extended Data Fig. 11D). Similar to the Stripes model, RASP outperformed all other methods in speed and produced accurate label predictions across its default parameters.

2.9. PC number impacts RASP performance

For the real ST analyses detailed in Sections 2.3-2.7, we computed 20 spatially-smoothed PCs using RASP and performed location-based clustering on the components. Determining the optimal rank for dimensionality reduction is a long-standing problem in statistical analysis and ongoing topic of debate within the bioinformatics community for scRNA-seq and ST data. In practice, analyses typically use a default of between 20 and 50 PCs. Tools like Scanpy and Seurat default to retaining 30 and 50 PCs respectively, serving as a useful benchmarks.

To evaluate robustness, we ran RASP using the best-performing parameters across the DLPFC, mouse ovary, and breast cancer datasets while retaining 100 PCs. We iteratively clustered each dataset using an increase number of retained PCs, from 5 – 60, to understand the correlation between PC count and cluster label accuracy (Extened Data Fig.8A). For the DLPFC dataset, stable ARI values were observed with increasing PC number for the Louvain, Leiden, and Mclust clustering methods, and an increase for Walktrap. This invariance suggests these clustering models describe data distribution well regardless of dimensions. Conversely, in the mouse ovary dataset, we noted an upward ARI trend with more PCs for the Louvain and Leiden clustering methods. This behavior is consistent with the ground truth annotations of cell types present in this tissue, as higher order PCs are adept at capturing the nuanced details required for accurate cell typing (Extended Data Fig.15A, middle). Interestingly, Mclust improved at the lowest PC count, likely because Mclust assumes clusters have consistent size and shape, which isn’t the case here. In the breast cancer dataset, Louvain and Leiden algorithms showed increased ARIs up to 30 PCs, and then values decreased, while Mclust ARIs decreased as the number of PCs increased. This reduction in classification accuracy aligns with the nature of the classification task in this dataset, which deals with more extensive spatial domains rather than finer cellular granularity, thereby indicating that broader spatial domains might be more appropriately captured with fewer PCs (Extended Data Fig.15A, right).

2.10. Influence of β on inverse distance weighting

To examine the effects of varying the parameter β on inverse distance weighting, we computed a sparse distance matrix representing the spatial relationships among data points. Subsequently, we applied an inverse weighting transformation to this distance matrix, raising it to different powers of ft. The relationship between distance (um) and the corresponding weights, calculated at β values between 0 and 2, is visualized (Extended Data Fig. 16,17). A β value of 0 results in a relatively uniform weighting across distances, effectively creating a rectangular kernel. In contrast, β = 2 increases the importance of small distances, and downweighing the importance of larger distances. The inverse distance weightings for β = 0, 1, 2 were applied to the first six PCs computed on the Mouse Ovary dataset and visualized spatially (Extended Data Fig. 16B). Setting β = 0 results in large-scale smoothing across the structures present within the tissue. In contrast, β = 2 facilitates a localized smoothing effect, resulting in finer granularity of the observed structures. These results underscore the importance of β in modulating the influence of neighboring points on the distance-weighted computations.

2.11. Differential distance metrics and weighting analysis

For the RASP results presented in Sections 2.3-2.7, spatial smoothing was performed using a sparse matrix whose elements represent the inverse Euclidean distances raised to the power of β between ST locations and diagonal values set to the minimum distance for that column (or half the minimum distance in the case of Visium technology). To assess alternative spatial smoothing approaches, we compared Euclidean and Manhattan distances using inverse distance weighting and kernel-based weightings (gaussian, quadratic) for the human breast cancer dataset (Extended Data Fig.18A). Following the application of these alternative smoothing methods, we performed clustering on the resulting smoothed PCs and calculated the ARI to evaluate the performance of each approach (Extended Data Fig.18 B,D). Our results demonstrate that the inverse Euclidean distance method consistently outperforms the other smoothing techniques in terms of clustering accuracy. This finding reinforces the effectiveness of our chosen approach in preserving the biological signals within the ST data. To provide further insight into the impact of the various smoothing strategies, we visualized the first four PCs generated by each method as kernel density plots (Extended data Fig.18C). This visualization highlights the subtle differences in how each method transforms the data, revealing distinct distributions that may influence the subsequent clustering results.

3. Discussion

Spatially informed dimensionality reduction of ST data is crucial for downstream tasks such as spatial domain identification, cell type annotation, and reduced rank gene signature reconstruction. In this study, we described RASP, a computationally efficient spatially informed dimensionality reduction method that seamlessly integrates into the standard analysis workflow provided by the Scanpy package. RASP utilizes randomized PCA followed by spatial smoothing to generate latent variables that can be leveraged for a wide range of downstream analyses including clustering to predict cell types or spatial domains. RASP also facilitates the reduced rank reconstruction of spatially smoothed gene signatures, enhancing the interpretability of transcriptomic data in spatial contexts. An important feature of RASP is its support for the incorporation of non-transcriptomic covariates via an additional round of randomized PCA. Our findings demonstrate that RASP achieves similar, or higher, accuracy than existing methods for cell type and spatial domain detection tasks, with runtime performance that is orders-of-magnitude faster. RASP can flexibly accommodate various tissue types and ST technologies. Specifically, RASP accurately identifies biologically relevant tissue domains in structures that are homogeneous in size and shape, such as in the DLPFC dataset, while also providing robust labels akin to cell types in the highly heterogeneous mouse ovary sample. Notably, RASP is currently the only method among the techniques we evaluated that can efficiently handle the massive datasets produced by the 10x Xenium platform.

3.1. Parameter selection considerations

Several parameters require careful consideration when applying the RASP method to ST data. These include kNN threshold, β value, and the number of PCs to compute. To simplify parameter selection, we provide a recommended range of values for each parameter. Please see SI Table 2 for details.

Note that we chose to use inverse distance weighting instead of the more commonly employed Gaussian kernel weighting given the more direct biological interpretation (e.g., inverse squared distances correspond to the concentration of a secreted molecule) and increased flexibility (i.e., users can control both the power parameter and diagonal values instead of just a bandwidth parameter). While RASP defaults to inverse distance weighting, the method can be used with alternative distance metrics and smoothing approaches, each with different characteristics and potential benefits/disadvantages.

3.2. Non-transcriptomic covariates

The RASP method allows for the incorporation of additional location-specific covariates via a two-stage randomized PCA (these are specified using the covariate matrix Y in Algorithm 1). Potential covariates include local cell density and cell volume (for technologies that provide this information), the library size of each location/cell, and protein abundance estimates (for technologies that support joint transcriptomic/proteomic measurements). However, it is important to note that using cell density may be somewhat redundant, as the kNN threshold inherently accounts for local density variations. However, depending on the cluster algorithm, this covariate can still impart useful information (see Fig. 3B). Therefore, users should carefully select these parameters based on the specific objectives of their analysis. RASP’s fast runtime allows users to quickly screen a large range of parameter values to identify the optimal configuration. RASP contains functionality to quickly calculate Moran’s I and CHAOS scores on the resulting clusters and select the appropriate parameters for the final implementation. This is important because it allows users to leverage a priori knowledge to incorporate biologically relevant covariates that are not present in the transcriptomic data in a flexible manner, functionality other tools lack.

3.3. Limitations

While RASP has numerous advantages relative to existing spatially aware dimensionality reduction techniques, it is not without limitations. One of the primary concerns involves the specification of user-provided parameters. Although the default parameters are generally appropriate for a variety of situations, RASP’s smoothing threshold parameter can profoundly influence clustering results. Some degree of trial and error is often necessary to optimize this parameter. However, unlike competing tools that may require upwards of 30 minutes to run, RASP executes quickly, facilitating a straightforward optimization process for users. To assist with parameter selection, we provide guidelines that can help streamline this aspect of the workflow (SI Table 2). Additionally, it is important to note that for certain datasets, RASP may not be the best performing method. For instance, in the task of cortical layer assignment in the DLPFC, models such as SpatialPCA outperform RASP.

3.4. Future directions

Multi-slice alignment and integration of sections from the same organism over different time points are two areas that have witnessed numerous computational innovations over the last five years. Methods such as MOFA, MEFESTO and moscot utilize matrix factorization-based models to simultaneously model both temporal and spatial data [55-57]. PALMO employs variance decomposition analysis and incorporates some spatial modeling capabilities [58]. In contrast, methods such as PASTE2 and DeST-OT utilize optimal transport principles to align sections from the same tissue sample, enabling 3D models [59, 60]. Despite their promising capabilities, these models are computationally intensive, do not scale well to large datasets, and, importantly, lack streamlined approaches for joint dimensionality reduction and clustering. In the current study, we evaluated RASP’s performance on single sections of ST data. Future directions include scaling RASP to support joint spatially informed dimensionality reduction of multiple ST slices, encompassing various tissues or multiple time points. While this enhancement can be simply realized using RASP by analyzing the concatenation of the cell-by-gene and coordinate matrices, how best to handle potential batch effects is a non-trivial problem. Given its computational efficiency, RASP stands out as the sole method capable of jointly reducing massive subcellular ST datasets. Another promising avenue of investigation involves integrating multi-modal information into the spatial reduction process. For instance, data related to protein abundance from methodologies like CITE-seq or chromatin accessibility assessed via spatial ATAC-seq could significantly enhance spatial domain detection [61, 62]. RASP already includes functionality to support the incorporation of non-transciptomic covariates, and future research will focus on understanding the impact of these diverse data types in improving spatial domain detection and characterization.

4. Methods

4.1. RASP method

One of the primary motivations for spatially informed dimensionality reduction of ST data is the identification of spatial domains via unsupervised clustering. Clustering on standard PCs is often sufficient to identify cell types, but identification of larger tissue structures benefits from spatial information. The RASP method performs dimensionality reduction and spatial smoothing on normalized ST data and an optional set of location-specific covariates. The two-stage approach used by RASP was chosen to optimize computational efficiency and support the specification of covariates with an overall goal of enabling the flexible and efficient analysis of high-resolution ST datasets. Specifically, RASP uses a sparse matrix representation of the ST data, performs dimensionality reduction using randomized PCA, and generates spatially smoothed PCs using inverse distance weights sparsified according to a k-nearest neighbors (kNN) threshold (Fig. 1A). To support the integration of location-specific covariates into the latent variables, a second stage randomized PCA is optionally performed on the matrix created by appending the smoothed covariates to the smoothed PCs generated in the first stage. The spatially smoothed PCs generated by the RASP method have utility in a wide range of downstream analyses with clustering and reduced rank reconstruction of gene-level expression important use cases. A detailed description of the method is provided by Algorithm 1. For readability, this version of the algorithm automatically applies the same spatial smoothing to both the first stage PCs and the covariates, however, flexible smoothing of the covariates is supported so that users have control over whether to apply smoothing to each covariate and, if applied, the type of threshold that is employed.

4.2. Clustering on spatially smoothed principal components

For evaluation of the RASP method, clustering was performed on the spatially smoothed PCs (P_s or P_c) generated by RASP 1. Specifically, clustering was performed by first computing a nearest neighbors distance matrix and a neighborhood graph of spatial locations as described in Mclnnes et al., 2018 [63]. See SI for programming language specific implementation. Note that the distances used for constructing this kNN graph are measured on the spatially smoothed PCs and capture transcriptomic differences between spatial locations, which is distinct from the spatial distances between coordinates used in RASP 1. Construction of the neighborhood graph requires specification of the number of nearest neighbors, with the recommended value being between 5 and 50. In general, the number of nearest neighbors should be set based on the size of the target spatial features. For example, users interested in cell type annotation are advised to set the threshold to a smaller value between 5 and 10, while those seeking to annotate spatial domains may consider increasing the threshold above 20. In most cases, a default of 10 is sufficient.

Distinct clusters representing spatial domains or cell types are then computed on the kNN graph using a user-defined clustering algorithm. The RASP method currently supports the use Mclust, Louvain, Leiden, and Walktrap methods. Note that for large datasets, users may want to avoid the use of the Walktrap clustering method as the computational complexity of the algorithm will increase runtime to an unreasonable level on a local machine.

4.3. Reduced rank reconstruction of gene-level expression

In addition to domain identification, an important application of spatially smoothed PCs is the reconstruction of gene-level expression values. Reduced rank reconstruction is motivated by the fact that the biological signal in transcriptomic data typically has a much lower rank than the rank of the observed measurements. Projecting ST data onto a lower dimensional space borrows the signal from correlated genes and can effectively mitigate both sparsity and noise [30]. For the spatially smoothed PCs generated by RASP 1, the reconstruction process follows Algorithm 2. Important features of this method relative to standard PCA-based reduced rank reconstruction include the use of spatially smoothed PCs to incorporate information from adjacent locations, the optional reconstruction of the smoothed PCs to account for location-specific covariates, and the thresholding and optional scaling of the reconstructed values following the approach used by the ALRA method [30].

Algorithm 1 RASP method

$$\begin{array}{cc}\hfill & \text{Inputs:}\hfill \\ \hfill & •\mathbf{X}:n\times m\mathsf{location}\text{-}\mathsf{by}\text{-}\mathsf{gene}\mathsf{matrix}\mathsf{that}\mathsf{holds}\mathsf{normalized}\mathsf{expression}\mathsf{values}\mathsf{for}n\mathsf{spatial}\mathsf{locations}\hfill \\ \hfill & \mathsf{and}m\mathsf{genes}.\hfill \\ \hfill & •\mathbf{C}:n\times 2\text{matrix of spatial coordinates}.\hfill \\ \hfill & •p:\mathsf{Number}\mathsf{of}\mathsf{spatially}\mathsf{smoothed}\mathsf{PCs}\mathsf{to}\mathsf{compute}.\hfill \\ \hfill & •\mathit{threshold}:\mathsf{Threshold}\mathsf{value}.\mathsf{This}\mathsf{is}\mathsf{the}\mathsf{number}\mathsf{of}\mathsf{nearest}\mathsf{neighbors}\mathsf{to}\mathsf{retain}.\hfill \\ \hfill & •\alpha :\text{The diagonal elements of the distance matrix are set to this value to avoid division by zero}\hfill \\ \hfill & \text{when computing inverse distances}.\text{Default value is the minimum off-diagonal distance for the row}.\hfill \\ \hfill & •\beta :\text{Power used for inverse distance weighting}.\text{Default is}2.\hfill \\ \hfill & •\mathbf{Y}:\text{Optional}n\times b\text{matrix of}b\text{location-specific covariates},\text{e.g.},\text{local cell density}.\text{While}\hfill \\ \hfill & \text{spatial smoothing is automatically applied to these variables in the algorithm, flexible control}\hfill \\ \hfill & \text{over covariate smoothing is possible in practice}.\hfill \\ \hfill & \text{Outputs:}\hfill \\ \hfill & •{\mathbf{P}}_{\mathbf{s}}:n\times p\text{matrix that holds the spatially smoothed projection of}\mathbf{X}\text{onto the top}p\text{PCs}.\hfill \\ \hfill & •{\mathbf{Y}}_{\mathbf{s}}:n\times b\text{matrix that holds the spatially smoothed version of}\mathbf{Y}.\hfill \\ \hfill & •\mathbf{W}:m\times p\text{matrix of loadings for the top}p\text{PCs of}\mathbf{X}.\hfill \\ \hfill & •{\mathbf{P}}_{\mathbf{c}}:\mathbf{If}\mathbf{Y}\text{is specified},\text{the}n\times p\text{matrix that holds the projection of}[{\mathbf{P}}_{\mathbf{s}},{\mathbf{Y}}_{\mathbf{s}}]\text{onto the top}p\text{PCs}\hfill \\ \hfill & \text{where}[{\mathbf{P}}_{\mathbf{s}},{\mathbf{Y}}_{\mathbf{s}}]\text{is the}(p+b)\times n\text{matrix formed by appending the columns of}{\mathbf{Y}}_{\mathbf{s}}\text{to}{\mathbf{P}}_{\mathbf{s}}.\hfill \\ \hfill & •{\mathbf{W}}_{\mathbf{c}}:\text{if}\mathbf{Y}\text{is specified},\text{the}(p+b)\times p\text{matrix of loadings for the top}p\text{PCs computed on}[{\mathbf{P}}_{\mathbf{s}},{\mathbf{Y}}_{\mathbf{s}}].\hfill \\ \hfill & \text{Notation:}\hfill \\ \hfill & •\text{Let}\mathbf{W}[]\text{represent a subsetting of the matrix}\mathbf{W}\text{with}\mathbf{W}[i,j]\text{the element in the}{i}^{th}\text{row and}{j}^{\mathit{th}}\hfill \\ \hfill & \text{column and with}\mathbf{W}[.,.]\text{the iteration over all}i,j\text{index combinations}.\hfill \\ \hfill & 1:\mathbf{W}\leftarrow \mathit{randomizedPCA}(\mathbf{X},p)▷\text{Perform randomized PCA on}\mathbf{X}\text{for rank}p.\text{By default},\mathbf{X}\text{is centered but}\hfill \\ \hfill & \text{not scaled}.\mathsf{See}\mathsf{Algorithm}S1\mathsf{for}\mathsf{details}.\hfill \\ \hfill & 2:\mathbf{P}\leftarrow \mathbf{XW}▷\text{Project}\mathbf{X}\text{onto the top}p\text{PCs}.\hfill \\ \hfill & 3:\mathbf{NN}\leftarrow \mathit{fitNearestNeighbors}(\mathbf{C},\mathit{threshold})▷\text{Fit a nearest neighbors data structure on the spatial}\hfill \\ \hfill & \text{coordinates for}\mathit{threshold}\mathsf{neighbors}.\hfill \\ \hfill & 4:\mathbf{D}\leftarrow \mathit{sparseMatrix}(\mathbf{NN})▷\text{Compute a sparse distance matrix using}\mathbf{NN}.\text{See Algorithm S}4\text{for details}.\hfill \\ \hfill & 5:\mathit{diag}(\mathbf{D})\leftarrow \alpha ▷\text{Set diagonal elements of}\mathbf{D}\text{to}\alpha .\hfill \\ \hfill & 6:{\mathbf{D}}_I[.,.]\leftarrow 1∕\mathbf{D}{[.,.]}^{\beta }▷\text{Create an inverse distance matrix using power}\beta .\text{Inverse distances for}\hfill \\ \hfill & \text{thresholded elements are set to}0.\hfill \\ \hfill & 7:{\mathbf{D}}_I[.,.]\leftarrow \frac{{\mathbf{D}}_I[.,.]}{\max ({\mathbf{D}}_I[.,.])}▷\text{Standardize each column of}{\mathbf{D}}_I\text{to the maximum of that column}.\hfill \\ \hfill & 8:{\mathbf{P}}_{\mathbf{s}}\leftarrow {\mathbf{D}}_I\mathbf{P}▷\text{Spatially smooth the PCs using inverse distance weights}.\hfill \\ \hfill & 9:\text{if}\mathbf{Y}\mathbf{then}▷\text{If covariate matrix}\mathbf{Y}\text{is specified}.\hfill \\ \hfill & 10:{\mathbf{Y}}_{\mathbf{s}}\leftarrow {\mathbf{D}}_I\mathbf{Y}▷\text{Spatially smooth covariates}.\hfill \\ \hfill & 11:{\mathbf{W}}_{\mathbf{c}}\leftarrow \mathit{randomizedPCA}([{\mathbf{P}}_{\mathbf{s}},{\mathbf{Y}}_{\mathbf{s}}],p)▷\text{Perform randomized PCA on the merge of}{\mathbf{P}}_{\mathbf{s}}\text{and}{\mathbf{Y}}_{\mathbf{s}}.\hfill \\ \hfill & 12:{\mathbf{P}}_{\mathbf{c}}\leftarrow [{\mathbf{P}}_{\mathbf{s}},{\mathbf{Y}}_{\mathbf{s}}]{\mathbf{W}}_{\mathbf{c}}▷\text{Project}[{\mathbf{P}}_{\mathbf{s}},{\mathbf{Y}}_{\mathbf{s}}]\text{onto the top}p\text{PCs}.\hfill \\ \hfill & \text{return}{\mathbf{P}}_{\mathbf{s}},\mathbf{W},{\mathbf{Y}}_{\mathbf{s}},{\mathbf{P}}_{\mathbf{c}},{\mathbf{W}}_{\mathbf{c}}\hfill \\ \hfill & 13:\mathsf{else}\hfill \\ \hfill & \text{return}{\mathbf{P}}_{\mathbf{s}},\mathbf{W}\hfill \\ \hfill & \mathsf{end}\hfill \end{array}$$

Algorithm 2 Reduced rank reconstruction

$$\begin{array}{cc}\hfill & \text{Inputs:}\hfill \\ \hfill & •\mathbf{X}:n\times m\text{location-by-gene matrix that holds normalized expression values for}n\text{spatial locations}\hfill \\ \hfill & \text{and}m\text{genes}.\text{This is the same matrix provided as input to Algorithm}1.\hfill \\ \hfill & •\mathbf{P}\mathbf{s},\mathbf{W},{\mathbf{P}}_{\mathbf{c}},{\mathbf{W}}_{\mathbf{c}}:\mathtt{Outputs}\mathtt{from}\mathtt{Algorithm}1.{\mathbf{P}}_{\mathbf{c}}\text{and}{\mathbf{W}}_{\mathbf{c}}\text{are optional and should only be included if}\hfill \\ \hfill & \text{location-specific covariates were specified as inputs to Algorithm}1.\hfill \\ \hfill & •q:\mathsf{Quantile}\mathsf{probability}\mathsf{used}\mathsf{for}\mathsf{thresholding}\mathsf{reconstructed}\mathsf{expression}\mathsf{values}.\hfill \\ \hfill & •\mathit{scale}:\text{True-to-scale reconstructed values to match variance of original expression}.\hfill \\ \hfill & \text{Outputs}:\hfill \\ \hfill & •{\mathbf{X}}_{\mathbf{r}}:\text{Reduced rank reconstruction of}\mathbf{X}.\hfill \\ \hfill & \text{Notation}:\hfill \\ \hfill & •\mathbf{Let}{1}^{x\times y}\text{be an}x\times y\text{matrix of ones}.\hfill \\ \hfill & •\text{Let}\mathbf{W}[]\text{represent a subsetting of the matrix}\mathbf{W}\text{with}\mathbf{W}[i,j]\text{the element in the}{i}^{\mathit{th}}\text{row and}{j}^{\mathit{th}}\hfill \\ \hfill & \text{column},\mathbf{W}[i,]\text{the}{i}^{\mathit{th}}\text{row},\mathbf{W}[,j]\text{the}{j}^{\mathit{th}}\text{column},\text{and}\mathbf{W}[\mathbf{r},\mathbf{c}]\text{the sub-matrix containing rows with}\hfill \\ \hfill & \text{indices in}\mathbf{r}\text{and columns with indices in}\mathbf{c}.\hfill \\ \hfill & 1:\mathbf{if}{\mathbf{P}}_{\mathbf{c}},{\mathbf{W}}_{\mathbf{c}}\text{then}▷\text{If}{\mathbf{P}}_{\mathbf{c}}\text{and}{\mathbf{W}}_{\mathbf{c}}\text{were specified}\hfill \\ \hfill & 2:[{\mathbf{P}}_{\mathbf{s},\mathbf{r}},{\mathbf{C}}_{\mathbf{r}}]\leftarrow {\mathbf{P}}_{\mathbf{c}}\cdot {{\mathbf{W}}_{\mathbf{c}}}^T▷\text{Reconstruct smoothed PCs and covariates}\hfill \\ \hfill & 3:{\overline{\mathbf{p}}}_{\mathbf{s}}\leftarrow ({{\mathbf{P}}_{\mathbf{s}}}^T{1}^{n\times 1})∕n▷\text{Compute column means of}{\mathbf{P}}_{\mathrm{s}}\hfill \\ \hfill & 4:{\mathbf{P}}_{\mathbf{s},\mathbf{r}}\leftarrow {\mathbf{P}}_{\mathbf{s},\mathbf{r}}+1^{n\times p}{\overline{\mathbf{p}}}_s^T▷\text{Add back column means}\hfill \\ \hfill & 5:{\mathbf{X}}_{\mathbf{r}}\leftarrow {\mathbf{P}}_{\mathbf{s},\mathbf{r}}{\mathbf{W}}^T▷\text{Reconstruct ST expression matrix using reconstructed PCs}\hfill \\ \hfill & 6:\text{else}\hfill \\ \hfill & 7:{\mathbf{X}}_{\mathbf{r}}\leftarrow {\mathbf{P}}_{\mathbf{s},\mathbf{r}}{\mathbf{W}}^T\hfill \\ \hfill & \mathsf{end}\hfill \\ \hfill & 8:\overline{\mathbf{x}}\leftarrow ({\mathbf{X}}^T{1}^{n\times 1})∕n▷\text{Compute column means of}\mathbf{X}\hfill \\ \hfill & 9:{\mathbf{X}}_{\mathbf{r}}\leftarrow {\mathbf{X}}_{\mathbf{r}}+1^{n\times m}{\overline{\mathbf{x}}}^T▷\text{Add back column means}\hfill \\ \hfill & 10:\mathsf{for}i\in \{1,\dots ,m\}\mathbf{do}▷\text{Threshold and optionally scale each reconstructed gene}\hfill \\ \hfill & 11:x_q\leftarrow \mathit{quantile}({\mathbf{X}}_{\mathbf{r}}[,i],q)▷\text{Find quantile of reconstructed values for probability}q\hfill \\ \hfill & 12:\mathsf{for}j\in \{1,\dots ,n\}\mathbf{do}▷\text{If absolute reconstructed value is greater than absolute quantile},\text{keep},\hfill \\ \hfill & \text{otherwise},\text{replace with original value if that was positive or set to zero}.\hfill \\ \hfill & 13:{\mathbf{X}}_{\mathbf{r}}[j,i]\leftarrow \{\begin{array}{cc}{\mathbf{X}}_{\mathbf{r}}[j,i]\hfill & \text{if}\mid {\mathbf{X}}_{\mathbf{r}}[j,i]\mid \ge \mid x_q\mid \hfill \\ \mathbf{X}[j,i]\hfill & \text{if}\mathbf{X}[j,i]>0\text{and}\mid {\mathbf{X}}_{\mathbf{r}}[j,i]\mid <\mid x_q\mid \hfill \\ 0\hfill & \text{if}\mathbf{X}[j,i]=0\text{and}\mid {\mathbf{X}}_{\mathbf{r}}[j,i]\mid <\mid x_q\mid \hfill \end{array}\phantom{\}}\hfill \\ \hfill & \mathsf{end}\hfill \\ \hfill & 14:\text{if scale then}\hfill \\ \hfill & 15:{\sigma }_i\leftarrow \mathit{stddev}(\mathbf{X}[,i]\mid \mathbf{X}[,i]>0)▷\text{Compute standard deviation of non-zero}\text{X}[,i]\hfill \\ \hfill & 16:{\sigma }_{r,i}\leftarrow \mathit{stddev}({\mathbf{X}}_{\mathbf{r}}[,i]\mid {\mathbf{X}}_{\mathbf{r}}[,i]>0)▷\text{Compute standard deviation of non-zero}{\text{X}}_{\text{r}}[,i]\hfill \\ \hfill & 17:s\leftarrow {\sigma }_i∕\mathit{max}({\sigma }_{r,i},1e-10)▷\text{Compute scaling factor}(\text{avoiding division by zero})\hfill \\ \hfill & 18:{\mathbf{X}}_{\mathbf{r}}[,i]\leftarrow s{\mathbf{X}}_{\mathbf{r}}[,i]▷\text{Apply scaling}\hfill \\ \hfill & \mathsf{end}\hfill \\ \hfill & \mathsf{end}\hfill \\ \hfill & \text{return}{\mathbf{X}}_r\hfill \end{array}$$

4.4. Application of RASP to existing ST datasets

4.4.1. Dataset descriptions

We applied RASP on four publicly available ST datasets, as well as two simulated ST, that were generated using different techniques and resolutions on human and mouse tissues. Details of data preprocessing can be found in Supplementary Information, details on data simulation can be found in 4.4.4. In brief, raw count matrices where filtered to remove cells/locations with mitochondrial and ribosomal RNA > 10%, cells/locations with < 100 reads and > 5 times the mean absolute deviation (MAD). Resulting data were scaled, and log-normalized prior to running RASP (SI 1.5). The mouse ovary dataset (Vizgen MERFISH technology) consisted of 43,038 cells and 228 genes. The tissue section used in our analysis comes from a series of ST experiments profiling mouse ovaries at 0, 4hr, and 12hr after induced ovulation with human choronic gonadotropin (hCG). The tissue investigated in this method was from the 12hr group and has subcellular resolution. Ground truth cell type annotations for eight cell types were taken from the original publication [15]. The mouse olfactory bulb dataset (STOmics Stereo-seq technology) consisted of 19,109 and 14,367 genes. This coronal section of the olfactory bulb is also profiled at sub-cellular resolution however the original publication did not provide ground truth regional annotations [44]. The human breast cancer tumor dataset (10x Genomics, Xenium technology) consisted of 565,916 cells and 541 genes, making it the largest dataset tested. This is a 5um thin resected infiltrating ductal carcinoma it situ with ground truth annotations provided in the SubcellularSpatialData R package [48]. Original annotations were determined from the accompanying histology image. The human DLPFC dataset (10x Genomics, Visium technology) consisted of 3,639 spots and 15,124 genes. The data here is a portion of the DLPFC that spans six neuronal layers plus white matter. The ground truth layer annotations come from the spatialLIBD R package [43]. The two simulated datasets (Stripes, and Dots) consisted of 10,000 cells and 150 genes each.

4.4.2. RASP evaluation

To evaluate the RASP method, we applied the algorithm to the processed gene expression matrices of the six datasets described above. The locations where then clustered according to the spatially smoothed PCs contained in either the P_sor P_c matrices output by RASP. Cluster quality was assessed using spatial continuity and compactness analysis quantified by the Moran’s I value and CHAOS score (see SI 1.3-1.4) for all data sets and, if ground truth annotations were available, the adjusted Rand index (ARI) (see SI 1.2). The evaluation process required optimizing key parameters in the RASP pipeline, particularly the distance threshold used for constructing the sparse distance matrix and the β value used to exponentiate the inverse distance matrix.

Given the large parameter space, a systematic parameter sweep was performed. To efficiently explore this space, we adopted a one-parameter-at-a-time optimization strategy. All other variables were held constant while one parameter incremented upon. Specifically, the number of kNN for spatial smoothing was swept from 1 to 50 for the MERFISH, Visium and Stereo-seq datasets, and from 1 to 120 for the Xenium and simulated datasets. Each kNN sweep was performed a total of nine times, for β = 0, 0.25, 0.5, , 0.75, 1, 1.25, 1.5, 1.75, 2. This approach allowed us to comprehensively explore the influence of each parameter individually. For each combination of parameters, the ARI (SI 1.2) was computed to quantify the similarity between the resulting clusters and the ground truth labels, where available, enabling the identification of the optimal parameter set for each dataset. For the olfactory bulb dataset ground truth annotations were unavailable and spatial continuity and compactness analysis was quantified by the CHAOS score, and the spatial spatial autocorrelation of each cluster was calculated using Moran’s I score in place of an accuracy measurement.

4.4.3. Comparison methods for spatial domain detection

We compared the performance of RASP against five other spatially informed dimensionality reduction or spatial domain detection algorithms: BASS (R, version 1.3.1), GraphST (Python, version 1.1.1), SEDR (Python, version 1.0.0), SpatialPCA (R, version 1.3.0), and STAGATE pyG (Python, version 1.0.0)[24, 26-29]. These methods where selected as they demonstrated the best performance across a variety of tissues and employ diverse model architecture. BASS models ST data in a hierarchical Bayesian framework and treats the cell type or spatial domain label for each spot or cell as a latent variable and infers them through an efficient inference algorithm. GraphST utilizes a GNN encoder to learn latent representations of the variables and further refines the representations using self-supervised contrastive learning. A decoder is employed to reconstruct gene expression, where clustering is employed. SEDR learns a low-dimensional representation of the data embedded with spatial information via a masked self-supervised deep auto-encoder and a variational graph convolutional auto-encoder. This low dimensional embedding is then clustered to identify spatial domains. SpatialPCA models gene expression as a function of latent factors through a factor analysis model. It builds a kernel matrix using the coordinates from the experiment to model the spatial correlation structure of the latent factors across the tissue. The inferred low-dimensional components are then clustered to identify spatial domains. Finally, STAGATE learns a low-dimensional latent representation of spatial information and gene expression via a graph attention auto-encoder trained on a spatial neighbor network (SNN). The output is then clustered to identify spatial domains. For each method, we followed the associated GitHub tutorials to run the stated methods for spatial domain and cell type detection, unless otherwise specified. Details are provided in SI 2.3-2.6. All methods where run on a Linux server with 15 CPUs for fair comparison of runtime. The latent dimensions found by each method were clustered using three clustering algorithms, Mclust, Louvain, and Leiden. The Walktrap clustering method was also used on the DLPFC dataset. Accuracy of the clusters generated by each method was determined by calculating ARI and CHAOS scores (SI 1.2-1.3). Note that these methods required allocations of arrays > 1TB for the human breast cancer dataset and as such where unable to run successfully.

4.4.4. Generation of simulated ST data

We simulated ST data according to two different tissue models, Stripes and Dots, using the SRTsim package’s GUI [54], following the reference free example provided at https://xzhoulab.github.io/SRTsim/03_Reference_Free_Example/. The Stripes model represents a laminar tissue with eight vertical domains (Extended Data Fig 8A). The Dots dataset represents a more complex tissue architecture that includes circular clusters of cells of varying sizes, small striped regions, and a large proportion of background cells making up a background region (Extended Data Fig 11A). Both models include eight annotated spatial domains. Count data was simulated according to these models for 10,000 cells, 50 low signal genes, 50 high signal genes and 50 noise genes using a zero inflated negative binomial distribution with μ = 2, zero % = 0.5, and θ = 0.5. Each annotated tissue region was assigned a unique fold change value (relative to μ) to simulate differences in spatial domains seen in real tissue. We passed the Stripes and Dots models to the reGenCountshiny function with different seed-number values to generate 100 replicate count matrices for each. For model details, spatial coordinates and domain labels for the simulated datasets please refer to 4.4.4.

For details on how the simulated data was processed prior to running RASP see SI 1.6.

Data availability

Mouse ovary: Please contact the authors for access the the Mouse ovary dataset.

DLPFC: We downloaded the raw DLPFC data (slide #151673) from the spatialLIBD website (http://research.libd.org/spatialLIBD/index.html).

Mouse Olfactory bulb: The dataset was downloaded from the SEDR publication GitHub repository [64] (https://github.com/JinmiaoChenLab/SEDR_analyses/tree/master/data).

Breast Cancer: We downloaded the Breast cancer Xenium dataset from the SubcellularSpatialData R package via ExperimentHub, dataset # EH8567, sample ID ’IDC’ (https://www.bioconductor.org/packages/release/data/experiment/html/SubcellularSpatialData.html).

Simulated datasets and RASP implementation: Please visit our GitHub to access the simulated datasets and their associated models.

Please visit our GitHub for details on RASP implementation: https://github.com/Goods-Lab/RASP.