spMosaic: multi-sample integration for scalable spatial domain discovery based on spatial transcriptomics

Abstract

Spatial transcriptomics enables high-resolution profiling of gene expression within intact tissue architecture, providing new opportunities to study the spatial organization of tissue structures and cell types. However, identifying spatial domains that are reproducible across samples and individuals remains challenging due to batch effects, inter-individual heterogeneity, increasing dataset sizes and the incomparability of spatial coordinates across independently processed tissue sections. We propose spMosaic, a scalable computational framework for multi-sample spatial domain identification. The method integrates two key components: (i) generalized additive model–based spatial smoothing to identify spatially variable genes shared across samples, and (ii) a batch-aware dual-encoder autoencoder that learns low-dimensional representations while encouraging sample-associated variation to be represented separately from domain-relevant biological variation. A subsequent deep embedded clustering refinement step further improves the separation and compactness of inferred domain assignments. We evaluated spMosaic against nine existing methods in both single-individual and multi-individual settings across diverse datasets spanning five technologies. spMosaic consistently achieved strong clustering performance relative to existing methods while maintaining competitive sample integration. In addition, under the computational settings used in our benchmark, spMosaic was the only method among those evaluated that successfully scaled to a large cohort, enabling the analysis of 378 918 spatial spots from 31 frontal cortex samples across 12 mice. These results demonstrate that spMosaic provides an integrative and scalable approach for multi-sample spatial transcriptomics analysis, enabling accurate and interpretable characterization of spatial tissue organization.

Introduction

Spatially resolved transcriptomics (SRT) encompasses a family of technologies that quantify gene expression while preserving the spatial organization of intact tissues [1]. Unlike dissociative single-cell RNA sequencing, SRT methods retain spatial context, enabling the study of tissue architecture and spatially coordinated biological processes. Over the past decade, rapid technological advances have led to two major categories of SRT platforms: imaging-based and sequencing-based approaches. Imaging-based technologies, such as MERFISH [2] and STARmap [3], use multiplexed fluorescence in situ hybridization or in situ sequencing to directly visualize and quantify RNA molecules within tissue sections. These methods achieve high spatial resolution, often at single-cell or even subcellular precision, but are typically limited to predefined gene panels. In contrast, sequencing-based platforms, including 10x Genomics Visium, Stereo-seq [4], and the Spatial Transcriptomics (ST) method [5], capture messenger RNA transcripts using spatially barcoded probes followed by next-generation sequencing. These approaches enable transcriptome-wide profiling but generally operate at spot-level resolution. The continued development of SRT technologies has enabled systematic characterization of transcriptomic and cellular landscapes across complex tissues, leading to important biological insights [6].

An important task in SRT data analysis is the identification of spatial domains (or spatial clusters), which refer to subregions of tissue where spots share similar transcriptomic profiles while exhibiting local spatial continuity [7, 8]. Unlike conventional clustering that groups observations solely by gene expression similarity, spatial domain analysis incorporates spatial context to capture the underlying organization of tissues [9]. These domains often correspond to anatomical structures, histological layers, or functional tissue compartments [10], and their characterization enables the identification of region-specific molecular programs, providing insights into tissue organization and disease mechanisms. For example, SRT studies of neurological diseases have revealed region-specific transcriptional alterations associated with disease pathology. In Alzheimer’s disease models, spatial domain analysis has identified distinct transcriptional programs in tissue regions surrounding amyloid plaques, highlighting how molecular changes are spatially organized around pathological structures [11].

As SRT datasets increasingly encompass multiple tissue sections, developmental stages, or individuals, identifying spatial domains in a reproducible and comparable manner across samples becomes a key computational challenge. Joint modeling of multiple samples enables information sharing across samples and can potentially improve the robustness and accuracy of spatial domain detection [12]. However, integrating SRT data across samples presents several key challenges. First, similar to multi-sample single-cell RNA-seq data, sample-specific batch effects may arise from technical variation or individual heterogeneity. Second, spatial coordinates from independently processed tissue sections are not directly comparable, complicating cross-sample integration. Third, as sample sizes and spot numbers increase, computational scalability becomes a critical concern for large-scale datasets [8].

To address multi-sample integration in SRT data, several probabilistic modeling approaches have been proposed, including BASS [12], BayesSpace [13], and PRECAST [14]. BASS and BayesSpace primarily focus on spatial domain identification within individual samples and rely on external batch correction tools such as Harmony [15] when applied to multiple samples. Spatial relationships are modeled only within individual samples by considering local spot neighborhoods. BASS employs a hierarchical Bayesian framework to jointly model cell-type composition and spatial domains, whereas BayesSpace uses a Markov random field prior to encourage spatial smoothness of cluster labels. PRECAST extends probabilistic modeling to multi-sample settings by jointly performing dimension reduction, clustering, and cross-sample alignment. These methods address spatial domain detection through explicit probabilistic structures, but differ in their modeling assumptions, including assumptions about probabilistic spatial-domain membership, spatial-dependence priors, latent cell-type composition, or cross-sample alignment in a shared latent space. These assumptions provide useful structure for modeling spatial organization. However, these approaches either depend on external batch correction procedures that are not tailored to SRT data or involve complex inference procedures that can limit scalability for large or heterogeneous datasets.

Another class of approaches integrates multiple SRT samples using graph neural networks (GNNs). In these methods, SRT data are represented as graphs in which spots are represented as nodes and edges encode spatial proximity. Through neighborhood aggregation along graph edges, GNNs capture local spatial continuity while learning low-dimensional embeddings that integrate gene expression and spatial context. Representative methods include spCLUE [16], STAligner [17], DeepST [18], SPIRAL [19], SEDR [20], and GraphST [21]. These methods differ in how they address batch effects across samples. For example, spCLUE uses contrastive learning with a batch prompting module, STAligner applies alignment based on mutual nearest neighbors, DeepST and SPIRAL adopt adversarial strategies introduced in domain-adversarial neural networks (DANNs), SEDR relies on external batch correction (e.g. Harmony), and GraphST learns joint embeddings on pre-aligned samples (e.g. using PASTE [22]). These methods incorporate spatial relationships through graph-based message passing and have demonstrated improvements in certain spatial domain detection and integration tasks. At the same time, they often require explicit construction of spatial graphs and extensive model training, which can limit scalability for large datasets. Their effectiveness also depends on how spatial neighborhoods are defined, how graph information is propagated during representation learning, and how variation across samples is handled. Some workflows further rely on spatial pre-alignment or external batch correction before or during integration. These design choices can be powerful when spatial correspondence across samples is available or desired, but they may be less suitable when tissue sections have distinct coordinate systems or when alignment itself presents a major challenge.

To address the aforementioned challenges, we propose spMosaic (Multi-sample integration for scalable spatial domain discovery based on SRT data), a unified computational framework for scalable cross-sample integration and spatial domain identification. Unlike existing approaches that rely on graph construction or external batch correction, spMosaic decouples spatial modeling from representation learning, enabling efficient integration of multiple samples while preserving biologically meaningful spatial structure. spMosaic is designed for the setting in which multiple tissue sections may have distinct spatial coordinate systems, sample-specific spatial expression patterns, and technical or individual-level variation. Rather than applying spatial smoothing, feature selection, and batch correction as independent off-the-shelf preprocessing steps, spMosaic connects these operations through a task-specific two-stage design for multi-sample spatial domain discovery. In the first stage, spMosaic performs sample-wise spatial modeling based on generalized additive models (GAMs) to estimate spatially coherent expression patterns within each tissue section and to identify spatially variable genes (SVGs) that are reproducibly informative across samples. Although GAMs have been used in SRT for related tasks such as morphology-aware SVG detection [23], trajectory-dependent gene discovery [24], and disease-remodeling analysis [25], they have not, to our knowledge, been used as an upstream sample-specific spatial smoothing and feature-construction layer for multi-sample spatial domain discovery. In the second stage, spMosaic uses the standardized smoothed expression of common SVGs as input to a batch-aware dual-encoder representation model. This architecture is designed to learn domain-relevant biological embeddings while explicitly representing sample-associated variation, rather than relying on external batch correction, adversarial learning, or explicit dependence penalties alone. A subsequent deep embedded clustering refinement step further improves the compactness and coherence of inferred domain assignments. This design decouples within-sample spatial modeling from cross-sample representation learning, avoids the need for spatial coordinate alignment, and enables scalable analysis of large multi-sample SRT datasets. Together, these task-specific design choices enable accurate, robust, and scalable spatial domain identification across heterogeneous samples and large cohorts.

Methods

Overview of the spMosaic method

The spMosaic framework is designed for integration and spatial domain detection in multi-sample SRT data. Here, samples may refer to either different tissue slices from the same individual or slices originating from multiple individuals. spMosaic identifies spatial domains across samples through a two-stage workflow that combines SVG detection based on spatial smoothing with batch-aware representation learning. Compared with existing approaches, spMosaic introduces (i) sample-wise spatial smoothing to recover spatial signals without requiring spatial alignment across samples, (ii) a batch-aware dual-encoder deep embedding clustering framework that aims to disentangle biological signals from sample-related variation to enable robust cross-sample spatial domain detection, and (iii) a computationally scalable design that processes slices independently during the smoothing stage and then learns shared representations across samples, allowing the method to handle large-scale datasets containing many spatial slices efficiently.

In the first stage, spMosaic performs spatial smoothing within each sample and identifies SVGs that are consistently informative across samples. SRT data are often sparse and noisy, due to limited transcript capture and technical variability. To mitigate these issues, spMosaic first performs gene-wise spatial smoothing to denoise gene expression measurements by leveraging spatial information and borrowing strength from neighboring spots. We illustrate this workflow using a toy example consisting of three samples (Fig. 1A). Spatial smoothing is performed independently within each sample using GAMs, where spatial coordinates are incorporated as smooth covariates. Fitting the smoothing model separately for each sample avoids the need for explicit spatial alignment across datasets, since spatial coordinates are typically not directly comparable across samples and gene expression measurements may be affected by sample-specific technical variation. The spatial smooth component from the fitted model provides denoised gene expression estimates that leverage spatial neighborhood information. To focus downstream analysis on biologically informative signals, spMosaic next identifies SVGs within each sample based on the contribution of spatial effects. We then retain genes that are consistently identified as spatially variable across samples, referred to as common SVGs. These genes represent shared spatially structured biological signals and serve as input to the second stage of the framework. For example, Fig. 1B presents a common SVG across the three spatial samples. The comparison between the original expression and the GAM-smoothed expression demonstrates that spatial smoothing reduces noise while preserving spatial patterns.

Figure 1.

Overview of spMosaic for multi-sample spatial domain identification. (A) Sample-wise spatial smoothing and common SVG selection. For each sample, spot-level gene expression is smoothed using GAMs to capture spatial expression patterns and reduce noise. SVGs are identified within each sample, and genes consistently detected as spatially variable across all samples are retained as common SVGs for downstream analysis. (B) Illustration of spatial smoothing. Original and smoothed expression levels of one common SVG across spots in three samples. (C) Batch-aware dual-encoder DEC framework for spatial domain identification. The standardized smoothed expression of common SVGs () is used as input to a deep embedding clustering framework. During pretraining, a dual-encoder autoencoder is trained to disentangle biological representations () from sample-specific batch effects () using an auxiliary batch classifier. In the subsequent DEC refinement stage, clustering is performed on the biological embedding while the batch encoder is fixed. (D) Illustration of spatial domain identification. UMAP visualization of spot embeddings from the biological representation (), where spots are colored by inferred spatial domains or sample labels.

In the second stage (Fig. 1C), spMosaic identifies spatial domains using a batch-aware dual-encoder deep embedding clustering framework adapted from the original DEC model [26]. The input to this model is the standardized smoothed expression matrix of the common SVGs. During the pretraining stage, a dual-encoder autoencoder is trained to disentangle biological variation from sample-specific batch effects. Specifically, encoder learns biological representations , while encoder captures batch-related representations . The concatenated latent representations are then used by the decoder to reconstruct gene expression. In the meantime, an auxiliary batch classifier operating on encourages the batch encoder to explicitly capture sample-specific batch effects. Under the pretraining objective , the model jointly minimizes reconstruction loss and batch classification loss. This design encourages batch-associated signals to be absorbed into , allowing to preferentially encode biological variation relevant for spatial domain identification. After pretraining, C cluster centers are initialized using the biological embeddings . DEC refinement is then performed with the batch-related encoder frozen, so that the sample-associated representations learned during pretraining are retained while the biological embeddings are refined for spatial-domain clustering. During this refinement, cluster assignments are iteratively refined by minimizing the Kullback–Leibler (KL) divergence between the soft cluster assignments and a target distribution, thereby improving cluster compactness and separation. The final spatial domains correspond to the refined cluster assignments based on the biological embeddings (). Visualization of these embeddings using UMAP illustrates how spatial domains are expected to form distinct clusters, while spots originating from different samples are well mixed, indicating successful integration across samples (Fig. 1D).

First stage of the spMosaic method: common SVG smoothing and detection

The first stage of spMosaic extracts spatially informative gene features through sample-wise spatial smoothing and cross-sample SVG selection (Fig. 1A). We perform gene-wise spatial smoothing using GAMs, which extend generalized linear models by allowing the linear predictor to include smooth, data-adaptive functions of covariates. This enables the estimation of a smooth expression surface over spatial coordinates. Spatial smoothing is performed independently within each sample, since spatial coordinates are typically not directly comparable across samples and gene expression measurements may be affected by sample-specific technical variation. From each fitted model, we extract the spatial smooth component to obtain denoised gene expression values that capture local spatial structure. Genes that are consistently identified as spatially variable across samples are then selected as common SVGs.

Sample-wise spatial smoothing. Depending on the SRT technology, gene expression may be recorded as count data (e.g. UMI counts from sequencing-based platforms such as 10x Visium) or as continuous normalized values (e.g. imaging-based platforms such as MERFISH). We assume that the same set of J genes is measured across all L samples, which may originate from the same or different individuals. In practice, this can be ensured by taking the intersection of genes measured across samples. For gene j, where j = 1,… ,J, let denote the observed expression measurement at spot i in sample l, for and l = 1,… ,L. Let denote the spatial coordinates of spot i in sample l. In the smoothing step, we fit a separate GAM for each gene within each sample, using thin-plate regression splines to model spatial variation in gene expression.

When gene expression is recorded as counts, we assume a Poisson distribution

(1)

The log link function is specified as

(2)

where is the baseline of gene j in sample l, and denotes the library size of spot i from sample l, included as an offset to account for varying sequencing depth. The spatial smooth is represented using thin-plate regression splines,

(3)

where are spline basis functions and are coefficients estimated from the data. The number of spline basis functions K controls the flexibility of the spatial smooth and is treated as a hyperparameter, selected as described in the Supplementary Methods. The model is fitted using the gam function from the R package mgcv [27]. The smoothed gene expression is then computed as

(4)

We apply the log transformation to the exponentiated fitted spatial smooth to improve numerical stability for sparse count data. When expression is extremely sparse, the fitted spatial smooth on the log scale, may take very large negative values, which can lead to numerical instability. The transformation provides a stable monotone mapping that regularizes extreme values while preserving the relative ordering of smoothed expression levels.

We use the Poisson GAM as the default for count data because the primary purpose of this stage is to construct spatially coherent features for downstream domain identification, rather than to serve as the final endpoint for gene-level SVG inference. We nevertheless acknowledge that SRT count data may exhibit overdispersion, in which case Poisson-based tests can yield anti-conservative P-values. We evaluated negative-binomial GAMs in a sensitivity analysis across representative count-based datasets. The negative-binomial formulation generally produced larger P-values and retained fewer common SVGs, indicating that the count distribution can affect SVG inference. However, when the resulting features were propagated through the full spMosaic pipeline, negative-binomial GAMs did not improve downstream clustering performance under the evaluated settings (Supplementary Figures S1 and S2). In addition, negative-binomial GAM fitting was substantially more computationally demanding in our implementation. We therefore retain the Poisson GAM as the default upstream screening and feature-construction model for spMosaic, while recognizing that negative-binomial or other overdispersion-aware formulations may be preferable when rigorous gene-level SVG inference is the primary objective.

When gene expression is available as non-count continuous measurements, we assume that library size effects have already been removed and that the expression values have been log-transformed to stabilize variance. These assumptions are reasonable for imaging-based SRT platforms, where expression values are typically normalized during preprocessing and reported on a continuous scale. Under these conditions, we assume

(5)

The link function becomes

(6)

where is the baseline and denotes a spatial smooth function represented using thin-plate regression splines, as defined above. After model fitting, the smoothed gene expression is defined as

(7)

Common SVG detection. Next, we focus on genes that carry spatially structured biological signals by restricting downstream spatial domain detection to SVGs. Under our GAM framework, spatial structure is captured by smooth functions of the within-sample spatial coordinates. For a true SVG, incorporating these coordinate-dependent smooth terms should explain a substantial portion of the expression variability beyond a nonspatial baseline model. For each gene and each sample, we therefore compare the fitted full GAM (Equations (1) and (2) for count data or Equations (5) and (6) for continuous data) to a baseline model that excludes the coordinate-related smooth terms. Specifically, for gene j, the baseline model for count data specifies the link function as

(8)

while for continuous data the link function becomes

(9)

Evidence for spatial variation is assessed using a likelihood ratio test based on a chi-squared approximation, which compares the full GAM (including spatial smooth terms) with the corresponding baseline model that excludes the spatial smooth. This test produces a P-value for each gene within each sample. Within each sample, we control the false discovery rate using the Benjamini–Hochberg procedure and declare genes with adjusted P-values <.05 as SVGs. The test is implemented using the anova function for GAM models provided by the mgcv package.

Although SVGs are detected separately within each sample, the resulting sets may differ across samples due to biological heterogeneity or technical variation. To focus on spatial signals that are consistently present across samples and facilitate robust cross-sample integration, we define the common SVGs as genes that are identified as SVGs in all samples. These genes capture spatial expression patterns that reflect shared biological structure rather than sample-specific effects. We acknowledge that this definition may exclude biologically meaningful sample- or subgroup-specific spatial patterns. We therefore evaluated alternative prevalence-based definitions, including SVGs identified in a majority of samples and the union of sample-wise SVG sets. Strict common SVGs provided the strongest performance in the within-individual analyses and remained highly competitive in the more heterogeneous across-individual analyses, where a 60% prevalence threshold yielded only a modest improvement. Increasingly permissive thresholds and the union strategy generally reduced performance (Supplementary Methods and Supplementary Figure S3). We therefore retain common SVGs as the default for identifying shared spatial structures while recommending that the selection threshold be adapted when subgroup-specific spatial patterns are of primary interest.

Let denote the set of common SVGs, with . This common set is used as the input feature set for subsequent spatial domain detection. Before the second stage, we standardize the smoothed expression of the common SVGs within each sample to remove sample-specific scale differences driven solely by expression magnitude. For sample l, let denote the smoothed expression of gene at spot i, where . We perform gene-wise standardization across spots within each sample by computing

(10)

and then obtain the standardized expression

(11)

Thus, for each gene within sample l, the standardized expression across spots has mean 0 and standard deviation 1.

For each sample l, we construct the matrix whose (i,j) entry is for spot i and gene . We then concatenate all samples along the spot dimension to obtain a single integrated feature matrix

(12)

In this representation, each row corresponds to a spatial spot from one of the L samples and each column corresponds to a common SVG.

Second stage of the spMosaic method: spatial domain detection based on multiple samples

In the second stage, to identify spatial domains while integrating multiple samples, we employ a batch-aware dual-encoder DEC framework. The framework consists of two steps (Fig. 1C): (i) a pretraining step that uses a dual-encoder autoencoder with an auxiliary batch-prediction branch to encourage biological and sample-associated variation to be represented in separate latent components, and (ii) a DEC refinement step that performs clustering based on the learned biological embeddings. Let denote the concatenated standardized and smoothed expression matrix restricted to the common SVG set . After concatenation, we index spatial spots by . The feature vector for spot i is denoted by , and denotes the sample label indicating the sample from which the spot originates.

Pretraining: dual-encoder autoencoder with batch prediction. Our pretraining module uses an unsupervised representation learning framework that decomposes each spot representation into two components: biological embeddings and batch embeddings. The model takes the standardized smoothed expression matrix as input. For spot i, two encoders of identical architecture, denoted by and , map into biological embeddings and batch embeddings , respectively. Each encoder consists of fully connected layers with ReLU activation functions in the hidden layers, and d denotes the dimension of the bottleneck embedding layer. Let . For , the t-th layer of the two encoders is defined as

(13)

where and denote learnable weight matrices and bias vectors, respectively, and denotes the elementwise ReLU activation function used in the hidden layers. The output of the -th layer is taken as the embedding without applying a nonlinear activation, yielding

(14)

The design of the specific encoder architecture, including the number of layers and the number of hidden units in each layer, is discussed in Supplementary Methods.

To reconstruct the input expression profile, the biological and batch embeddings are concatenated:

(15)

The combined representations are passed through a decoder network to reconstruct the input feature vector. Let and let denote the number of decoder layers. The decoder layers are defined as

(16)

where denotes the ReLU activation function for hidden layers. The reconstructed expression vector is

(17)

The decoder architecture follows a mirrored structure of the encoder, except that the input dimension of the first decoder layer is 2d due to the concatenation of the two embeddings.

To encourage the batch embedding to capture sample-specific variation, we introduce a batch classifier that predicts the sample label from . The classifier is implemented as a fully connected network with one hidden layer:

(18)

where and denote learnable weight matrices and bias vectors, respectively, and denotes the elementwise ReLU activation. The predicted batch probabilities are obtained using the softmax transformation

where represents the predicted probability that spot i originates from sample l.

During pretraining, the model parameters are optimized by jointly minimizing a reconstruction loss and a batch prediction loss:

(19)

where the first term is the mean squared reconstruction loss between the input feature vector and its reconstruction . The second term is the cross-entropy (CE) loss between the true sample label and the predicted batch probability vector . The parameter controls the strength of the batch supervision and is treated as a hyperparameter, selected as described in the Supplementary Methods.

DEC refinement: clustering on the biological embeddings with a fixed batch encoder. After pretraining, we refine the clustering structure using the DEC framework [26] applied to the biological embedding space. Specifically, we initialize C clusters by applying a Gaussian mixture model (using mclust [28] in R) to the biological embeddings and use the resulting cluster means as the initial cluster centers . During the DEC refinement stage, the batch encoder is frozen so that the batch embeddings remain unchanged. The biological encoder, decoder, and cluster centers are then updated iteratively to refine the clustering structure in the biological embedding space. The batch classifier is used only during the pretraining stage to encourage the batch encoder to capture sample-specific variation and is not involved in the DEC optimization.

Given cluster centers , we compute the soft assignment of spot i to cluster c using a Student’s t-distribution with one degree of freedom:

(20)

Following the DEC framework, we construct an auxiliary target distribution that emphasizes confident assignments:

(21)

The model parameters are then optimized by minimizing the KL divergence between the target distribution and the soft assignment distribution , while retaining the reconstruction loss for training stability:

(22)

where α controls the contribution of the clustering loss. After DEC refinement, the spatial domain label for spot i is assigned as

The parameter α is treated as a hyperparameter and selected as described in the Supplementary Methods.

Spatial label refinement by neighborhood majority voting. DEC assigns each spot i a preliminary domain label . Inspired by GraphST [21] and spCLUE [16], we further refine labels using spatial neighborhood information to reduce isolated label noise and encourage local spatial coherence. Let denote the sample identity of spot i, and let be its spatial coordinates. For each sample l, we construct a fixed k-nearest-neighbor (KNN) graph using Euclidean distance among spots belonging to that sample. Denote by the set of spatial neighbors of spot i within sample l. By default, the number of neighbors k is set to 30.

Starting from the DEC labels , we iteratively refine the labels. At iteration t≥ 1, for each spot i in sample l, the updated label is the most frequent label among its neighbors

(23)

In the case of ties, we keep the previous label such that . Updates are applied synchronously, meaning that all votes at iteration t are computed from labels at iteration t-1. The KNN graph is fixed throughout refinement. The refinement process stops when either (i) the fraction of changed labels falls below a threshold ε (default ), or (ii) a maximum number of iterations is reached (default ). The final refined spatial domain assignment is given by .

Implementation details of spMosaic

All neural network components in spMosaic were implemented in PyTorch. Random seeds were fixed to ensure reproducibility. For both the dual-encoder autoencoder pretraining stage and the subsequent DEC refinement stage, the data were randomly split into training (80%) and validation (20%) sets. During pretraining, the dual-encoder autoencoder was optimized using the Adam optimizer with a learning rate of . The model was trained for up to 500 epochs using mini-batch gradient descent (batch size =1024). Early stopping was applied based on the validation loss, with a patience of 10 epochs and a minimum relative improvement threshold of . A ReduceLROnPlateau learning-rate scheduler (factor =0.5, patience =5) was used to reduce the learning rate when validation performance plateaued.

For DEC refinement, cluster centers were initialized using a Gaussian mixture model implemented in the mclust R package. The DEC model was then optimized using the Adam optimizer with learning rate for up to 10 epochs using mini-batch training (batch size =1024). A ReduceLROnPlateau learning-rate scheduler was also applied during this stage. Convergence was monitored using the cluster assignment change rate across epochs, and training was terminated early when the change rate fell below 1%. After DEC optimization, a neighborhood-based label refinement step was applied to further enhance spatial coherence of the inferred domains. This refinement step was applied to the Visium_DLPFC, MERFISH_POA, and MERFISH_FC datasets. It was not applied to the StereoSeq_MOSTA dataset due to its highly complex spatial structures, nor to the ST_OFB and STARmap_PFC datasets because they had relatively few spots per sample, for which additional smoothing may lead to over-smoothing of spatial domain assignments. The same refinement procedure was applied to the clustering results of all alternative methods as well.

Hyperparameter selection for spMosaic

spMosaic contains several model hyperparameters and settings that may influence performance. For each hyperparameter, all other settings were fixed while evaluating a predefined range of candidate values. Clustering performance was assessed using the adjusted Rand index (ARI). Hyperparameter choices were first determined using within-individual multi-sample analyses and were subsequently validated in across-individual settings to ensure robustness and generalizability. The selection of the following hyperparameters is described in detail in the Supplementary Methods: the number of smooth basis functions in the GAM (K), the procedure for SVG detection, the encoder architecture in the batch-aware DEC framework, the weight of the batch prediction branch during pretraining (), and the weight of the KL divergence during DEC refinement (α).

Benchmarking methods

To ensure a fair and reproducible comparison, all benchmarking methods (spCLUE [16], BASS [12], BayesSpace [13], PRECAST [14], STAligner [17], DeepST [18], SPIRAL [19], SEDR [20], and GraphST [21]) were implemented by following their official tutorials and recommended workflows whenever possible. Default parameter settings provided by the corresponding software packages were used unless modifications were necessary to ensure compatibility across datasets. For example, the number of principal components (PCs) was adjusted for imaging-based datasets with smaller gene panels. For methods that produce low-dimensional embeddings, spatial domain labels were obtained by applying Gaussian mixture models using mclust. The number of clusters was set to the number of annotated spatial domains in each dataset. To ensure consistent post-processing across methods, we applied the same spatial label refinement procedure used in spMosaic (see “Spatial label refinement by neighborhood majority voting”). The detailed settings of individual methods are summarized in the Supplementary Methods.

For computational benchmarking, R-based software tools were executed on CPU nodes with a memory limit of 128 GB. CPU usage was limited to 12 cores for within-individual analysis and six cores for across-individual analysis. For spMosaic, the GAM smoothing step was parallelized using 12 CPU cores for all analyses. Deep learning methods implemented in Python were executed on GPU nodes, while CPU usage for auxiliary computations was limited to 12 cores with a 128 GB memory cap. Since different methods vary in their software architecture and hardware support (e.g. CPU-only versus GPU-accelerated implementations), these runtime and memory usage comparisons reflect practical implementation performance rather than a fully hardware-matched benchmark.

Evaluation metrics

To evaluate the performance of multi-sample integration methods, we use metrics reflecting two complementary objectives: spatial-domain accuracy and sample mixing. Spatial-domain accuracy is assessed using ARI, normalized mutual information (NMI), and cell-level local inverse Simpson’s index (cLISI). Sample mixing is assessed using integration local inverse Simpson’s index (iLISI) and batch average silhouette width (batch ASW). Because these metrics evaluate different aspects of the learned representations, cLISI is calculated using the full learned embedding, whereas iLISI and batch ASW are calculated after projecting each embedding to its first two PCs, as described below.

Adjusted Rand index. The ARI quantifies the similarity between predicted and annotated spatial domain labels while correcting for agreement expected by chance. Let TP denote the number of spot pairs assigned to the same cluster in both the true and predicted labels, TN the number of spot pairs assigned to different clusters in both partitions, FP the number of spot pairs assigned to the same cluster in the predicted partition but to different clusters in the true partition, and FN the number of spot pairs assigned to the same cluster in the true partition but to different clusters in the predicted partition. The ARI is defined as

(24)

where E is the expected number of agreements under random labeling. The ARI score has the range of [-1, 1], where 1 indicates perfect agreement between the predicted and annotated labels. ARI is computed using the adjustedRandIndex function in the R package mclust [28].

Normalized mutual information. Similar to ARI, NMI measures the similarity between predicted and annotated spatial domain labels. Let P denote the predicted clustering and T the ground-truth domain labels. Let H(P) and H(T) represent the entropies of P and T, respectively, and let denote their mutual information. NMI is defined as

(25)

NMI ranges from 0 to 1, where 1 indicates perfect agreement and 0 indicates no mutual information between the two clusterings. NMI is computed using the NMI function from the R package aricode [29].

Clustering-based local inverse Simpson’s index. The cLISI score evaluates clustering coherence in the learned embedding space by assessing the extent to which spots assigned to the same predicted spatial domain form locally coherent neighborhoods in the embedding space [15], so it is calculated using each method’s full learned embeddings. To compute cLISI, we first construct a weighted KNN graph based on the Pearson correlation between full spot embeddings (e.g. the biological embeddings learned by spMosaic). Following the LISI framework [15], neighborhood sizes are determined using a fixed perplexity parameter (5%) multiplied by the overall number of spots across samples. For each spot i, the LISI is calculated as

(26)

where denotes the proportion of neighbors of spot i belonging to predicted cluster c, and C is the total number of predicted clusters. The cell-level LISI scores (ranging from 1 to C) are computed using the compute_lisi function from the lisi package [15]. To enable comparison across datasets with different numbers of clusters, we normalize cLISI following benchmarking practices in single-cell genomics [30]. The overall dataset-level cLISI is defined as

(27)

Under this normalization, cLISI ranges from 0 to 1, where 1 indicates that the local neighborhoods are dominated by a single predicted cluster (high clustering coherence).

Integration local inverse Simpson’s index. The iLISI score evaluates batch mixing performance in the embedding space [15] by quantifying how well spots from different samples are mixed within local neighborhoods. For iLISI, each method’s embeddings are first projected to its first two PCs, and local neighborhoods are constructed in this 2D principal component analysis (PCA) space. This fixed projection is applied uniformly across methods and is used to assess whether sample identity dominates the leading low-dimensional structure of the learned representation. A weighted KNN graph is then constructed in this reduced space, with the neighborhood size defined as in the cLISI calculation. For each spot i, iLISI is computed as

(28)

where denotes the proportion of neighbors of spot i belonging to sample l. The overall dataset-level iLISI is then defined as

(29)

After this normalization, iLISI ranges from 0 to 1, where 1 indicates that samples are evenly represented in the local neighborhoods (strong batch mixing).

Batch average silhouette width (batch ASW). Batch ASW evaluates global sample separation in the embedding space by quantifying the extent of sample mixing using silhouette scores computed with respect to sample labels. To keep the sample-mixing metrics comparable, batch ASW is calculated in the same 2D PCA space used for iLISI. Pairwise Euclidean distances are then computed in the PCA space. Let denote the batch (sample) label of spot i, where L is the total number of samples. For each spot i, the silhouette width is defined as

(30)

where a(i) is the average distance between spot i and all other spots within the same sample, and b(i) is the minimum average distance between spot i and spots from any other sample.

In the context of batch integration, both large positive and large negative silhouette values indicate poor mixing, as they reflect strong sample-specific structure in the embedding space. Therefore, we define

(31)

where I is the total number of spots. Under this definition, values closer to 1 indicate stronger sample mixing. Silhouette widths for each spot are computed using the silhouette function from the R package cluster [31].

Thus, ARI and NMI assess agreement between inferred domains and reference annotations, cLISI assesses local domain coherence in the full representations used for clustering, and iLISI and batch ASW summarize sample mixing in the dominant low-dimensional representations. The sample-mixing metrics should be interpreted as complementary measures of cross-sample integration rather than as evidence that all sample-associated variation has been removed from the full embeddings.

Results

Within-individual spatial domain detection

We first evaluated spMosaic on SRT datasets containing multiple samples from the same individual. The within-individual benchmark included datasets generated using both sequencing-based and imaging-based technologies. Specifically, the datasets included Visium_DLPFC [32] (12 human prefrontal cortex samples from three donors, Fig. 2A), StereoSeq_MOSTA [4] (four samples from one mouse embryo, Fig. 2B), ST_OFB [33] (six olfactory bulb samples from three mice, Fig. 2C), MERFISH_FC [34] (eight frontal cortex samples from three mice, Fig. 2D), MERFISH_POA [35] (five hypothalamic preoptic area samples from one mouse, Fig. 2E), and STARmap_PFC [3] (three prefrontal cortex samples from one mouse, Fig. 2F). These datasets vary in spatial resolution, gene panel size, and the number of spatial spots, providing a diverse benchmark for evaluating spatial domain detection across multiple platforms. Detailed dataset statistics are summarized in Supplementary Table S1. Across all evaluated datasets, gene expression matrices exhibit substantial sparsity (Fig. 3A), which poses challenges for reliable spatial domain identification. In addition, spatial coordinates are not aligned across samples, meaning that corresponding anatomical regions do not share a common coordinate system. This lack of spatial alignment presents an additional challenge for multi-sample spatial domain detection.

Figure 2.

Datasets for within-individual analysis. (A) Visium_DLPFC. Human prefrontal cortex samples generated using the 10x Genomics Visium platform. The dataset contains 12 samples from three donors, with 33 538 genes shared across samples and an average of 3944 spots per sample. (B) StereoSeq_MOSTA. Mouse embryo samples at embryonic day E9.5 generated using Stereo-seq. The dataset contains four samples, with 21 583 genes shared across samples and an average of 5126 spots per sample. (C) ST_OFB. Olfactory bulb samples generated using the ST platform. The dataset contains six samples from three mice, with 20 484 genes shared across samples and an average of 216 spots per sample. (D) MERFISH_FC. Mouse frontal cortex samples generated using MERFISH. The dataset contains eight samples from three mice, with 374 genes measured across samples and an average of 11 287 spots per sample. (E) MERFISH_POA. Mouse hypothalamic preoptic area samples generated using MERFISH. The dataset contains five samples, with 155 genes commonly measured across samples and an average of 5663 spots per sample. (F) STARmap_PFC. Mouse prefrontal cortex samples generated using STARmap. The dataset contains three samples with 166 genes shared across samples and an average of 1063 spots per sample.

Figure 3.

Accuracy of spatial domain detection methods in within-individual analysis. (A) Data sparsity across datasets. Each point represents one sample included in the within-individual analysis. Sparsity refers to the proportion of zero values in the gene expression matrix. (B) Accuracy of spatial domain detection. Heatmaps summarize ARI, NMI, and cLISI for each method across all within-individual datasets. Neighborhood-based spatial label refinement was applied to Visium_DLPFC, MERFISH_FC, and MERFISH_POA (see Methods). (C) Performance of multi-sample integration. Heatmaps summarize iLISI and batch ASW for each method across all within-individual datasets. Methods are ranked based on their average scores across individuals. The gray cells indicate missing values due to method incompatibility.

We compared spMosaic with nine existing spatial domain detection methods that support multi-sample analysis (see Methods). Some methods could not be applied to specific datasets due to their input requirements. For example, BayesSpace and PRECAST require count-based input data and therefore cannot be applied to datasets with normalized continuous expression values. GraphST relies on PASTE [22] to align spatial coordinates across samples, which also requires count-based input. In addition, PRECAST failed to run on datasets with relatively small numbers of spots, such as ST_OFB and STARmap_PFC. To evaluate spatial domain detection and sample integration performance, we used several complementary metrics. Accuracy of spatial domain detection relative to annotated spatial domains was assessed using ARI and NMI. Clustering consistency in the embedding space was evaluated using cLISI, while sample integration performance was assessed using iLISI and batch ASW (see Methods). The last three metrics cannot be calculated for BASS or BayesSpace as they do not generate latent embeddings in addition to the spatial domain labels.

We first evaluated clustering accuracy relative to annotated spatial domains, the primary objective of spatial domain detection, and further assessed clustering consistency in the embedding space to determine whether the inferred clusters were well supported by the learned representations. Across all individuals and datasets, spMosaic achieved the best overall clustering accuracy in terms of ARI and NMI (Fig. 3B). spCLUE and BASS generally followed as the second- and third-best methods depending on the metric. Notably, spMosaic achieved the highest mean ARI of 0.62 on the Visium_DLPFC dataset, which had the highest sparsity among all evaluated datasets (Fig. 3A). We further evaluated clustering consistency in the learned embedding space using cLISI (Fig. 3B), which measures how well spots assigned to the same spatial domain are grouped together in the embedding space. PRECAST achieved the highest cLISI values based on its performance on four individuals, while spMosaic ranked second across all 12 individuals, demonstrating its ability to produce coherent embedding structures in which spots belonging to the same spatial domain are positioned close to one another.

While accurate spatial domain identification is our primary objective, we also assessed sample integration as a complementary criterion. spMosaic ranked fifth in iLISI and fourth in batch-adjusted ASW (Fig. 3C). Although not the top performer, its average scores remained close to those of the top methods. Specifically, the average iLISI of spMosaic was 0.92, compared with 0.97 for the top-performing method (PRECAST), and the average ASW was 0.96 for spMosaic versus 0.97 for the top-performing method (GraphST). Methods with higher iLISI values often show reduced clustering accuracy or limited applicability. For example, SPIRAL achieved the second-highest average iLISI but yielded the lowest ARI and NMI among all methods, while PRECAST was restricted to Visium_DLPFC and StereoSeq_MOSTA among the six datasets. This comparison highlights a trade-off between preserving biological structure and achieving strong batch mixing. spMosaic achieved a favorable balance by maintaining competitive integration performance while preserving accurate spatial domain identification.

To further illustrate the performance of different methods across datasets, we visualized the inferred spatial domains of the Visium_DLPFC dataset (three donors) in both the spatial coordinate space and the learned embedding space. On this dataset, spMosaic achieved the highest average ARI of 0.62 among all evaluated methods. For donor 1, spMosaic achieved the highest ARI of 0.58, whereas the other nine methods achieved ARI values ranging from 0.18 to 0.54 (Fig. 4A). Beyond the quantitative improvement, the spatial domains identified by spMosaic showed strong agreement with the reference cortical layer annotations across all four samples. For example, layers 5 and 6 were each characterized by continuous bands; however, methods such as spCLUE and BASS produced fragmented domains in certain samples, splitting these layers into multiple regions and indicating inaccurate recovery of cortical layer topology. The consistency of spMosaic’s spatial domain identification was also reflected in the embedding space (Fig. 4B). The UMAP projections showed that spots from different annotated layers were organized in an interpretable structure, with neighboring layers appearing adjacent, indicating that the learned biological embeddings captured meaningful differences among cortical layers. Furthermore, the embeddings illustrated the trade-off between preserving biological structure and removing sample-specific effects. For methods such as PRECAST and SPIRAL, layer-specific distinctions were largely diminished, even though spots from different samples were well mixed. For the other two donors, spMosaic achieved the highest ARI of 0.72 for donor 2 and the third-highest ARI of 0.56 for donor 3. Consistent with these results, the learned biological embeddings also exhibited clear structured patterns corresponding to different cortical layers (Supplementary Figures S4 and S5).

Figure 4.

Spatial domain detection results for Visium_DLPFC donor 1. (A) Spatial domain assignments for all samples from donor 1, shown in their spatial coordinates. Each column corresponds to a method and each row to a sample. ARI values, computed by pooling spots across samples from this donor, are reported above each method. (B) Embedding visualization. UMAP projections of spot-level embeddings are shown for each method to illustrate how biological variation is represented in the embedding space. Points are colored by reference annotation, predicted spatial domain, and sample identity. Methods that do not produce spot embeddings are excluded.

To assess whether the inferred domains were associated with known cortical gene-expression patterns, we visualized representative layer-specific marker genes previously reported for cortical layers in donor 1 [32] (Fig. 5). Spatial patterns of standardized smoothed expression showed clear layer-specific enrichment consistent across samples. All displayed marker genes were identified as common SVGs by spMosaic, suggesting that the input to the dual-encoder DEC framework contained strong biological signals corresponding to annotated cortical layers. In sample 151507, spMosaic identified a small spatial domain (with label 5) in a corner region (Fig. 4A). Examination of common SVGs revealed several genes with localized enrichment in this region (Supplementary Figure S6), providing gene-expression support for this annotation-discordant pattern. We therefore interpret this region as a candidate spatial subdivision, while noting that additional histological, anatomical, or independent molecular validation would be needed to determine whether it reflects biological refinement or annotation-resolution differences.

Figure 5.

Expression of marker genes in Visium_DLPFC donor 1. (A) Standardized observed gene expression of previously reported layer-specific marker genes [32] across all four samples. Raw count data were normalized by library size, log-transformed, and standardized to zero mean and unit variance within each sample. (B) Standardized smoothed gene expression learned by spMosaic. Smoothed expression values were obtained by sample-wise smoothing using GAMs, and used as input for downstream spatial domain identification. Adjusted P-values (Benjamini–Hochberg correction) are shown in the figure. Values smaller than fall below machine precision and are reported as 0.

We next evaluated performance on an imaging-based dataset, MERFISH_FC, which contained three mice (10, 11, and 12). The best-performing methods varied across mice; overall, spMosaic achieved the second-highest average ARI of 0.64 and the highest average cLISI of 0.92. For mouse 10 with three samples, spMosaic attained the second-highest ARI (0.63), following spCLUE (0.70). Comparison between annotated and predicted spatial domains showed that spMosaic recovered anatomically coherent regions in the frontal cortex (Fig. 6A). In the embedding space, annotated domains and predicted domains exhibited clear correspondence while spots from different samples remained well mixed, indicating effective integration across samples (Fig. 6B). To assess the biological relevance of the identified domains, we further examined representative marker genes among the common SVGs identified by spMosaic (Fig. 7). Across spatial domains, we observed supporting spatial expression patterns among the common SVGs, suggesting that the domains identified by spMosaic were grounded in biologically meaningful spatial signals. For mice 11 and 12 (Supplementary Figures S7 and S8), similar patterns were observed. spMosaic achieved competitive ARI values (0.67 for mouse 11 and 0.60 for mouse 12) and produced spatial domain assignments that aligned well with annotated anatomical structures. In the embedding space, annotated regions remained clearly organized while spots from different samples were well mixed, further demonstrating the robustness of spMosaic for multi-sample integration in MERFISH data.

Figure 6.

Spatial domain detection results for MERFISH_FC mouse 10. (A) Spatial domain assignments for all samples from mouse 10, shown in their spatial coordinates. Each column corresponds to a method and each row to a sample. ARI values, computed by pooling all spots, are reported above each method. (B) Embedding visualization. UMAP projections of spot-level embeddings are shown for each method to illustrate how biological variation is represented in the embedding space. Points are colored by reference annotation, predicted spatial domain, and sample identity. Methods that do not produce spot embeddings are excluded.

Figure 7.

Expression of marker genes in MERFISH_FC mouse 10. (A) Standardized observed expression of representative annotated domain marker genes across all samples from mouse 10. To identify these marker genes, for each annotated domain and for each sample separately, we performed a one-versus-rest differential expression analysis restricted to the set of common SVGs. Candidate marker genes were first identified within each sample by requiring (i) significantly higher expression in the target domain than in the remaining domains (Wilcoxon rank-sum test with Benjamini–Hochberg adjusted P <.05), (ii) a positive effect size, and (iii) expression in at least 10% of spots in the target domain. We then aggregated evidence across samples using Fisher’s method to combine P-values for each gene within each domain. Genes were subsequently ranked based on the adjusted combined significance and their average effect size across samples, and the top-ranked genes were reported as representative domain markers. Original gene expression data were standardized to zero mean and unit variance within each sample. (B) Standardized smoothed gene expression learned by spMosaic. Smoothed expression values were obtained by sample-wise smoothing using GAMs, and used as input for downstream spatial domain identification. The adjusted P-values (Benjamini–Hochberg correction) are shown in the figure. Values smaller than fall below machine precision and are reported as 0.

We next evaluated performance on two additional sequencing-based datasets, ST_OFB and StereoSeq_MOSTA, to assess robustness across different tissue structures and experimental settings. For the ST_OFB dataset, spMosaic achieved competitive clustering performance, with ARIs of 0.56 and 0.58 in mice A10 and A11, respectively, ranking second among evaluated methods in both cases (Supplementary Figures S9 and S10). Its inferred spatial domains exhibited coherent organization consistent with the layered structure of olfactory bulbs. In contrast, mouse A12 was substantially more challenging for all methods (Supplementary Figure S11). The highest ARI was achieved by BayesSpace (0.37), while spMosaic yielded an ARI of 0.22, ranking fourth. In particular, the middle cell layer (MCL) and granule cell layer (GCL) were difficult to separate across methods. This reduced separability was also reflected in the embedding space, where these layers showed weaker distinction, suggesting limited transcriptomic differences between these regions. For the StereoSeq_MOSTA dataset, all evaluated methods showed difficulty in recovering high-resolution annotated spatial domain structures (Supplementary Figure S12). Among the compared methods, BASS achieved the highest overall ARI of 0.30, and spMosaic obtained an ARI of 0.25. The reduced performance likely reflected intrinsic challenges of this dataset. The annotated domain structures are highly complex, with some domains (e.g. cavity and mesenchyme) spanning multiple spatial subregions. In addition, the annotated domains varied across samples; certain domains appeared only in specific samples, such as pancreas primordium and notochord in sample S2, and branchial arch in samples S1 and S3. Taken together, these results highlight that performance varies with dataset complexity.

For the remaining two imaging-based datasets, MERFISH_POA and STARmap_PFC, spMosaic still demonstrated strong performance in both spatial domain identification and sample integration. For the MERFISH_POA dataset (Supplementary Figure S13), clustering performance was relatively comparable across methods, with BASS achieving the highest ARI (0.57), followed by spCLUE (0.50) and spMosaic (0.48). The spatial domains identified by spMosaic remained anatomically coherent across samples and preserved the major annotated regions. In the embedding space, annotated regions formed organized structures that aligned well with the inferred domains, while spots from different samples were well mixed, indicating effective integration. For the STARmap_PFC dataset (Supplementary Figure S14), spMosaic achieved the highest ARI (0.78). The inferred domains closely matched the annotated cortical layer structures across samples. In the embedding space, the arrangement of clusters also preserved the spatial topology of cortical layers, whereas this organization was less apparent in embeddings produced by other methods.

To better understand how gene selection influences cross-sample spatial domain detection, we examined the overlap between sample-specific SVGs and retained common SVGs identified by spMosaic (Supplementary Figures S15 and S16). For sequencing-based datasets (10x Visium, Stereo-seq, and ST), common SVGs constituted between 10% and 66% of all measured genes. In contrast, for imaging-based datasets (MERFISH and STARmap), common SVGs accounted for ∼75% of measured genes on average. This difference likely reflected the curated design of gene panels in imaging-based platforms, where genes were preselected for biological relevance. Consistent with this observation, imaging-based datasets exhibited higher clustering performance, with an average ARI of 0.63 across individuals for spMosaic, compared with 0.49 for sequencing-based datasets. This suggested that curated gene panels might facilitate more stable cross-sample spatial domain identification.

We next examined the proportion of common SVGs relative to sample-specific SVGs within each individual to assess cross-sample reproducibility. Across all datasets, common SVGs comprised the majority of SVGs identified in individual samples, indicating that spatially variable expression patterns are largely reproducible across samples from the same individual. However, the degree of overlap varied by dataset. The overlap was lowest in Visium_DLPFC and ST_OFB, where common SVGs accounted for ∼55% to 66% of sample-specific SVGs, indicating greater sample-to-sample heterogeneity. In contrast, StereoSeq_MOSTA, STARmap_PFC, and MERFISH_POA showed very high overlap, with >94% of sample-specific SVGs retained as common SVGs, which is consistent with highly stable spatial patterns across samples. MERFISH_FC exhibited an intermediate-to-high level of overlap, ranging from 82% to 96% across individuals.

Across-individual spatial domain detection

In contrast to the within-individual analysis, where spatial domain detection is performed separately for each individual, the across-individual setting requires jointly identifying spatial domains across samples from different individuals. This introduces additional challenges due to inter-individual variability, including differences in spatial organization, gene expression patterns, and technical effects, making it necessary to balance biological alignment with robustness to sample heterogeneity. For the across-individual analysis, we evaluated spMosaic on four multi-individual datasets. The first dataset was Visium_DLPFC (Fig. 2A), where the 12 samples from three donors used in the within-individual analysis were now jointly analyzed. The second dataset was StereoSeq_MOSTA (Fig. 8A), which extended the within-individual setting by including samples from two mouse embryos at embryonic day E9.5. The third dataset was ST_OFB, where the same six samples from three mice as in the within-individual analysis were jointly analyzed (Fig. 2C). The fourth dataset was MERFISH_FC (Fig. 2D), where the same eight samples were analyzed across mice.

Figure 8.

Accuracy of spatial domain detection methods in across-individual analysis. (A) StereoSeq_MOSTA dataset for across-individual analysis. Generated using Stereo-seq, this dataset contains five samples from two mouse embryos at embryonic day E9.5. While gene panels differ across samples, 21 440 genes are shared across all samples. (B) Quantitative evaluation of spatial domain detection and sample integration. Heatmaps summarize clustering accuracy metrics (ARI, NMI, and cLISI) and sample integration metrics (iLISI and batch ASW) for each method across all four across-individual datasets. Methods are ordered according to their average performance across datasets. The gray cells indicate that a method is not applicable to a given dataset. Neighborhood-based spatial label refinement was applied to the Visium_DLPFC and MERFISH_FC datasets (see Methods). (C-D) Representative spatial domain visualizations for Visium_DLPFC and MERFISH_FC datasets, respectively. For each dataset, one sample is shown for each individual. The ARI computed by pooling all spots across individuals is reported above each method. (E) Comparison of sample-level ARIs between within-individual and across-individual analyses.

For comparison, eight spatial domain identification methods were applied alongside spMosaic. GraphST was excluded from this analysis because its integration strategy is designed for vertically or horizontally aligned tissue sections, which does not apply to samples from independent individuals. Across all datasets, spMosaic ranked first in ARI and NMI and second to DeepST in cLISI (Fig. 8B). In terms of sample integration performance, spMosaic ranked third in iLISI and first in batch ASW. Notably, spMosaic consistently ranked among the top three methods across all evaluation metrics, demonstrating robust performance in identifying spatial domains across individuals while maintaining competitive batch mixing. These results demonstrated a balanced trade-off between preserving biological structure (the primary objective) and achieving effective integration across samples (a secondary objective). Among the competing methods, several also showed practical limitations in scalability and data compatibility. BayesSpace and PRECAST encountered the same issues as in the within-individual analysis, as neither could be applied to non-count datasets. PRECAST failed on the ST_OFB dataset because the number of spots per sample was too limited for model fitting. Scalability limitations were observed for deep learning–based methods. DeepST failed on the Visium_DLPFC and MERFISH_FC datasets due to out-of-memory errors, and STAligner similarly failed on MERFISH_FC because of memory constraints.

When examining individual datasets, spMosaic achieved the highest overall ARI (0.50) on the Visium_DLPFC dataset. PRECAST and STAligner ranked second (ARI = 0.42). Spatial maps of the identified domains (Fig. 8C and Supplementary Figure S17) showed that spMosaic produced consistent cortical layer structures that aligned well with the annotations across samples. For samples 151507 (donor 1) and 151675 (donor 3), spMosaic achieved the second-highest ARI: 0.54 compared with 0.55 for STAligner in sample 151507, and 0.53 compared with 0.58 for STAligner in sample 151675. Despite these small differences, spMosaic produced smoother and more spatially consistent domains that better matched the annotated laminar structure, whereas STAligner exhibited more fragmented patterns, particularly in layers 2 and 5. For sample 151672 (donor 2), spMosaic achieved the highest ARI of 0.58 and more clearly recovered white matter while separating layers 5 and 6. In contrast, several other methods failed to distinguish these layers from each other. In the embedding space, the representations formed a coherent layered organization, with neighboring layers arranged adjacently (Supplementary Figure S18). This structure was consistent across donors while maintaining effective sample mixing, further supporting that the learned embeddings captured biologically meaningful variation across the 12 jointly analyzed samples.

Consistent with the within-individual analysis, the StereoSeq_MOSTA dataset remained challenging due to the complex and sample-specific nature of the spatial domains. In the across-individual analysis, these challenges were further amplified, as domains must be jointly identified across heterogeneous samples. For example, in the within-individual analysis, the four Embryo 2 samples contained a total of 16 annotated spatial domains. In contrast, incorporating Embryo 1 in the across-individual analysis increased the total number of domains to 18, with the dermomyotome and neural crest appearing only in Embryo 1. As a result, all evaluated methods showed limited accuracy in recovering the annotated structures (Supplementary Figures S19 and S20). Among the compared methods, BASS achieved the highest overall ARI of 0.25, while spMosaic attained an ARI of 0.21. Even though spMosaic did not rank among the top methods by overall ARI, its performance remained comparable. In addition, spMosaic successfully recovered several biologically meaningful domains, including the heart, primitive gut tube, and lung primordium. For the heart, spMosaic achieved an F1 score of 0.76 (precision = 0.86, recall = 0.68); for the primitive gut tube, an F1 score of 0.52 (precision = 0.45, recall = 0.63); and for the lung primordium, an F1 score of 0.44 (precision = 0.41, recall = 0.46). These results placed spMosaic among the top-performing methods for these evaluated domains.

In the ST_OFB dataset, spMosaic achieved an overall ARI of 0.50, second only to BayesSpace (ARI = 0.51), while DeepST ranked third with an ARI of 0.33 (Supplementary Figure S21). Both spMosaic and BayesSpace substantially outperformed the remaining methods. One contributing challenge in this dataset was the relatively small number of spots per sample (mean = 216), which made it difficult to incorporate spatial neighborhood information without oversmoothing. Consistent with this, methods with lower ARI values tended to identify larger domains that incorporated multiple annotated regions. UMAP visualizations further showed that spMosaic preserved biologically meaningful structure while limiting individual-driven separation (Supplementary Figure S22). Together, these results indicated that spMosaic remained robust when spatial smoothing must be carefully balanced against domain resolution under limited sample sizes.

For the MERFISH_FC dataset, spMosaic achieved the highest ARI (0.65) across samples from three mice, outperforming BASS (ARI = 0.60) and spCLUE (ARI = 0.58). The frontal cortex slices exhibited heterogeneous spatial coverage across samples, with some sections capturing bilaterally symmetric structures while others included only unilateral or partial regions. Despite this variability, comparison between annotated and predicted spatial domains showed that spMosaic produced domains that aligned well with annotated cortical regions (Fig. 8D and Supplementary Figure S23). In the embedding space, the latent representations learned by spMosaic showed clear organization of annotated domains while maintaining effective integration across samples (Supplementary Figure S24). Notably, spMosaic preserved the transcriptomic distinction of pia mater spots in the embedding space, whereas this region was less clearly distinguished by other methods.

We further compared the retained common SVGs used in across-individual analysis with the sample-wise SVGs in each dataset (Supplementary Figure S25). Overall, the proportion of common SVGs relative to the sample-wise SVGs was lower in the across-individual analysis than in the within-individual analysis, indicating greater heterogeneity across individuals. For Visium_DLPFC, common SVGs represented ∼36% to 44% of the sample-wise SVGs in the across-individual setting, compared with 55% to 66% in the within-individual analysis. For ST_OFB, this proportion further decreased to ∼16% to 18%, whereas within-individual common SVGs accounted for the majority of the sample-wise SVGs. MERFISH_FC showed an intermediate level of retention, with common SVGs comprising ∼58% to 68% of the sample-wise SVGs across individuals. In contrast, StereoSeq_MOSTA remained highly stable, with the proportion staying near 98% across samples. Together, these results indicated that common SVGs captured an important proportion of spatial variation in gene expression across samples and individuals, while also reflecting increased heterogeneity in the across-individual setting.

To assess how joint analysis across individuals affects clustering performance at the sample level, we compared sample-wise ARI between within-individual and across-individual analyses for spMosaic (Fig. 8E). Across-individual analysis improved the average ARI for the ST_OFB dataset (from 0.44 to 0.54) and the MERFISH_FC dataset (from 0.64 to 0.66), suggesting that joint analysis could enhance performance in certain settings. The improvement was particularly evident for mouse A12 in the ST_OFB dataset, which was substantially more challenging in the within-individual analysis. In the within-individual setting, spMosaic achieved an ARI of 0.22 for mouse A12, with weaker separation between the MCL and GCL. In the across-individual analysis, the individual-wise ARI for mouse A12 increased to 0.40. To further investigate this improvement, we compared the feature sets used in the within-individual and across-individual analyses. In the ST_OFB dataset, the 688 common SVGs retained for across-individual analysis were a subset of the 2250 within-individual SVGs selected for mouse A12 (Supplementary Figure S26A). Restricting the within-individual analysis of individual A12 samples to these 688 genes substantially improved ARI (Rep23: 0.23 to 0.39; Rep24: 0.22 to 0.31). Joint across-individual analysis further improved ARI (Rep23: 0.23 to 0.42; Rep24: 0.22 to 0.37), with external plexiform layer and GCL structures becoming more clearly delineated (Supplementary Figure S26B).

In contrast, for the Visium_DLPFC and StereoSeq_MOSTA datasets, sample-wise ARI was slightly lower in the across-individual setting. In particular, samples 151669 and 151670 from donor 2 in Visium_DLPFC showed noticeably reduced ARI under across-individual analysis. To investigate the reduced ARI observed for Visium_DLPFC in the across-individual analysis, we compared the common SVG sets used for domain detection. The within-individual analysis retained 4419 common SVGs for donor 2, while the across-individual analysis retained a more restrictive set of 2459 common SVGs. This across-individual SVG set was entirely contained within the SVG set identified for donor 2 alone (Supplementary Figure S27A). When restricting the analysis to the four donor 2 samples using the 2459 across-individual SVGs, ARI showed only limited improvement relative to the across-individual analysis (Supplementary Figure S27B), indicating that gene selection is not the primary driver of the observed performance differences. The ARI decrease was largely associated with identification of spatial domain 1. Notably, this structure was also detected by several other methods such as BayesSpace and spCLUE (Supplementary Figure S17), suggesting that it may represent a reproducible biological pattern. Using the 2459 shared SVGs, we identified marker genes for spatial domain 1. The top three marker genes, HPCAL1, CAMK2N1, and ENC1, exhibited spatial expression patterns consistent with this domain not only in donor 2 but also across other individuals (Supplementary Figure S27C–E). These results suggested that the annotation-discordant domain was supported by reproducible spatial gene expression patterns. In addition, across-individual integration enabled clearer identification of white matter in samples 151669 and 151670, which was less well resolved in within-individual analysis. These findings indicate that some of the observed performance differences may reflect spatial structures that are not fully represented in the annotations.

Taken together, the across-individual benchmark results indicated that spMosaic performed robustly across datasets for multi-individual spatial domain detection. While ARI might decrease slightly in certain datasets relative to within-individual analysis, gene-level evidence suggested that cross-individual integration can help nominate spatial patterns that are not fully captured by the original annotations. By leveraging common SVG selection and batch-aware embedding learning, spMosaic allowed information to be shared across individuals, which may contribute to improved reproducibility and biological interpretability of spatial domain identification.

As an additional robustness analysis, we evaluated the sensitivity of spMosaic to cluster-number specification. In the main benchmark analyses, the number of clusters was set to the annotated number of spatial domains to enable controlled comparisons across methods. When the specified number of clusters was moderately under- or over-specified by one or two clusters, spMosaic remained competitive across both within-individual and across-individual analyses (Supplementary Figure S28). We also evaluated a simple internal strategy in which the cluster number was selected from a candidate range using ASW based on the biological embeddings. The ARI values obtained using the selected cluster number were generally close to those obtained using the annotated number of domains, suggesting that spMosaic was relatively robust when the exact number of domains was not known (Supplementary Figure S29). However, in STARmap_PFC, the silhouette-based criterion selected a smaller cluster number that produced under-segmentation, indicating that internal criteria should be interpreted together with biological plausibility and spatial coherence of the inferred domains (Supplementary Figure S30). Very small numbers of clusters should be interpreted cautiously when they merge visually or biologically distinct spatial structures.

Scalability and computational efficiency

To evaluate scalability in the across-individual setting, we applied spMosaic to a substantially larger dataset. The full MERFISH_FC dataset contains 378 918 spots from 31 samples across 12 mice (Fig. 9 and Supplementary Figures S31 and S32). The size of this dataset exceeds that of most commonly benchmarked SRT datasets and poses significant computational challenges for many existing integration methods [36]. Despite this scale, spMosaic successfully identified spatial domains across all mice, achieving an overall ARI of 0.63 and an NMI of 0.60. For the LISI and ASW metrics, due to the computational cost on the full dataset, we randomly subsampled 5000 spots and repeated the evaluation 50 times. The average cLISI was 0.88, indicating stable organization of spatial domains in the embedding space. The average iLISI and batch ASW were 0.74 and 0.93, respectively, demonstrating effective sample mixing across the 31 samples and 12 mice.

Figure 9.

Across-individual analysis of the full MERFISH_FC data. Spatial domain identification was performed on the complete MERFISH_FC dataset comprising 31 samples from 12 mice (378 918 spots in total). (A) Annotated tissue regions; (B) Spatial domains identified by spMosaic.

Among the methods that successfully processed the smaller MERFISH_FC subset (mice 10–12) in across-individual analysis, including BASS, spCLUE, SPIRAL, and SEDR, none were able to complete the analysis on the full MERFISH_FC dataset under our computational settings (see Methods). Specifically, spCLUE and SEDR failed due to out-of-memory errors on GPUs. SPIRAL failed due to internal errors when handling intermediate files across 31 samples, and BASS failed on a CPU with 128 GB of RAM. In contrast, spMosaic completed the full analysis within 20 min. These results demonstrated the practical scalability of spMosaic for large multi-individual SRT datasets, while reflecting the combined effects of algorithmic design, software implementation, and hardware support.

We further quantified runtime and peak memory usage across datasets under practical execution settings for each method (Fig. 10; see Methods). For within-individual analysis (Fig. 10A), both runtime and memory usage for spMosaic tended to increase with the number of spots per individual. However, for larger datasets such as MERFISH_POA and MERFISH_FC, peak RAM usage remained relatively stable, and runtime did not increase proportionally. For across-individual analysis (Fig. 10B), even as the total number of spots increased substantially, peak RAM usage for spMosaic remained below 50 GB across all datasets, including the full MERFISH_FC dataset. These results suggested favorable scaling behavior of spMosaic for large multi-individual datasets. To further understand this behavior, we decomposed runtime and memory usage into the two main components of spMosaic (Supplementary Figure S33). The GAM smoothing stage typically accounted for a larger portion of runtime but required less memory, whereas the spatial domain detection based on batch-aware DEC was more memory-intensive.

Figure 10.

Runtime and memory usage in within-individual and across-individual analyses. (A) Within-individual analyses; (B) Across-individual analyses. Within each panel, CPU-based methods are shown in the left column and GPU-accelerated methods in the right column. The top row reports peak RAM usage (MB), and the bottom row reports runtime on the scale in seconds. spMosaic is included in both panels for convenient comparison. For within-individual analyses, datasets are ordered by the average number of spots per individual, and runtime and memory are averaged across individuals when multiple individuals are available. For across-individual analyses, datasets are ordered by the total number of spots across individuals. For each curve plot, points corresponding to dataset-method combinations without valid results are omitted.

To better understand the scalability of spMosaic, we analyzed the computational complexity of the SVG smoothing and detection stage and the spatial domain detection stage. In the first stage, a GAM is fitted independently for each gene within each sample using penalized regression splines [27, 37]. Aggregating across all genes and samples yields a total computational complexity of , where J is the number of genes used for smoothing, K is the number of spline basis functions, I is the total number of spatial spots across samples, and L is the number of samples. In practice, K is fixed and therefore treated as a constant, implying that the spatial smoothing stage scales approximately linearly with respect to both the number of spatial spots and the number of genes (see Supplementary Methods for derivation). In the second stage, we employ a dual-encoder autoencoder followed by DEC refinement [26]. Let M denote the number of common SVGs used as input features, and C the number of clusters. Under fixed network architecture and training settings (i.e. embedding dimension and number of training epochs), the overall computational complexity of this stage is O(IM + IC) (see Supplementary Methods for derivation). These results indicate that spMosaic scales favorably with dataset size, with computational cost growing approximately linearly with respect to both the number of spatial spots and the number of genes.

For comparison, several benchmarked SRT methods (e.g. spCLUE and STAligner) rely on GNNs. Although these methods typically exhibit linear scaling with respect to the number of spatial spots [38, 39], they require explicit construction and storage of spatial graphs as well as repeated neighborhood aggregation across graph edges [40]. These graph operations introduce additional computational cost and memory overhead, particularly for large spatial datasets. In contrast, spMosaic separates spatial smoothing and representation learning into two stages. GAM smoothing is performed on CPU, while neural network training is performed on GPU. This design reduces GPU memory usage and avoids graph-based message passing operations, improving practical scalability for large SRT datasets.

Ablation studies and component contributions

Finally, we conducted ablation studies to evaluate the contribution of each major component of spMosaic to spatial domain identification accuracy. The full spMosaic framework consists of three key components: (i) common SVG smoothing and detection, which denoises gene expression while incorporating local spatial proximity and integrates spatial structure into the signal prior to domain identification; (ii) a dual-encoder autoencoder with batch prediction, which encourages disentanglement of batch-related variation from the biological embedding; and (iii) DEC refinement following autoencoder pretraining, which iteratively updates cluster assignments and reinforces biologically meaningful structure in the embedding space.

To quantify the impact of each component, we constructed three reduced models and evaluated them on the same datasets used in the previous within-individual and across-individual analyses. In the first reduced model (w/o GAM), we removed the common SVG smoothing and detection step. Instead, for each sample, we selected the top 3000 highly variable genes (HVGs) using Seurat, and retained common HVGs across samples or individuals. The log-scaled counts per million of these HVGs were used as input to the dual-encoder autoencoder for spatial domain identification. In the second reduced model (w/o BatchPred), we removed encoder and the batch prediction branch entirely. Under this setting, no explicit batch disentanglement was performed, and the model relied solely on the biological encoder for representation learning. In the third reduced model (w/o DEC), we removed the DEC refinement stage. After autoencoder pretraining, we directly performed clustering on the learned biological embeddings using the mclust algorithm without iterative refinement.

We summarized the ARI values for each dataset under the reduced models and compared them with those from the full spMosaic framework (Supplementary Figure S34). Overall, the full spMosaic model achieved the highest average ARI across all datasets (0.54), followed by the model without DEC refinement (w/o DEC; ARI = 0.51), the model without the batch prediction module (w/o BatchPred; ARI = 0.48), and the model without GAM smoothing and SVG detection (w/o GAM; ARI = 0.35). This ranking reflected the relative contribution of each component to accurate spatial domain identification. In particular, removing GAM smoothing and SVG detection led to the largest performance drop, indicating that this step played a critical role in refining biologically meaningful spatial signals and preserving spatial continuity before domain identification. Together, these results suggested that the three components acted in a complementary manner, with GAM-based smoothing providing the strongest contribution and the batch prediction and DEC modules further enhancing clustering accuracy and robustness.

Motivated by the ablation results showing that GAM smoothing and common SVG selection were important contributors to spMosaic performance, we further evaluated whether the full framework provided benefits beyond applying standard dimension reduction or integration methods to the same GAM-smoothed features. Specifically, we compared spMosaic with three simplified pipelines using the GAM-smoothed expression matrix of common SVGs as input: PCA using the top 32 PCs, Harmony batch correction applied to the top 32 PCs, and Seurat integration followed by PCA. For all three baselines, we applied the same mclust clustering procedure and, where applicable, the same neighborhood-based label refinement used in the main benchmark. As shown in Supplementary Figure S35, the average ARI values of PCA, Harmony, and Seurat were 0.37, 0.40, and 0.35, respectively, all lower than that of spMosaic. These results indicated that, beyond GAM-based smoothing and common SVG selection, the batch-aware dual-encoder representation learning framework further improved spatial domain identification compared with standard dimension reduction or integration pipelines applied to the same features.

Beyond the component-wise ablation study, we further evaluated alternative strategies for reducing sample-associated information in the biological embedding. We considered two variants of spMosaic. First, we constructed a DANN-based variant by removing the batch encoder and introducing a batch discriminator that operates directly on the biological embeddings through a gradient-reversal layer. DANN provides a commonly used strategy for learning batch-invariant representations and has been adopted by several SRT methods, including DeepST and SPIRAL. Second, we added a Hilbert–Schmidt Independence Criterion (HSIC) penalty between the biological embeddings and the sample labels to construct an HSIC-regularized variant [41]. This penalty directly discourages dependence between and sample identity. For both variants, we evaluated a predefined range of regularization weights and selected the value that achieved the best downstream spatial-domain ARI. The selected weights were for the DANN adversarial loss (Supplementary Figure S36) and for the HSIC penalty (Supplementary Figure S37). All variants subsequently underwent the same DEC refinement and downstream clustering procedure.

To assess whether these strategies preserved spatial-domain information in the biological embeddings while reducing sample-associated variation, we compared the learned representations from spMosaic, the DANN-based variant, the HSIC-regularized variant, and the GAM–PCA baseline (Supplementary Figure S38). For the DANN and HSIC variants, we used the best-performing regularization weights selected from the corresponding sensitivity analyses, providing a favorable setting for these alternatives. To evaluate preservation of biological information, we applied mclust to each representation using the annotated number of spatial domains and calculated ARI against the reference annotations. The biological embeddings of spMosaic achieved the highest average spatial-domain ARI of 0.54, followed by the biological embeddings of the HSIC variant and the DANN variant, and by GAM–PCA, with average ARIs of 0.48, 0.46, and 0.37, respectively. In contrast, the batch embeddings of spMosaic and the HSIC variant yielded much lower average spatial-domain ARIs of 0.14 and 0.12. We then assessed sample-associated structure by clustering each representation into the known number of samples and calculating ARI against the sample labels. The batch embeddings of the HSIC variant and spMosaic showed the strongest agreement with sample identity, with average ARIs of 0.73 and 0.67, respectively, whereas the average sample-label ARIs of the biological embeddings from spMosaic, the HSIC variant, and the DANN variant were all below 0.01. These results indicated that sample-associated structure was concentrated primarily in the batch embeddings, whereas spatial-domain information was preserved more strongly in the biological embeddings. Importantly, spMosaic achieved higher spatial-domain accuracy than both the DANN and HSIC alternatives even under favorable regularization settings for these methods. The more direct suppression of sample-associated information by the HSIC penalty was accompanied by reduced spatial-domain accuracy relative to the original spMosaic model, consistent with a trade-off between sample-effect removal and biological signal preservation. We therefore interpret the dual-encoder architecture as providing a useful, although not complete, separation of sample-associated and spatial biological information.

Discussion

In this work, we introduce spMosaic as a scalable and robust framework for integrating SRT data across multiple samples, regardless of whether the samples originate from the same individual or from different individuals. The framework consists of two main stages. In the first stage, GAMs are used to smooth gene expression along spatial coordinates, integrating spatial proximity with transcriptomic signals. This step enhances local spatial continuity and stabilizes downstream domain detection. Owing to the linear computational complexity of the GAM formulation with respect to gene and spot numbers, this stage provides an efficient and scalable approach for incorporating spatial information. In the second stage, spMosaic employs a batch-aware dual-encoder autoencoder to learn shared low-dimensional representations across samples while mitigating sample-specific variation. A batch prediction branch encourages sample-associated variation to be captured by dedicated batch embeddings, and a subsequent DEC refinement step sharpens domain boundaries, improving spatial coherence and clustering accuracy. Across both within-individual and across-individual settings, spMosaic consistently achieves strong clustering performance relative to existing methods, while maintaining competitive sample mixing across datasets generated from five distinct SRT technologies. Notably, it maintains scalability and accuracy when applied to a large cohort comprising 378 918 spatial spots from 31 samples across 12 mice, completing the analysis within 20 min.

Beyond scalability and accuracy, an important strength of spMosaic is the interpretability provided by common SVGs and the learned gene expression patterns underlying spatial domain identification. By focusing on genes that consistently exhibit spatial variation across samples, the framework highlights shared spatial gene-expression patterns that contribute to domain structure, enabling inferred domains to be linked to coherent molecular profiles. One example is provided by our analysis of the Visium_DLPFC dataset. In donor 2 (samples 151669–151672), across-individual integration identified an annotation-discordant spatial subdivision with spatially coherent marker-gene expression (Supplementary Figure S27). A similar pattern was identified by multiple computational methods, suggesting that this region may reflect a reproducible transcriptomic pattern rather than a result specific to spMosaic. We therefore interpret this annotation-discordant subdivision as a hypothesis-generating candidate spatial pattern: marker-gene expression and agreement across computational methods provide supportive evidence, but do not replace orthogonal histological, anatomical, or independent molecular validation. More broadly, these results suggest that across-individual integration, guided by common SVGs, can enable samples to borrow information from one another while preserving interpretable gene-level signals. This may help nominate candidate spatial subdivisions supported by coherent expression patterns across individuals, which can then be prioritized for biological validation in follow-up studies.

In addition to its role in interpretability, gene selection also plays a critical role in spatial domain detection performance. Retaining genes that capture biologically informative spatial variation improves the robustness of downstream clustering, and our results show that using common SVGs yielded the strongest overall performance in the within-individual analyses and remained competitive in the across-individual analyses. However, this strategy introduces an inherent trade-off between capturing shared biological structure and preserving sample-specific variation. By restricting analysis to genes that are spatially variable across all samples, common SVG selection emphasizes signals that are consistently expressed and aligned across samples, thereby stabilizing cross-sample domain identification. At the same time, this restriction may exclude genes that exhibit spatial variation only in a subset of samples, potentially limiting the ability to detect sample-specific domains arising from differences in anatomical regions, developmental stages, or conditions. This trade-off suggests an important direction for future work. Rather than relying on a fixed set of shared genes, adaptive strategies that jointly model shared and sample-specific spatial variation may enable more flexible domain identification. For example, extending the framework to incorporate both common SVGs and sample-specific SVGs, or to weight genes based on their degree of cross-sample consistency, could improve sensitivity to heterogeneous spatial patterns while preserving the stability of shared domain structures. In addition, integrating prior biological knowledge, such as marker genes, may further improve the resolution of spatial domain identification while maintaining stability.

While the GAM-based smoothing step effectively enhances spatial coherence in gene expression patterns and exhibits favorable linear scaling with respect to the number of genes and spots, further efficiency gains are desirable as datasets continue to grow in both dimensions. In particular, applying gene-wise smoothing to tens of thousands of genes across large spatial cohorts can still incur substantial cumulative computational cost. Future work could explore more efficient strategies that leverage shared structure across genes. For example, highly correlated genes could be grouped into modules, and smoothing could be performed at the module level rather than for each gene individually. Such an approach may reduce computational burden while retaining key biological signals that drive spatial domain identification. More generally, developing methods that better balance computational efficiency and signal preservation will be important for scaling spatial transcriptomic analysis to increasingly large datasets.

More broadly, computational analysis of SRT data is rapidly expanding, with methods developed for related but distinct objectives, including cell-type or tissue-region annotation, spatial alignment and registration, SVG detection, spatial differential expression, perturbation-associated spatial analysis, and multimodal spatial-omics integration [42–45]. These tasks are closely connected to spatial domain discovery but emphasize different inferential targets. For example, alignment and registration methods establish spatial correspondence across tissue sections or resolutions; multimodal integration methods link spatial transcriptomic profiles with other molecular or imaging measurements; and gene-level inference methods identify spatially variable, condition-associated, or perturbation-associated genes.

spMosaic is designed for a specific setting within this broader landscape: scalable integration of multiple SRT samples to identify shared spatial domains without requiring explicit spatial coordinate alignment. Its outputs can therefore be complementary to methods with other objectives. For example, spatial alignment or multimodal integration may help relate spMosaic domains to histological or proteomic features when such data are available, while spatial differential expression or perturbation-aware analyses may help characterize genes and pathways associated with inferred domains across conditions [46]. Thus, rather than replacing methods developed for alignment, multimodal matching, or gene-level inference, spMosaic provides a domain-discovery layer that can be combined with these approaches depending on the biological question and available data.

Overall, spMosaic provides a scalable and interpretable framework for multi-sample spatial domain discovery. By integrating explicit spatial modeling, informative feature selection, batch-aware representation learning, and clustering refinement, the method enables robust characterization of tissue architecture across heterogeneous samples. As SRT datasets continue to grow in size and complexity, approaches that jointly address biological signal refinement, cross-sample comparability, and computational efficiency will be essential for advancing large-scale spatial analysis.

Key Points

• spMosaic is a computational framework for multi-sample spatial domain discovery that is designed for studies spanning multiple tissue sections, individuals, and/or experimental batches.

• The method combines sample-wise spatial smoothing with generalized additive models and shared spatially variable gene selection to provide an interpretable foundation for downstream integration.

• A batch-aware dual-encoder framework enables joint analysis across samples while preserving biologically meaningful structure and reducing unwanted sample-specific variation.

• spMosaic is scalable to large datasets, including a MERFISH dataset with 378 918 spatial spots from 31 samples across 12 mice. Across datasets from multiple spatial transcriptomics platforms, spMosaic achieves strong clustering performance for accurate spatial domain identification.

Author contributions

Shiwei Fu (Formal analysis, Investigation, Methodology, Software, Writing—original draft, Writing—review & editing), Han Li (Methodology, Writing—review & editing), and Wei V. Li (Conceptualization, Methodology, Supervision, Writing—review & editing)

Conflicts of interest

None declared.

Funding

This work was supported in part by the National Institutes of Health (NIH) NIGMS R35GM142702 (to W.V.L.).

Data availability

All datasets analyzed in this study are publicly available. The Visium_DLPFC dataset was downloaded from https://github.com/LieberInstitute/spatialLIBD. The StereoSeq_MOSTA dataset was downloaded from https://db.cngb.org/stomics/mosta/download/. The ST_OFB dataset was downloaded from https://data.mendeley.com/datasets/6s959w2zyr. The MERFISH_FC dataset was downloaded from https://cellxgene.cziscience.com/collections/31937775-0602-4e52-a799-b6acdd2bac2e. The MERFISH_POA and STARmap_PFC datasets were downloaded from http://sdmbench.drai.cn. The spMosaic package is implemented in Python and is available at https://github.com/ShweiSTAT/spMosaic and https://doi.org/10.5281/zenodo.19801624.