MLN2SVG: domain-aware spatially variable gene detection using contrastive variational autoencoder and multi-level neighbor search

Abstract

Spatial transcriptomics (ST) technologies have transformed our ability to examine gene expression within intact tissues, yet accurately identifying spatially variable genes (SVGs) remains challenging due to spatial heterogeneity, data sparsity, and incomplete modeling of domain-level dependencies. To address these limitations, we propose MLN2SVG, a domain-aware framework that integrates contrastive variational autoencoding with a multi-level neighbor (MLN) search algorithm to jointly learn tissue domains and SVGs. MLN2SVG constructs a weighted spatial graph to capture both local and long-range spatial relationships, employing a deep contrastive variational autoencoder to align augmented and original data representations while preserving biological diversity. The MLN algorithm dynamically expands neighborhood connectivity to mitigate sparsity and enhance domain coherence. Across multiple human and mouse ST datasets, including dorsolateral prefrontal cortex, breast cancer, and brain tissues, MLN2SVG consistently outperformed existing methods in clustering accuracy, robustness, and biological interpretability. Notably, in breast cancer tissues, MLN2SVG uncovers fine-grained spatial organization of tertiary lymphoid structures, delineating region-specific immune architectures spanning intratumoral, tumor-edge, and extratumoral compartments. Through the integration of spatial domain discovery and SVG detection, MLN2SVG delivers a robust and biologically interpretable framework for uncovering the molecular and structural complexity of tissue organization.

Introduction

Recent breakthroughs in genomic technologies have significantly improved our ability to examine gene expression within complex tissues. Although conventional RNA-sequencing methods, such as bulk RNA-seq [1] have provided important insights, they do not capture the spatial context of gene activity within tissues. Spatial transcriptomics (ST) has emerged as a transformative tool, combining gene expression profiling with tissue histology to map gene activity within specific tissue regions. Unlike single-cell RNA sequencing, which provides detailed gene expression data but loses spatial information, ST allows for a more comprehensive understanding of how genes function in their native tissue environments. As this technology advances, it holds great promise for advancing our knowledge of tissue organization, cellular interactions, and disease mechanisms, offering new opportunities for therapeutic development. ST techniques can be divided into two main categories: imaging-based and sequencing-based methods [2]. Imaging approaches, such as smFISH [3], MERFISH [4], and seqFISH [5], offer excellent resolution but are limited in the number of genes they can analyze. However, sequencing-based methods such as 10x Visium [6] and Slide-seq [7] allow higher throughput but lack the ability to resolve gene expression at the single-cell level. Both types of methods come with challenges, including high costs, time requirements, and the need for expert interpretation of complex data, making them less ideal for investigating diseases with intricate gene expression profiles.

ST enables direct mapping of gene expression within intact tissues, revealing spatial gene organization. However, high costs and limited accessibility restrict widespread adoption [8]. In addition, constrained spatial resolution and missing expression values reduce analytical accuracy, underscoring the need for efficient and robust computational methods [9].

Identification of spatially variable genes (SVGs) represents a core analytical task in ST. SVGs exhibit region-dependent expression patterns across tissues or within predefined spatial domains, providing insight into tissue heterogeneity and the biological processes that drive spatial organization [10]. Detecting SVGs also reduces the high dimensionality of ST data and prioritizes genes carrying meaningful biological signals, in a manner analogous to highly variable gene (HVG) selection in single-cell RNA sequencing. Two primary strategies have been developed for SVG identification [11]. One strategy relies exclusively on gene expression values and ignores spatial coordinates, applying feature selection methods adapted from single-cell analysis, including variance-based filtering and probabilistic models such as Poisson or binomial distributions. The second strategy jointly models gene expression and spatial coordinates to identify genes whose expression varies across tissue regions or spatial domains, enabling interpretation within a spatial context. SVG identification provides the basis for downstream analyses such as spatial domain characterization and unsupervised clustering, and it is distinct from later steps focused on spatial visualization or regional expression comparisons [12]. Robust SVG detection therefore supports deeper investigation of tissue architecture, disease mechanisms, and therapeutic targets.

In recent years, SVG identification has become a key analytical task for understanding how gene expression is organized within tissues, motivating a wide range of computational methods. Existing approaches broadly fall into Euclidean space analysis and graph driven modeling. Euclidean methods treat spots as 2D coordinates and evaluate expression variation relative to spatial distance, whereas graph-based methods construct spot networks and model variation through connectivity. Euclidean approaches further include model based kernel methods and model free kernel free strategies.

Kernel-based and model-based approaches define covariance between spatial spots and using a kernel function that typically decreases with increasing Euclidean distance. SpatialDE applies Gaussian process regression to decompose gene expression into spatial and nonspatial components [13], while SpatialDE2 extends modeling flexibility but remains limited in scalability [14]. SPARK employs generalized linear mixed models to capture spatial variation [15], and SPARK-X provides a nonparametric and scalable alternative for high-resolution datasets [16]. SOMDE clusters spatial locations using self organizing maps and fits Gaussian processes within clusters, although computational demands increase for fine grained data. nnSVG introduces nearest neighbor Gaussian processes with gene-specific length scale parameters [17]. Kernel-free or model-free approaches avoid predefined kernels and enable adaptive discovery of spatial patterns. Trendsceek models expression as a marked point process [18]. Deep learning methods such as SPADE integrate image features with ST [19]. HEARTSVG converts 2D expression into 1D series and applies the Portmanteau test with Holm correction [20]. HybridSVG combines global and local spatial autocorrelation measures within an ensemble framework to detect genes with both broad tissue-scale gradients and fine-grained spatial niches [21]. PROST treats gene expression as an image and ranks genes using regional uniformity and Moran’s I [22].

Graph-driven approaches identify SVGs by constructing networks of neighboring spatial spots and analyzing connectivity patterns within tissues. The effectiveness of these methods depends on how accurately the graph represents underlying tissue organization. Spatial autocorrelation statistics such as Moran’s I [23] and Geary’s C [24] are commonly used, although scalability remains challenging. Platforms including MERINGUE and Giotto [25] streamline analysis, while Hotspot [26] and SINFONIA [27] focus on localized spatial patterns.

Identifying spatially coherent domains in ST is key to understanding tissue organization and gene expression dynamics. Traditional clustering methods like k-means and Louvain often neglect spatial continuity, leading to biologically inconsistent groupings. To address this, probabilistic models such as Hidden Markov Random Fields [28] and Bayesian frameworks [29] enforce spatial coherence by clustering adjacent regions with similar profiles, while graph-based deep learning methods like GraphST [30], STAGATE [31], and SpaGCN [32] leverage spatial dependencies to learn meaningful latent representations. GraphST employs graph self-supervised contrastive learning for spatial clustering, integration, and deconvolution. While it enhances representation learning through contrastive objectives, it does not explicitly address domain-aware SVG detection. Recent approaches further refine domain identification through contrastive learning, as demonstrated by SpaGIC [33] and ConGI [34], and additional frameworks such as SCGDL [35], DeepST [36], and stLearn [37] enhance clustering accuracy by incorporating histological features.

Accurate detection of SVGs in complex tissue architectures remains challenging despite advances in ST analysis. Existing methods suffer from several limitations. Spatial domain identification and SVG detection are often treated as separate tasks, producing genes with weak spatial localization. Sparse expression data are inadequately integrated with spatial context, limiting the use of multi-level spatial dependencies and noise suppression. Many approaches rely on static neighborhood definitions, reducing adaptability to dynamic tissues such as developing embryos. Robustness to rotational variance and multi-slice misalignment is also limited, constraining cross dataset generalizability. Although graph-based deep learning methods such as STAGATE and SpaGCN model spatial dependencies, they frequently overlook domain-specific gene–tissue interactions. Emphasis on global summaries or rigid grids further leads to false negatives and biologically implausible mappings.

Although graph neural networks, variational autoencoders, and contrastive learning strategies have been applied in ST for domain identification and representation learning, most existing frameworks rely on fixed neighborhood graphs and treat spatial domain detection and spatially variable gene (SVG) identification as separate tasks. Graph-based methods such as STAGATE, SpaGCN, and GraphST primarily focus on clustering and latent embedding learning, while statistical approaches including SPARK-X, SpatialDE, and PROST emphasize global spatial variation without explicitly integrating learned domain structures. These approaches may therefore underutilize higher-order spatial relationships and domain-specific expression patterns. To address these limitations, we propose MLN2SVG, a domain-aware framework that integrates adaptive multi-level neighbor (MLN) refinement with contrastive variational graph modeling to jointly learn spatial domains and detect domain-specific SVGs within a unified optimization framework.

To address these gaps, in this study we propose MLN2SVG, a domain-aware spatially variable gene detection framework that combines contrastive variational autoencoding with a novel MLN search algorithm. Unlike conventional methods, which treat these tasks separately or rely on static neighborhood structures, MLN2SVG dynamically integrates spatial context, contrastive representation learning, and multi-scale graph modeling to capture complex tissue heterogeneity. The major contributions of this work are summarized as follows.

• We propose MLN2SVG, which jointly learns spatial domains and SVGs within a single architecture by coupling graph-based variational autoencoding with contrastive learning, enabling biologically coherent and interpretable spatial gene discovery.

• A novel MLN algorithm is introduced to dynamically capture higher-order spatial relationships, bridging local and long-range dependencies across tissue regions and mitigating data sparsity and noise in high-resolution ST.

• Comprehensive evaluations on multiple human and mouse ST datasets demonstrate that MLN2SVG consistently outperforms existing methods in clustering accuracy, robustness to spatial variation, and biological interpretability.

Materials and methods

Overview of the MLN2SVG framework

Our proposed framework, MLN2SVG, outlines the process for detecting SVGs in a domain-aware manner, as depicted in the workflow illustration (Fig. 1). MLN2SVG begins by leveraging ST data to construct a weighted graph that captures both local interactions and broader tissue context. A contrastive learning strategy is then applied to compare the original and augmented gene expression profiles, ensuring that the latent space robustly reflects biologically meaningful differences while maintaining tissue heterogeneity. The framework employs a deep contrastive variational autoencoder that uses a graph convolutional encoder to derive robust embeddings and a graph deconvolutional decoder to reconstruct the expression profiles, thereby mitigating oversmoothing issues. To enhance this representation, our novel MLN algorithm dynamically incorporates higher-order neighbors, alleviating sparsity and improving connectivity among spatial spots. Finally, these refined embeddings are used for domain-specific SVG detection via a reconstruction loss, which isolates spatially coherent domains and accurately identifies genes with consistent spatial expression patterns.

Figure 1.

The MLN2SVG framework for SVG detection. (A) Input data consisting of ST expression matrices and spatial coordinates. (B) Graph construction and augmentation: a weighted spatial graph is constructed from multimodal data, and gene expression is augmented by aggregating information from neighboring spots to create a spatial contrastive structure (). (C) Model architecture: MLN2SVG employs a graph convolutional encoder and deep contrastive variational autoencoder to generate low-dimensional representations (, ). The model reconstructs data through graph convolutional decoders while minimizing reconstruction and contrastive losses. (D) Domain/clustering prediction: reconstructed gene expression enables spatial domain identification and clustering of spots based on similar expression patterns. (E) Downstream analysis: MLN2SVG identifies biologically relevant SVGs by leveraging higher-order neighborhood relationships and reconstructed graph structures.

Data collection and preprocessing

Evaluation of MLN2SVG was conducted across multiple publicly available ST datasets generated from diverse platforms, including 10x Genomics Visium, Stereo-seq, STARmap, and SeqFISH (Supplementary Table S1). The 10x Visium datasets include 12 human dorsolateral prefrontal cortex (DLPFC) sections (slices 151507–151676), one human breast cancer tissue section (Block A Section 1), and one mouse brain coronal section. Additional high-resolution datasets include a Stereo-seq mouse brain sample (SS200000135TL_D1), a STARmap mouse visual cortex dataset, and a SeqFISH mouse embryo (E8.5) dataset. Together, these datasets encompass a wide range of spatial resolutions and gene coverage (351–36 601 genes), spanning both human and mouse samples under healthy and disease conditions. This diversity enables comprehensive evaluation of clustering accuracy, spatial domain identification, and SVG detection across biological systems and experimental platforms.

Preprocessing pipeline

All datasets were processed using a standardized pipeline. Genes expressed in fewer than three spots or cells were removed, and HVGs were selected using Scanpy. Expression matrices were normalized, log transformed, and scaled. Datasets retaining >3000 genes used the top 3000 HVGs, whereas datasets with fewer genes used all retained genes as input to MLN2SVG.

Graph construction and spatial augmentation

MLN2SVG starts from a gene expression matrix , where denotes spatial spots and denotes genes. A degree matrix is defined with . Expression profiles are refined through neighborhood aggregation using , improving signal to noise for downstream analysis. Spatial structure is incorporated through the augmented matrix in Equation (1),

(1)

where controls neighborhood influence. ST data are modeled as a weighted graph , with edge weights derived from Euclidean distances between spatial coordinates. A KNN graph defines using spatial coordinates, with for 10x Visium and for Slide seqV2 and Stereo seq. PCA projects expression to , and cosine distances between neighbors yield similarities

(2)

Row normalization produces , enforcing and as in Equation 3:

(3)

Graph-based encoder–decoder architecture

Spatial relationships in gene expression are modeled using a graph-based contrastive variational autoencoder that captures biological variation and local spatial structure. The framework combines a graph convolutional encoder, a graph deconvolutional decoder, and a contrastive variational module to learn robust latent representations aligned across original and augmented spatial inputs.

Graph convolution and deconvolution

We decompose as , where contains the eigenvectors, and is the diagonal matrix of eigenvalues. The normalized Laplacian matrix is defined as

(4)

where is the adjacency matrix and is the degree matrix. We will use Equation (4) throughout our analysis.

Graph convolution

Spectral convolution of the signals and is performed as

(6)

We adopt for the graph convolution. As indicated in Equation 5 and Equation 6, this choice corresponds to a linear spectral filter defined on the graph Laplacian eigenvalues.

Graph deconvolution

The reconstructed signals are obtained via deconvolution as

(8)

To minimize the reconstruction error under stochastic noise, the deconvolution filter is defined as

(9)

where and denote the variances of the noise terms. As shown in Equation 7, Equation 8, and Equation 9, the reconstruction is achieved through a Wiener-type spectral deconvolution that balances signal fidelity and noise suppression.

Encoder (graph convolutional network layers)

A graph convolutional network (GCN) encoder extracts spatially informed representations from the original expression matrix and the augmented matrix . Graph convolution incorporates local topology through Laplacian smoothing and captures multi-level spatial features via recursive neighborhood aggregation. For each layer , the encoder outputs and are defined as

(11)

where each layer applies spectral graph convolution followed by a layer specific nonlinear activation (Equations 10 and 11).

Decoder (graph deconvolutional network layers)

Graph convolution can oversmooth fine scale expression patterns and reduce global variability, motivating a graph deconvolutional network (GDN) decoder. The decoder counteracts encoder smoothing and improves reconstruction by approximating an inverse operation. For layers , the decoder produces and via Wiener type spectral deconvolution:

(12)

(13)

Contrastive loss and variational objective

Balanced latent representations are learned by incorporating deep contrastive learning as a soft constraint during training. This strategy aligns latent features from the original expression matrix and its spatially augmented counterpart while preserving their distinctive patterns. Spatial consistency is enforced using a contrastive loss

(14)

which penalizes discrepancies across all encoder and decoder layers. The variational objectives for and combine reconstruction and Kullback–Leibler divergence terms, as defined in Equations 14 and 15. The overall objective

(15)

jointly optimizes reconstruction fidelity and spatial consistency, where controls the influence of the contrastive term.

Spatial domain prediction

Spatial tissue domains were delineated by computing the average of the final layer latent embeddings from the original and augmented inputs, expressed as , followed by clustering with the Gaussian mixture based Mclust algorithm. The reconstructed augmented gene expression matrix was used to denoise ST profiles, thereby improving signal clarity and domain separability.

Evaluation criteria

Clustering performance

Clustering performance of MLN2SVG in identifying spatial domains was quantitatively evaluated using two widely adopted metrics: adjusted Rand index (ARI) and normalized mutual information (NMI). These metrics assess the consistency between predicted cluster assignments and ground truth labels from complementary statistical perspectives.

Adjusted Rand Index

The ARI measures the similarity between two clustering results while correcting for chance agreement. It is defined as

(16)

As shown in Equation 16, denotes the number of samples shared between the th predicted cluster and the th ground truth cluster, and represent cluster sizes, and denotes the total number of samples.

Normalized Mutual Information

NMI quantifies the mutual dependence between predicted and true cluster assignments and is defined as

(17)

where and denote the sets of true and predicted labels, respectively, and represents the entropy function. As shown in Equation 17, NMI ranges from 0 to 1, with higher values indicating stronger agreement. Coupled with, ARI and NMI provide a comprehensive evaluation of clustering performance.

Domain-aware SVG detection

MLN2SVG performs domain-aware SVG detection by identifying genes whose reconstructed expression patterns are enriched within individual spatial domains rather than across the entire tissue section [32, 34, 38, 39]. After latent representation learning, spatial domains are inferred from the averaged embedding

(18)

and reconstructed expression matrices and are used to rank candidate SVGs according to spatial variance, domain enrichment, and gradient-like expression patterns. This design allows MLN2SVG to prioritize genes with localized and biologically interpretable spatial structure.

Multi-level neighbor search

Unlike static KNN graphs adopted in existing graph-based ST methods, the proposed MLN algorithm dynamically refines graph topology by incorporating selectively filtered second-order neighbors. This adaptive expansion enables modeling of higher-order spatial dependencies while constraining noise propagation through similarity-based filtering and weight regularization. Consequently, the graph structure itself becomes domain-aware and data-adaptive, rather than fixed throughout training.

For each target node, second-order neighbors are ranked by similarity, and only the most relevant candidates are retained to establish additional connections for downstream SVG scoring.

This algorithm 1 ensures that second-order connections are selectively integrated, enhancing the model’s ability to identify domain-specific SVGs. These enhanced neighborhood graphs are then used to compute domain-specific gene scores, with the top-ranking genes selected as spatially variable markers.

Spatial patterns identified by the MLN search are evaluated under the null hypothesis of complete spatial randomness, where observed structure is assumed to arise from chance alone [40]. Following MLN graph reconstruction, spatial dependence is quantified using a statistic related to Moran’s I, with expectation and variance derived from two classical formulations. One formulation assumes fixed expression values with interchangeable spatial locations [41], while the other models both expression values and locations as realizations of a random spatial process [42]. These assumptions support normal-based significance assessment, as the statistic typically exhibits near normal behavior under spatial randomness [43]. Situations in which this approximation may be unreliable are addressed through permutation testing, where expression values are repeatedly reassigned across the MLN graph to generate an empirical reference distribution [44]. The pseudo -value is computed as the proportion of permuted statistics that equal or exceed the observed value, and multiple hypothesis testing across genes is controlled using the Benjamini Hochberg procedure [45].

Within the MLN2SVG framework, analyses are conducted using consistent baseline parameters. SVGs are retained when statistical significance satisfies , expression is enriched within the focal domain, detection exceeds of domain spots, and in domain expression shows a minimum -fold increase relative to surrounding regions. Sensitivity analysis varying log fold change and in domain expression thresholds confirms a trade-off between detection yield and reliability, supporting robust domain-aware SVG identification. However, in this work, we focus on the MLN2SVG framework, which improves upon existing techniques by detecting SVGs in a more localized, domain-aware manner.

Results

MLN2SVG enhances spatial domain detection and reveals fine-grained SVGs in the human DLPFC dataset

Spatial domain identification and SVG detection performance of MLN2SVG were benchmarked using 12 annotated sections of the human dorsolateral prefrontal cortex dataset from the spatialLIBD project, generated with the 10x Genomics Visium platform. All benchmark methods were executed using their publicly available implementations under consistent preprocessing settings; performance metrics were obtained by rerunning the methods rather than extracted from original publications. Additional comparisons, including GraphST, are provided in Supplementary Figure S14. Figure 2A shows the ground truth annotations for slice 151672, which is used as a representative example. MLN2SVG was evaluated against nine state-of-the-art methods, including SpaGIC, STAGATE, SEDR, DeepST, PROST, SpaGCN, stLearn, BayesSpace, and Scanpy. As shown in Fig. 2B and C, box plots of the ARI and NMI across all slices demonstrate that MLN2SVG achieved the highest median ARI (0.6505) and NMI (0.7085), with low variance, indicating robust and stable clustering performance. Competing methods such as STAGATE and DeepST exhibited lower accuracy and greater variability. Spatial visualizations in Fig. 2D further illustrate that MLN2SVG accurately delineates the layered cortical architecture, producing coherent domain boundaries compared with the over smoothed or fragmented outputs of other approaches.

Figure 2.

MLN2SVG accurately detects spatial domains and domain-specific genes in human DLPFC slice 151672. (A) Ground truth cortical layers (L3–L6) and white matter. (B, C) ARI and NMI boxplots across 12 slices show superior clustering accuracy versus nine benchmark methods. (D) Spatial clustering highlights smooth, anatomically aligned domain boundaries. (E) UMAP visualizations illustrate separability and compactness of method-specific embeddings; colors denote clusters predicted independently by each method. (F) PAGA graphs represent inferred connectivity among predicted clusters.(G) Spatial expression of representative SVGs per domain (PCP4, HAPLN4, SCGB2A2, KRT17, CARTPT) confirms biological specificity. (H) Gene clouds summarize prominent SVGs per domain. (I) Moran’s I and Geary’s C boxplots demonstrate that MLN2SVG identifies genes with higher spatial autocorrelation compared with benchmark methods.

Domain-level transcriptional organization was examined using UMAP embedding and PAGA trajectory inference, shown in Fig. 2E and F. In the UMAP visualizations, colors denote method-specific predicted clusters and are shown to illustrate the separability of the learned latent representations rather than one-to-one label correspondence across methods. The PAGA graphs summarize inferred connectivity between predicted domains. Compared with alternative methods, MLN2SVG produced more compact embeddings and more biologically plausible domain transitions.

MLN2SVG revealed clear separation of cortical layers and a biologically plausible trajectory from layers L3 through L6 to white matter, whereas alternative methods produced entangled embeddings and inconsistent trajectories. Beyond clustering, MLN2SVG integrates spatial statistics for high-resolution SVG detection. In slice 151672, domain-specific SVGs shown in Fig. 2G include PCP4, HAPLN4, SCGB2A2, KRT17, and CARTPT, each exhibiting spatially coherent expression patterns aligned with known cortical and epithelial functions. Consequently, the gene highlighted in the spatial expression panel (Fig. 2G) may not always appear as the largest word in the corresponding gene cloud. In cases where several genes show similar statistical significance, we selected representative genes for spatial visualization that display clear and distinguishable spatial patterns across the tissue. Domain-level gene clouds in Fig. 2H summarize the most predictive genes per domain and support interpretability. In the gene cloud visualization, word size reflects the relative fold-change of genes within each domain rather than the adjusted -value used to select the representative SVG displayed in the spatial plot. Finally, Fig. 2I demonstrates that MLN2SVG identifies genes with stronger spatial autocorrelation, as measured by Moran’s I and Geary’s C, compared with all benchmark methods.

Furthermore, functional enrichment analysis (Supplementary Figure S17) revealed biologically meaningful processes across the identified spatial domains. Specifically, neuronal processes, including synaptic organization and axon guidance, were enriched in PCP4 and HAPLN4 domains. Epithelial structural and signaling pathways were enriched in KRT17 and SCGB2A2 domains, while neuroendocrine signaling pathways were associated with CARTPT-enriched regions.

Additional analyses across all human DLPFC slices show that MLN2SVG consistently recovers established marker genes for major cortical cell populations, including neurons, oligodendrocytes, and astrocytes. Canonical markers such as NEFH, NEFM, MOBP, MBP, PLP1, AQP4, GFAP, RELN, RORB, CUX2, and CALB1 display reproducible and spatially coherent patterns aligned with cortical architecture (Supplementary Figures S1–S11). In slice 151674, MLN2SVG identified 840 SVGs passing both parametric and nonparametric tests (FDR ), with comparative counts shown in Supplementary Figure S15B and clustering performance summarized in Supplementary Table S2. Embedding-based clustering using k means, Louvain, and mclust demonstrated superior performance of mclust, achieving the highest ARI (Supplementary Figure S23).

Evaluating MLN2SVG on a heterogeneous breast cancer slice: spatial domain identification and SVG detection

MLN2SVG was evaluated for its ability to characterize spatial heterogeneity in complex tissues using a human breast cancer slice profiled with the 10x Genomics Visium platform. The section contained malignant, stromal, and potentially normal compartments, providing a stringent setting for spatial domain identification and gene level interpretation, and expert annotations are shown in (Fig. 3A). Clustering accuracy and stability exceeded competing methods, including SpaGIC, STAGATE, DeepST, SpaGCN, and Scanpy, across key metrics (Fig. 3B). Spatial domain maps in Fig. 3C showed clear and nonfragmented boundaries aligned with histological structure, whereas several baselines produced blurred or over smoothed partitions.

Figure 3.

MLN2SVG captures spatial heterogeneity and domain-specific gene expression in a human breast cancer slice. (A) Expert annotations reveal structural and pathological variations. (B) ARI and NMI scores across slices highlight MLN2SVG’s consistent advantage over benchmark methods. (C) Spatial clustering maps show smooth, biologically coherent domain boundaries aligned with histology. (D, E) UMAP and PAGA projections reveal well-separated, developmentally meaningful domains. (F) Spatial expression of top SVGs per domain confirms region-specific gene activity (e.g. SLITRK6, CHRDL1, ACKR1, LINC00645, LINC00052, AC087379.2, ALB, RASSF10). (G) Word clouds summarize prominent SVGs per domain. (H) Moran’s I and Geary’s C boxplots show that MLN2SVG detects genes with higher spatial autocorrelation than benchmark approaches.

UMAP and PAGA analyses further supported the quality of the learned representations by showing well-separated domains and coherent relationships between neighboring tissue regions (Fig. 3D and E). MLN2SVG also identified domain-specific SVGs with clear spatial structure, including CRISP3, LINC00645, SLITRK6, and ALB, all of which showed strong spatial autocorrelation. Representative SVGs for distinct domains included SLITRK6 (domain 1), CHRDL1 (domain 2), ACKR1 (domain 3), LINC00645 (domain 9), LINC00052 (domain 12), AC087379.2 (domain 15), ALB (domain 18), and RASSF10 (domain 20) (Fig. 3F). SVG clouds summarize the leading SVGs per domain (Fig. 3G). Gene Ontology (GO) enrichment analysis supported biological relevance (Supplementary Figure S18A and B), with enrichment in extracellular matrix remodeling, PI3K Akt signaling, focal adhesion, and immune regulation, involving genes such as COL14A1, MMP14, GSK3B, ITGA5, VCAM1, CCL19, and IL7R, alongside additional signals related to negative regulation of Wnt signaling, neuron differentiation, lipid droplets, endocytic vesicle membranes, late estrogen response, peroxisome activity, fatty acid transport, and mammary gland development. Immune-associated annotations such as Staphylococcus aureus infection and viral myocarditis likely reflect shared immune and metabolic programs rather than direct infection. Model selection based on Moran’s I and Geary’s C on DLPFC retained PROST, SpaGCN, SPARK-X, and MLN2SVG for breast cancer SVG comparison, and MLN2SVG achieved the highest median Moran’s I and lowest median Geary’s C (Fig. 3H). MLN2SVG also identified more SVGs than PROST and SPARK-X while maintaining robust specificity (Supplementary Figure S15B).

Spatial hierarchy of tertiary lymphoid structure organization in human breast cancer

Using the MLN2SVG framework, we resolved the spatial architecture of TLS across histopathologically annotated breast cancer regions by integrating SVG programs with ground-truth tissue labels (Fig. 4A–C). TLS-associated gene expression was quantified using curated TLS gene signatures, revealing a total of 226 TLS genes across all domains (Fig. 4D). Domains with five or more TLS genes (TLS-rich domains) were identified in four of 20 domains, spanning both intratumoral and tumor-edge regions (Fig. 4E). The most TLS enriched domains included Domain 15 (Tumor_edge_1, 39 genes), Domain 11 (IDC_4, 19 genes), Domain 12 (IDC_5, 17 genes), and Domain 14 (IDC_7, six genes). These domains exhibited coordinated expression of chemokines (CCL19, CCL21, CXCL12, CXCL13) and lymphocyte markers, including T-cell genes (CD3E, CD3D, CD3G, TRAC, TRBC1/2) and B-cell genes (CD79A, MS4A1, BLK), reflecting the presence of both lymphoid chemokine networks and structured organization. In contrast, DCIS and healthy domains contained limited TLS activity, typically expressing isolated chemokines (e.g. CXCL10) without evidence of organized lymphoid structures. Notably, CXCL13 expression is a hallmark of active TLS formation and it was spatially restricted to select intratumoral and tumor-edge domains, as visualized in the conserved spatial gene patterns and domain-wise presence matrix (Fig. 4F,G). Immune-related pathways were enriched among high-expression TLS genes (Supplementary Figure S18C), whereas low-expression TLS genes showed enrichment in metabolic processes (Supplementary Figure S18D).

Figure 4.

Integrated analysis of TLS in human breast cancer tissue. (A) H&E-stained tissue section with the pathologist-annotated TLS region highlighted in red. (B) Ground-truth spatial annotation of tissue regions. (C) Spatial domains predicted by MLN2SVG. (D) Functional composition of TLS-associated genes. (E) Distribution of TLS-related signals across tumor microenvironment regions. (F) Spatial expression patterns of conserved TLS-associated genes across domains. (G) Binary presence matrix showing the distribution of TLS genes across domains.

Comparison of TLS programs across intratumoral, tumor-edge, and extratumoral regions further revealed region-specific immune organization (Fig. 4E). Intratumoral domains, particularly Domains 11 and 12, showed moderate TLS enrichment, whereas the tumor-edge region Domain 15 exhibited the strongest TLS signal, consistent with a more organized lymphoid architecture. In contrast, extratumoral healthy regions showed minimal TLS activity, indicating that TLS-associated immune programs were spatially restricted rather than uniformly distributed across the tissue.

Collectively, these results demonstrate that MLN2SVG effectively captures the spatial organization and transcriptional heterogeneity of cancer tissues with high precision and biological relevance.

MLN2SVG for cellular resolution analysis: application to mouse embryogenesis SeqFISH data

MLN2SVG performance at single-cell resolution was evaluated using the SeqFISH mouse embryo dataset with curated ground truth annotations (Fig. 5A). With 22 clusters, MLN2SVG recovered embryonic domain boundaries consistent with known structures and achieved superior ARI and NMI compared with competing methods (Fig. 5B). PAGA and UMAP analyses revealed continuous developmental topology and preserved intra domain coherence (Fig. 5C and D).

Figure 5.

MLN2SVG enables cellular-resolution analysis of mouse embryogenesis SeqFISH data. (A) Original annotations of SeqFISH-profiled embryo sections reveal distinct spatial domains corresponding to developmental structures. (B) Clustering into 22 groups shows that MLN2SVG outperforms other methods in ARI and NMI, aligning more closely with expert labels. (C, D) PAGA and UMAP visualizations reveal well-separated, biologically coherent domains with developmentally meaningful relationships. (E) Spatial expression of representative SVGs confirms cluster-specific biological patterns. (F) Word clouds illustrate top SVGs per cluster, capturing key transcriptional SVGs during embryogenesis. (G) Moran’s I and Geary’s C boxplots show higher spatial autocorrelation for genes identified by MLN2SVG compared with benchmarks.

Analysis of SVG expression patterns showed that MLN2SVG recovered representative developmental markers in the Mouse Embryogenesis SeqFISH dataset, including Col1a1, Nr2f1, Cdh5, Foxf1, Sfrp1, Ttn, Hoxb9, Hoxd4, Krt18, Alas2, Shh, Podxl, and Dlk1. These genes displayed spatially restricted expression across distinct embryonic regions and were consistent with known developmental patterning signals in the embryo. This spatial specificity indicates that the SVGs identified by MLN2SVG reflect biologically meaningful organization rather than spurious statistical effects (Fig. 5E). Domain-specific transcriptional programs are further summarized by word clouds (Fig. 5F), while Moran’s I and Geary’s C distributions confirm strong spatial autocorrelation among detected SVGs (Fig. 5G). Functional enrichment analysis (Supplementary Figure S19) reveals significant involvement of transcriptional regulation, DNA binding, RNA polymerase activity, skeletal system development, cell junctions, and extracellular matrix organization, consistent with progenitor and differentiating compartments. Comparative evaluation against PROST, SPARK-X, and SpatialDE shows that MLN2SVG achieves higher spatial autocorrelation scores and identifies a larger set of coherent SVGs (Fig. 5H and Supplementary Figure S15B). Collectively, these results establish MLN2SVG as a robust and biologically faithful framework for cellular resolution ST with enhanced sensitivity and interpretability.

MLN2SVG enables high-resolution spatial domain mapping in the mouse olfactory bulb using Stereo-seq

The mouse olfactory bulb provides a canonical example of laminar tissue organization and serves as a stringent benchmark for spatial domain detection. Using Stereo seq data with 19 109 spatial spots and 14 376 genes, MLN2SVG was evaluated against widely used methods. MLN2SVG accurately recovered expert annotated layers, including the rostral migratory stream, granule cell layer, inner plexiform layer, mitral cell layer, external plexiform layer, glomerular layer, and olfactory nerve layer, with smooth and biologically coherent boundaries (Supplementary Figure S12A and B). Consistent domain separation was observed in spatial clustering and UMAP embeddings (Supplementary Figure S12B and C), while PAGA connectivity analysis confirmed inter domain relationships aligned with known developmental and functional interactions (Supplementary Figure S12D).

Domain accuracy was further supported by the correspondence between the annotated olfactory bulb layers in (Supplementary Figure S12E) and the representative SVGs shown in Supplementary Figure S12F. The identified genes, including mt-Nd6, Apoe, Penk, Plp1, Cdhr1, Serpini1, and Kcnb2, exhibited spatially restricted expression patterns across distinct anatomical layers of the olfactory bulb. This concordance between annotated structure and SVG localization supports that MLN2SVG captures biologically meaningful laminar organization in the Stereo-seq mouse olfactory bulb dataset. Word clouds summarizing top SVGs per domain reinforced these layer specific transcriptional programs (Supplementary Figure S12G). Spatial autocorrelation analysis showed that MLN2SVG achieved the strongest global and local structure, with the highest mean Moran’s I (0.887620) and lowest mean Geary’s C (0.163990), outperforming PROST, SPARK X, and SpatialDE (Supplementary Figure S12H). MLN2SVG also detected more spatially coherent SVGs than PROST and SPARK X across datasets (Supplementary Figure S15B).

Functional enrichment analysis further supported biological relevance, with GO terms enriched for dendrites, neuron projections, dendritic membranes, and GABAergic synaptic transmission, reflecting the dominant neural circuitry of the olfactory bulb (Supplementary Figure S20). Additional enrichment in growth factor signaling, synapse assembly, and immune-related regulation highlights coordinated programs underlying laminar structure. As a whole, these results demonstrate that MLN2SVG captures genuine neural architecture and provides a robust framework for resolving laminar organization in complex brain tissues.

MLN2SVG accurately captures spatial domains in the mouse brain: analysis of 10x visium coronal sections

The 10x Visium platform provides high-resolution spatial transcriptomic profiles of the mouse brain, enabling evaluation of MLN2SVG in recovering known cytoarchitectural organization. Manual annotations of a coronal section define laminar and regional structure as biological reference (Supplementary Figure S13A). MLN2SVG accurately reconstructed these boundaries (Supplementary Figure S13B) and achieved higher ARI and NMI scores than SpaGIC, STAGATE, SpaGCN, Scanpy, DeepST, and SEDR, with particularly strong performance in cortical and hippocampal regions where smooth and coherent borders closely matched anatomical ground truth.

UMAP embeddings further demonstrated well-separated and biologically interpretable clusters for MLN2SVG, whereas competing methods showed partial overlap (Supplementary Figure S13C). PAGA connectivity analysis revealed domain relationships consistent with known neuronal circuitry, indicating preservation of biologically meaningful inter domain structure (Supplementary Figure S13D). Spatial expression maps from the Visium olfactory bulb dataset showed that SVGs aligned with functional layers (Supplementary Figure S13E). Genes such as Ddn, Slc30a3, Stx1a, Bcas1, Penk, Foxp2, Prkcd, and Clic6 marked neuronal domains associated with synaptic signaling and circuit organization, while Baiap3, Mal, Cabp7, Lamp5, and Mgp highlighted myelination zones, projection pathways, and metabolic support regions corresponding to glomerular and mitral cell layers. Word clouds of top SVGs reinforced these layer-specific transcriptional programs (Supplementary Figure S13F).

Spatial autocorrelation analysis showed that MLN2SVG achieved higher Moran’s I and lower Geary’s C values than PROST, SPARK-X, and SpatialDE, indicating stronger and more coherent spatial structure (Supplementary Figure S13G). MLN2SVG consistently outperformed PROST and SPARK-X in identifying genes with robust spatial signatures across platforms (Supplementary Figure S15B). GO enrichment analysis further supported biological relevance, with enrichment in calcium-dependent exocytosis, endosomal trafficking, trans Golgi network, neuron projection, and myelination processes, alongside molecular functions related to chloride channel regulation and protein kinase activity (Supplementary Figure S21). Taken Jointly, the spatial patterns and enrichment results demonstrate that MLN2SVG captures biologically coherent domains and underlying molecular programs, reinforcing its reliability for spatial transcriptomic analysis of complex brain tissues.

MLN2SVG analysis of the STARmap mouse visual cortex dataset

MLN2SVG was evaluated on high-resolution single-cell spatial transcriptomic data using the STARmap mouse visual cortex dataset comprising 1207 cells and 1020 genes, providing a stringent benchmark for fine scale spatial organization. MLN2SVG reconstructed the laminar structure of the visual cortex with strong agreement to expert manual annotations (Supplementary Figure S14), with the histological reference shown in (Supplementary Figure S14A). Comparative analysis against STAGATE, SpaGCN, DeepST, SEDR, and GraphST showed superior spatial domain identification with greater consistency and precision (Supplementary Figure S14B), and reconstructed domain maps captured coherent cortical transitions with reduced fragmentation (Supplementary Figure S14C). UMAP embeddings revealed well-separated clusters aligned with distinct cortical layers, indicating preservation of biological structure in the learned representations (Supplementary Figure S14D). PAGA connectivity further supported biologically meaningful inter domain relationships consistent with hierarchical cortical organization (Supplementary Figure S14E). Spatial maps of representative SVGs including Camk2a, Cplx1, Adarb2, and Plp1 demonstrated accurate localization of domain-specific markers across layers (Supplementary Figure S14F). Domain-level analysis indicated enrichment of 3110035E14Rik and PcsK2 in domain 1, Eg1 and Nrg1 in domain 2, Ncdn and Lamp5 in domain 3, Camk2a and Hoxa4 in domain 6, and Plp1 and P1 in domain 7, with word clouds summarizing top-ranked SVGs highlighting domain-specific programs (Supplementary Figure S14G). Spatial autocorrelation analysis showed the highest Moran’s I values and lowest Geary’s C values among all methods (Supplementary Figure S14H). MLN2SVG identified fewer SVGs than PROST and SPARK-X but prioritized genes with robust spatial organization, supporting higher confidence signals (Supplementary Figure S15B). GO biological process enrichment highlighted synaptic transmission, vesicle trafficking, axon guidance, and neuroglial support, consistent with visual cortex function (Supplementary Figure S22).

Constructing consensus and true SVGs

Consensus sets of SVGs were constructed by focusing on genes consistently identified by multiple high-performing SVG detection methods. Three state-of-the-art approaches, PROST, SPARK-X, and MLN2SVG, were selected based on their demonstrated ability to capture genes with strong and coherent spatial patterns across diverse datasets (Supplementary Figure S15C). For each of six ST datasets, genes detected by all three methods were intersected to form a consensus SVG set, representing reliable spatial markers and reflecting robust spatial heterogeneity in gene expression.

Beyond the consensus, additional SVGs were evaluated using Moran’s I and Geary’s C statistics to identify genes with strong spatial autocorrelation that were not shared across methods. Genes exhibiting high Moran’s I values were retained as true SVGs, as these scores indicate pronounced and biologically meaningful spatial structure (Supplementary Figures 2–5 and Figures S12–S14). This two-stage strategy enabled recovery of both highly reproducible SVGs and method-specific spatial signals, providing a more comprehensive and reliable characterization of spatial gene expression patterns.

Discussion and conclusion

This study introduces MLN2SVG, a domain-aware framework for SVG detection that jointly models spatial domains and gene expression through contrastive variational learning and adaptive MLN refinement. Joint modeling of these components improves clustering accuracy, strengthens spatial gene discovery, and enhances biological interpretability across diverse ST datasets.

Importantly, the novelty of MLN2SVG does not lie in the standalone use of graph neural networks or variational autoencoders, which are well established in ST, but in the integration of adaptive topology refinement, spectral convolution–deconvolution, layer-wise contrastive alignment, and domain-aware SVG statistical modeling within a single unified framework.

Across human and mouse datasets, MLN2SVG consistently outperformed competing methods in spatial domain identification and in the detection of genes with strong spatial autocorrelation. These improvements were evident in both healthy and diseased tissues, including DLPFC, breast cancer, mouse embryo, and brain datasets, where the method recovered biologically meaningful spatial programs with improved robustness under data sparsity.

MLN2SVG showed a balanced sensitivity–specificity profile for SVG detection across multiple ST datasets (Supplementary Figures S15B and S15C). In the breast cancer dataset, MLN2SVG identified 4817 SVGs, exceeding SPARK-X and PROST, while 327 genes were shared across all three methods, indicating agreement on core spatial signals. In DLPFC slice 151672, MLN2SVG detected fewer SVGs overall but retained nearly all high-confidence genes shared among methods, suggesting selective yet robust recovery of biologically meaningful signals. Higher overlap with other methods was observed in the mouse visual cortex and SeqFISH embryo datasets, whereas lower overlap in the mouse brain coronal and Stereo-seq datasets indicated a more selective detection profile. Together, these results show that MLN2SVG captures biologically relevant SVGs while maintaining concordance with established methods for key spatial markers.

Beyond human brain tissue, MLN2SVG was applied to breast cancer (Fig. 3), mouse olfactory bulb profiled by Stereo seq (Supplementary Figure S12), mouse brain coronal sections generated by 10x Visium (Supplementary Figure S13), and STARmap mouse visual cortex samples (Supplementary Figure S14). Across these datasets, MLN2SVG consistently outperformed competing approaches in clustering and SVG detection, achieving higher ARI and NMI scores (Supplementary Figure S16). In breast cancer tissue, genes such as SLITRK6, CHRDL1, and ACKR1 were identified as domain-specific markers (Fig. 3), supporting tumor progression and immune related functions. Spatial domains aligned with histopathology while revealing fine-scale immune organization and TLS programs (Fig. 4). Additional evaluation using mclust on MLN2SVG embeddings further demonstrated improved spatial domain identification compared with k means and Louvain (Supplementary Figure S23).

These findings demonstrate that MLN2SVG provides a unified and flexible framework to capturing spatial heterogeneity in complex tissues through the joint modeling of spatial domains and SVG expression. Integration of multi-level spatial dependencies with contrastive representation learning enables robust and biologically coherent spatial analyses across diverse tissue contexts. In breast cancer datasets, MLN2SVG further resolves immune-associated spatial regions, including TLS, underscoring its ability to identify subtle yet functionally relevant spatial patterns. Collectively, these results position MLN2SVG as a scalable and interpretable foundation for future spatial genomics studies, with clear potential for extension to multimodal, 3D, and disease-focused analyses. MLN2SVG is not explicitly designed as a batch correction or multi-slice integration framework. For datasets exhibiting substantial batch effects across samples or tissue sections, upstream integration methods such as Harmony or scVI may be applied prior to graph construction. While the contrastive alignment and adaptive neighborhood refinement in MLN2SVG can mitigate minor technical variation, explicit batch modeling remains an area for future extension.

Key Points

• MLN2SVG is a domain-aware method that jointly identifies spatial tissue regions and spatially variable genes, helping to address heterogeneity and sparsity in ST data.

• By integrating contrastive variational autoencoding with a multi-level neighbor strategy, MLN2SVG captures both local and long-range spatial dependencies while preserving biologically meaningful variation.

• MLN2SVG consistently outperforms existing approaches across multiple datasets and reveals biologically meaningful tissue organization, including immune-related spatial patterns in cancer.

Author contributions

Shabir Hussain (Conceptualization, Investigation, Methodology, Writing—original draft), Muhammad Ayoub (Investigation, Methodology, Resources), Fei Ye (Investigation, Methodology, Resources), and Xiao Liu (Conceptualization, Funding acquisition, Investigation, Methodology, Supervision, Writing—review & editing).

Conflicts of interest

The authors have no conflicts of interest to declare.

Funding

This work is supported by Scientific Research Start-up Funds (QD2021005N), Shenzhen Science and Technology Program (WDZC20220819134430002), and Shenzhen Universities Stable Funding Key Projects (WDZC20200821104802001).

Data availability

Detailed information about the datasets utilized in this study can be found in Supplementary Table 1. All experiments in this study were implemented using Python 3.8.0. The computational framework relies on the following libraries: NumPy 1.22.4, pandas 2.0.3, AnnData 0.9.2, Scanpy 1.9.8, Squidpy 1.2.3, SciPy 1.10.1, scikit-learn 1.3.2, matplotlib 3.7.5, numba 0.55.2, wordcloud 1.9.4, natsort 8.4.0, and rpy2 3.5.17. Deep learning models were trained using PyTorch 2.4.0 with CUDA 11.8 acceleration on an NVIDIA H800 GPU. Image processing and visualization were supported by OpenCV 4.11.0. All computations were performed on a Linux-based workstation.

The complete source code and implementation details of the MLN2SVG framework are publicly available at https://github.com/Hussaain/MLN2SVG and https://zenodo.org/records/18028269 .