Benchmarking segmentation methods for spatial transcriptomics

Xabier Martínez-de-Morentin; Sumeer A Khan; Alberto Maillo; Robert Lehmann; Narsis A Kiani; Vincenzo Lagani; David Gomez-Cabrero; Jesper Tegner

doi:10.1038/s41592-026-03272-x

. 2026 Oct 23;23(11):2286–2299. doi: 10.1038/s41592-026-03272-x

Benchmarking segmentation methods for spatial transcriptomics

Xabier Martínez-de-Morentin ^1,^✉,^#, Sumeer A Khan ^1,^2,^#, Alberto Maillo ¹, Robert Lehmann ¹, Narsis A Kiani ³, Vincenzo Lagani ^1,^2,⁴, David Gomez-Cabrero ^1,^✉, Jesper Tegner ^1,^2,^5,^6,^7,^✉

PMCID: PMC13736688 PMID: 42919952

Abstract

Cell segmentation is a critical yet under-standardized step in spatial transcriptomics, directly affecting cell identity assignment, spatial organization and downstream biological interpretation. Despite rapid advances in spatial transcriptomics technologies, systematic evaluations of segmentation methods across platforms and biological contexts remain limited. Here we benchmark 11 image-based, transcriptomics-based and multimodal segmentation methods across 15 datasets generated by 5 spatial transcriptomics technologies, spanning 12 human and mouse tissue types. We introduce biologically informed evaluation metrics that quantify segmentation quality, transcript recovery, cell composition, morphology, expression purity and spatial organization. Benchmarking reveals substantial variation in performance across datasets, technologies and evaluation criteria, highlighting trade-offs among segmentation completeness, transcriptomic fidelity and biological interpretability. Cross-technology analyses further demonstrate that segmentation choice critically shapes downstream analyses, including cell-type specificity, transcript contamination and ligand–receptor inference. Together, our unified Nextflow pipeline provides a scalable foundation for standardized evaluation of segmentation and method selection in spatial transcriptomics.

Subject terms: Transcriptomics, Comparative genomics

This Analysis benchmarks segmentation methods across different datasets and presents Nextflow, a pipeline for standardized method selection for spatial transcriptomics.

Main

Spatial transcriptomics technologies enable high-resolution visualization and analysis of tissue transcriptomic landscapes, preserving native spatial organization and cellular context^1–3. These approaches uncover cellular heterogeneity and complex tissue microenvironments, providing fundamental insights across development, homeostasis and disease^4,5. However, due to the complexity and heterogeneity of spatial omics datasets, platforms and tools for analysis continue to develop, and so do the analytical challenges they pose^6,7. Current analysis pipelines require multiple computational tasks, including data quality analysis (for example, denoising^8,9, imputation to handle data sparsity^10–12), preprocessing¹³, clustering to enhance resolution¹⁴ and deconvolution for cell-type identification^15–17. Additional tasks include detecting variable genes^18,19 and analyzing spatial gradients and tissue hierarchy^20,21. Central to all those tasks is cell segmentation^22,23. Once the precise boundaries of individual cells are detected, the algorithm assigns all the pixels from the image and all the gene transcripts to their correct cell, creating a detailed map of cellular identity and function. Accurate segmentation is the foundation upon which all further analysis is built. Errors at this stage can propagate, affecting cell phenotyping, spatial pattern discovery, ligand–receptor inference and cell-to-cell communication, yet segmentation remains one of the least standardized and most error-prone steps in spatial transcriptomics workflows.

Several computational frameworks have been proposed to support segmentation in spatial transcriptomics. Tools such as Squidpy²⁴, which includes several image-based segmentation algorithms, incorporate image processing techniques and advanced deep-learning models. While the SOPA²² package is a unified framework for image-based segmentation methods, it is unable to integrate images and transcripts. Despite extensive benchmarking efforts for imputation, deconvolution and clustering^25–27, segmentation has primarily been benchmarked using image-based data alone, lacking assessments incorporating transcriptomic context²⁸, as detected by sequence-based spatial methods. This represents a critical methodological gap. Furthermore, the lack of biologically relevant metrics impedes thorough comparative performance analysis of different segmentation methods. As a result, users face substantial uncertainty when selecting segmentation strategies appropriate for specific tissues, platforms or biological questions.

Benchmarking segmentation methods at scale is further complicated by the lack of a unified computational pipeline capable of handling heterogeneous data formats and large datasets. Users often face compatibility issues with different layers of information^29,30, such as spatial coordinates, expression data and cell segmentations, spread across various incompatible formats. Dataset size adds another layer of complexity, frequently necessitating ad hoc tiling or patch-based processing strategies that are not natively supported by many segmentation tools and require substantial manual interventions^31–33. Furthermore, the absence of a standardized pipeline means newly developed methods cannot be readily systematically evaluated or benchmarked³⁴.

Here, we address these challenges by developing a unified, scalable benchmarking framework, including introducing biologically relevant metrics, for cell segmentation in spatial transcriptomics. We systematically evaluate 11 segmentation methods, spanning image-based, transcriptomics-based and multimodal approaches relevant to spatial transcriptomics: Watershed, bidirectional cell segmentation for spatial transcriptomics (BIDCell), Cellpose, subcellular spatial transcriptomics cell segmentation (SCS), Segment Anything Model (SAM), Segment Anything in Images and Videos (SAM2), Segment Anything for Microscopy (mSAM), Baysor, CelloType, Swin-S and Mask R-CNN^23,35–43 (Fig. 1a and Methods). These methods were rigorously evaluated across 15 spatial transcriptomic diverse datasets, derived from 5 distinct technologies^44–48, encompassing 12 human and mouse tissue types and approximately 2.3 million cells, with dataset sizes ranging from ~700 to nearly 800,000 cells and capturing transcriptomic information spanning hundreds to thousands of genes.

To examine the biological consequences of segmentation performance, we conducted downstream analyses on three mouse brain datasets generated using distinct spatial technologies—Stereo-seq, STARmap PLUS and MERFISH—targeting comparable anatomical regions (Fig. 1b). We applied all benchmarked segmentation tools to each dataset to disentangle algorithmic effects from platform-specific properties across image and sequence-based technologies. We evaluated cell-type-specific gene expression, contamination levels and ligand–receptor interactions, providing a direct assessment of how segmentation quality shapes biological inference.

We also introduce a comprehensive, unified, open-source Nextflow⁴⁹ pipeline for segmentation in spatial transcriptomics (Fig. 1c), available on a GitHub repository via https://github.com/TranslationalBioinformaticsUnit/Benchmarking-SegmentationMethods-ST/. The pipeline integrates diverse segmentation methods, supports heterogeneous input formats, handles large datasets through patch-based processing and ensures consistent spatial scaling for visualization and downstream analyses. The pipeline includes an evaluation pipeline grounded in the cell segmentation performance assessment (CellSPA) package³⁵, which incorporates metrics across critical categories to benchmark segmentation performance, marking the first attempt to develop a standardized framework for running and evaluating segmentation in spatial transcriptomics.

Together, our analysis reveals distinct performance variability among the tested methods and technologies, underscoring the need for informed, application-specific tool selection. Our comparative analysis and evaluation framework aim to guide method selection and improve the reliability and reproducibility of spatial transcriptomics studies, supporting the development of next-generation segmentation methods and advance our understanding of complex biological systems through spatially resolved transcriptomic data.

Results

Benchmarking methodology—platforms, datasets and evaluation metrics

We evaluated 11 segmentation methods (Watershed, BIDCell, Cellpose, SCS, SAM, SAM2, mSAM, Baysor, CelloType, Swin-s and Mask R-CNN) across 15 spatial transcriptomics datasets generated using 5 different technologies: Stereo-seq, Xenium, CosMx, STARmap PLUS and MERFISH (Fig. 1a, Supplementary Table 1 and Methods). The datasets span two species (mouse and human), 12 tissue types (brain, breast, embryo, lung, pancreas, skin, colon, stomach, ovarian, kidney, prostate and utero) and a wide range of cell counts, from around 700 cells in an ovarian cancer dataset to large cohorts of more than 800,000 cells in a prostate cancer dataset. We selected this diversity to reflect the breadth of contemporary spatial transcriptomics applications and to challenge segmentation robustness under heterogeneous biological and technical conditions (Supplementary Fig. 1a,b). We also performed cross-technology benchmarking (Fig. 1b) to account for discrepancies between platforms. This design separated algorithm-specific effects from platform-dependent biases. We conducted all the analysis using a unified benchmarking pipeline (Fig. 1c) that applies standardized quality control (QC), patch-based processing and consistent preprocessing across methods to ensure fair and reproducible comparisons.

During preprocessing, we established data subsetting criteria for each dataset and method based on technical limitations (Methods). For methods that explicitly integrate transcriptomic information into their modeling, such as BIDCell, SCS and Baysor, we used the entire gene feature space when computationally feasible. When memory or runtime limitations were prohibitive, we restricted inputs to the top 1,000 highly variable genes (HVGs), consistent with standard single-cell analysis practices. To enable scalable and uniform computation, we processed image and transcriptomic data with a patchification strategy, segmenting each patch independently before reassembly into the original spatial coordinate space (Supplementary Table 2). In addition, as outputs can include spurious cells, we restricted evaluation to high-quality segmented cells passing established filtering criteria, including minimum transcript and gene-count thresholds. This approach mirrors practices used in prior segmentation studies, such as SCS and BIDCell (Methods), ensuring biologically meaningful outputs.

We assessed the 11 segmentation methods by realigning patchified outputs into the original data space and applying the full CellSPA metrics suite to quantify performance. This included 15 biologically informative metrics capturing cell detection accuracy and preserving meaningful spatial transcriptomic features.

First, we measured the number of high-quality cells segmented (‘Seg cells’), defined by filtering for cells with sufficient transcripts and genes, as a baseline of segmentation completeness and biological relevance. Complementing this, we quantified the proportion of transcripts successfully assigned to segmented cells (‘Pct tr assig’), reflecting transcript capture within cell boundaries.

We then evaluated transcriptomic richness per cell through the average number of transcripts per cell (‘T tr cell’) and the average number of genes expressed per cell (‘T gs cell’), reflecting the transcriptional activity and complexity captured by each segmentation. To assess gene-level performance, we compared the proportion of cells expressing each gene in the tool’s output with the reference (‘Pct gs’; see Methods for details on the reference), reflecting gene expression consistency across segmented cells.

Cell morphology was characterized by comparing cell-type-specific elongation values between the segmentation results and reference data (‘Tr versus red’ and ‘Elong versus Tr’; see Methods for details on the reference), and we further incorporated a compound score that combines elongation similarity with the average number of transcripts per cell, capturing the interaction between shape fidelity and content richness.

To assess the preservation of cell identity and population composition, we computed the average correlation of log₂ gene expression values across cell types between the reference and each tool (‘Expr versus ref’). We also computed compound metrics combining expression similarity with transcript abundance: the product of average expression correlations and transcript counts (‘Ct_cor’) and the product of the average proportion of gene expression and transcript counts per cell (‘Expr versus TT’). We also calculated the correlation between reference and tool-derived cell-type proportions (‘Celltype prop’) to evaluate how accurately each method recapitulates the spatial cellular composition.

To measure the purity of expression at the cell level, we assessed marker-based F1 scores for known positive and negative markers against the reference (‘Pos purity and F1’) and normalized this score by transcript abundance to control for technical biases.

Finally, we evaluated spatial expression diversity by correlating the Shannon entropy of cell types per region with the coefficient of variation of transcript counts (‘Tr versus Entropy’), linking cellular heterogeneity and molecular variability. Additionally, we computed silhouette scores (‘Silhouette’) using principal component analysis embeddings and cell-type annotations to assess how well each segmentation preserved biologically coherent groupings in a reduced-dimensional space, supporting robust downstream analyses. The values obtained are provided in Fig. 2 and Supplementary Table 3 (details on the metrics are provided in Methods).

Fig. 2 — A heat map summarizes the 15 evaluation metrics (rows) for every dataset–tool combination (columns). Metrics are grouped into overall characteristics, cell-level QC metrics, gene-level QC metrics, cell morphology metrics, cell composition similarity, expression purity, spatial diversity and uniform manifold approximation and projection (UMAP) silhouette categories. Circle color indicates the metric category, while circle size reflects the underlying normalized metric value. Numbers inside circles denote the relative ranking of each method for the corresponding metric, where 1 represents the best-performing method. Overall scores were computed using an equally weighted mean of these metrics (light-green bars). Final rankings are shown on the right using a green-to-red color scale, with ranks ranging from 1 to 9. The figure displays seven representative datasets for visualization clarity; complete benchmarking results across all 15 datasets and all evaluated methods are provided in Supplementary Table 3. Swin-S and Cascade Mask R-CNN were excluded from quantitative comparisons because of biologically implausible segmentation artifacts (Supplementary Fig. 16). Missing metrics for the MERFISH dataset indicate unavailable data, whereas for the pancreatic tissue dataset, the absence of distinct cell-type labels prevented metric computation.

Source data

All scores were normalized to 0–1 for comparability across tools and datasets. To balance segmentation quality with coverage, we computed adjusted overall scores that account for the ratio of segmented cells to reference cells, yielding an integrated performance measure that jointly reflects accuracy and completeness (see Methods for additional details on the metrics).

Before performance evaluation, we assessed the robustness of the benchmarking framework to gene-set size using an ablation analysis on the melanoma skin dataset. The three transcript-aware methods (BIDCell, SCS and Baysor) were evaluated using either 1,000 HVGs or the full gene set. This analysis revealed modest performance improvements for SCS and Baysor, whereas BIDCell remained largely robust to gene-set choice. Notably, Baysor also produced a higher number of segmented cells when using HVGs (Supplementary Fig. 2). Within the evaluated melanoma skin dataset, gene-set selection showed a limited but method-dependent impact on segmentation performance, with Baysor additionally exhibiting differences in segmentation yield between the 1,000 HVG and full-gene settings. Because this analysis was restricted to a single dataset and two gene-selection conditions, these findings should be interpreted as a preliminary sensitivity analysis rather than a general assessment of robustness to HVG selection.

The performance of segmentation methods varies with dataset and metric

We assessed the 11 methods by realigning patchified segmentation outcomes into the original data space, quantifying cell detection and transcriptomic accuracy using the CellSPA metrics. To facilitate high-level interpretation, we first computed an ‘Overall’ measure averaging all 15 measurements, used to derive a global ranking across datasets. Under this aggregate measure, SAM achieved the highest Overall score (0.58), followed by SAM2 (0.53), BIDCell (0.52) and SCS (0.48; Fig. 2 and Supplementary Table 4). Baysor and mSAM showed comparable intermediate performance (both 0.45), while Cellpose (0.42) and CelloType (0.37) ranked lower, and Watershed consistently underperformed (0.27).

We next adjusted the Overall score by the ratio of segmented to reference cells, as segmentation quality must be interpreted jointly with coverage. Under this coverage-adjusted Overall metric, SCS achieved the highest score (0.34), followed closely by BIDCell and Baysor (both 0.33), reflecting balanced performance when jointly considering segmentation quality and cell yield. In line with this, BIDCell and Baysor also accumulated the highest number of dataset wins (6 and 5, respectively), indicating recurrent top performance across datasets under multiple evaluation criteria.

At the individual dataset-level performance, segmentation coverage showed marked method-specific trends. BIDCell identified the highest number of high-quality segmented cells in Brain-Stereo-seq (Supplementary Figs. 3 and 4), Breast-Xenium (Supplementary Figs. 5 and 6), Embryo-Stereo-seq (Supplementary Figs. 7 and 8), Colon-Xenium and Kidney-Xenium (Supplementary Table 3), showing robust boundary delineation across diverse tissues. Cellpose achieved the highest segmentation coverage in Lung-CosMx (Supplementary Figs. 9 and 10), while SAM led in Brain-STARmap PLUS. SCS produced the largest number of segmented cells in Brain-MERFISH and Pancreas-CosMx (Supplementary Fig. 11), whereas mSAM ranked first in Stomach-CosMx and Utero-MERFISH. Baysor achieved the highest segmentation coverage in MERSCOPE Brain-MERFISH, Melanoma skin-Xenium and Ovarian cancer-CosMx, reflecting its tendency to generate extensive segmentations in selected datasets. Finally, Watershed achieved the highest ‘Seg cells’ score in Prostate-MERFISH, although this was not accompanied by strong performance in downstream quality metrics, indicating that coverage alone is insufficient (Fig. 2 and Supplementary Table 4).

We observed distinct method-specific strengths when examining evaluation metrics averaged across datasets. SAM achieved the highest transcriptomic richness, as reflected by transcripts per cell (T tr cell = 0.71) and genes per cell (T gs cell = 0.73), but showed relatively limited segmentation completeness (Seg cells = 0.28), indicating strong transcript recovery at the expense of coverage. SAM2 showed a similar pattern, with high transcriptomic richness (T tr cell = 0.69; T gs cell = 0.78), but even lower segmentation completeness (Seg cells = 0.12). BIDCell demonstrated a well-balanced performance across segmentation and expression fidelity, combining strong segmentation coverage (Seg cells = 0.62) with high concordance between recovered and reference expression profiles (for example, Expr versus ref = 0.89). SCS achieved robust segmentation quality (Seg cells = 0.50) and showed consistent performance in morphology-related and variability-related metrics, including elongation–transcript correlations and entropy-based measures. mSAM provided stable, intermediate performance across most metrics (for example, Seg cells = 0.47; T tr cell = 0.31), but was comparatively weaker in preserving expression fidelity. In contrast, Watershed, Cellotype and Cellpose underperformed overall, exhibiting lower scores across segmentation coverage and transcriptomic recovery metrics, while Baysor showed moderate segmentation and expression performance but less consistency across evaluation criteria (Fig. 2 and Supplementary Table 4).

Collectively, these observations highlight that each tool exhibits strengths or weaknesses depending on specific datasets, platform technology or type of evaluation metrics. These results underscore the complexity of spatial segmentation and highlight the limitations of relying on a single metric, reinforcing the need for multidimensional and context-aware evaluations.

Segmentation performance metrics converge in biologically meaningful macro categories

Because the 15 evaluation metrics capture overlapping and partially correlated aspects of segmentation performance, we grouped them into eight higher-level categories based on the biological scale they probe (cell, gene, tissue or spatial organization) and the properties they quantify (Methods) Each category aggregates conceptually related metrics using equal weighting, thereby summarizing complementary signals while reducing redundancy (Fig. 2). Importantly, while all individual metrics are still computed and reported, this aggregation aids interpretation rather than replacing the underlying scores. We do not define an optimal metric subset, as this likely depends on the dataset and biological context.

Because the 15 metrics are not independent and individually difficult to interpret in isolation, we organized them into 8 biologically meaningful performance categories. Each category is composed of an equal-weighted aggregation of its respective raw metrics (Fig. 2 and Methods). For example, the ‘Overall characteristics’ category aggregates core outputs such as the proportion of high-quality cells and the percentage of transcripts assigned. ‘Cell-level QC’ reflects the richness of transcriptomic content per cell, including the number of transcripts and genes captured. ‘Gene-level’ considers the percentage of genes assigned across cells, capturing dataset-level expression completeness. ‘Cell morphology’ evaluates how realistic the cell shapes are based on correlations between transcript density and known morphological features such as elongation. ‘Cell composition similarity’ assesses how well the observed expression profiles match reference expectations regarding cell-type abundance and structure. ‘Expression purity’ measures how distinct and specific the transcriptomic profiles are to their respective cell types. ‘Spatial characteristics’ capture the diversity of transcript distribution using spatial entropy and variability. Finally, ‘Silhouette scores’ evaluate the preservation of transcriptional structure in low-dimensional embeddings. Figure 3a shows the detailed aggregated category-level scores for each dataset and segmentation method.

Fig. 3 — a, Radar plots represent the average metric score obtained for each data characteristic studied. Each plot corresponds to a dataset, with colors denoting the segmentation method used for benchmarking. b, Table showing the exact numerical values corresponding to the radar plots from a.

Source data

This categorical decomposition reveals further method-specific strengths when averaging aggregated metric categories across datasets (Fig. 3b). SAM2 clearly excelled in cell-level (0.73) and gene-level (0.75) QC, reflecting strong transcriptomic fidelity, and showed solid performance in spatial diversity (0.61); however, this did not translate into competitive overall performance. BIDCell achieved the highest scores in overall characteristics (0.61) and Silhouette (0.64), consistent with its strong Overall score (0.50) and the highest number of dataset winner counts (6), highlighting robust and recurrent top performance across datasets. SCS performed best in spatial diversity (0.61) and achieved the highest Silhouette score (0.75), indicating effective preservation of spatial and transcriptional structure, although with a slightly lower overall ranking. Baysor showed competitive cell-level and gene-level QC and ranked highly under the coverage-adjusted Overall metric (0.24) as a five-time winner, indicating strong performance when accounting for segmentation yield. mSAM exhibited balanced intermediate performance across most dimensions, whereas Cellpose was weaker in transcriptomic QC despite moderate morphological scores. In contrast, Watershed and CelloType consistently underperformed across transcriptomic, structural and overall metrics (Fig. 3b).

Taken together, the analysis of the eight aggregated evaluation categories confirms that no single method universally excels across all performance domains. Instead, segmentation tools exhibit complementary strengths depending on the analytical objective—for example, SAM and SAM2 for transcriptomic fidelity (cell-level and gene-level QC), BIDCell for robust overall performance and coherent expression landscapes, and SCS for preserving spatial heterogeneity and morphological structure. Methods such as Baysor further benefit when segmentation coverage is explicitly considered. These results underscore that optimal method selection should be guided by study-specific objectives and biologically informed criteria rather than a single global score.

Metrics categories show trade-offs in the choice of segmentation methods

Expanding upon the observation that individual metrics are not independent, we analyzed correlations among the aggregated category-level metrics across all methods and datasets (Fig. 2) to uncover systematic trade-offs and associations among evaluation categories³⁵.

The analysis of various datasets reveals that several category metrics exhibit low correlations with other assessed metrics within each tissue type. For instance, the category Expression purity, which integrates two complementary purity metrics Pos purity and F1, consistently shows no correlation with other metrics across all analyzed datasets. These findings show that Expression purity is a distinct and essential measure, offering precise quantitative insights not captured by any other metric (Fig. 4a,b and Supplementary Fig. 12a,b). In some cases, negative correlations were observed, reflecting inherent trade-offs; for example, optimizing expression purity can come at the expense of increased transcript exclusion or reduced segmentation yield. Consistent with this interpretation, the two constituent metrics within the expression purity category (Pos purity and F1) were themselves negatively correlated (Supplementary Fig. 12a,b), which shows an evaluation trade-off even within a single conceptual category (Supplementary Fig. 12a,b).

Fig. 4 — a, Pearson correlation heat maps between evaluated categories scores. Dark blue indicates low correlation between the metrics, while yellow indicates high correlation. The order displayed is the same as in Fig. 3. Missing metrics for a dataset indicate unavailable data or that missing distinct cell-type labels prevented metric computation. b, As shown in a, the metrics represent the overlapping metrics computed across all datasets, and the values correspond to the average values obtained from these datasets (left). As in the left heat map, the right heat map displays the same matrix but represents the average correlations across all metric categories, computed using only the five datasets that contain complete information for all categories (excluding pancreas and MERFISH).

Source data

Conversely, several category-level measures are highly correlated across multiple datasets. For instance, ‘Cell-level QC + gene-level QC + cell composition similarity’ shows strong interdependence across most datasets (Fig. 4a and Supplementary Fig. 12a) and globally (Fig. 4b and Supplementary Fig. 12b). These correlations suggest that, in many contexts, these categories capture overlapping information related to transcriptomic completeness and expression concordance. Thus, all three metrics may add redundancy with little interpretive gain.

To assess the robustness of our benchmarking framework under reduced redundancy, we recomputed the overall method ranking using a collapsed version of the metrics that retained only one representative from the mentioned correlated categories. This simplification resulted in a ranking that remained broadly consistent with the original ordering, with BIDCell retaining the top position and SAM remaining the highest-ranked method by unadjusted Overall score. Under the collapsed metric scheme, SCS maintained a competitive top-tier position, ranking closely behind BIDCell and SAM, while SAM2 showed a relative decrease in segmentation coverage, reflected in a lower adjusted Overall score. Baysor exhibited a modest improvement in the coverage-adjusted ranking, consistent with its strong segmentation yield (Supplementary Table 5).

Together, these analyses demonstrate that while specific evaluation categories are partially redundant, reducing metric dimensionality preserves the principal performance trends observed in the full benchmark. More importantly, they highlight that distinct performance dimensions—particularly expression purity—capture biologically meaningful trade-offs that cannot be inferred from coverage or transcript-recovery metrics alone. These findings reinforce the value of multidimensional, coverage-aware evaluation.

Spatial transcriptomics technology impacts segmentation outcomes

To assess how platform choice influences segmentation and downstream interpretation, we analyzed three mouse brain datasets generated using distinct technologies: Stereo-seq, STARmap PLUS (Supplementary Figs. 13 and 14) and MERFISH (Supplementary Fig. 15). To ensure a fair comparison, we constrained all analyses to robustly expressed genes and three reliably annotated major brain cell types—astrocytes, endothelial cells and microglia (Methods)—isolating differences attributable to segmentation and technology.

To lay the groundwork for downstream biological interpretation, we first compared gene detection across the three spatial transcriptomics platforms (Fig. 5a). Although all platforms detected a common subset of genes, Stereo-seq and STARmap PLUS showed extensive pairwise overlap. In contrast, MERFISH exhibited a more distinct gene-detection profile. As a result, the strict three-way overlap was smaller than the complete pairwise overlap between Stereo-seq and STARmap PLUS, yet still defined a shared detection space suitable for direct cross-platform comparison.

Within the intersection space (Fig. 5a), we examined representative marker genes for major brain cell types—astrocytes (Igfbp5 (ref. ⁵⁰), Sulf2 (refs. ^51,52) and Gpc6 (refs. ^51,53)), endothelial cells (Flt1 (ref. ⁵⁴), Cldn5 (ref. ⁵⁵), Vtn⁵⁶ and Rgs5 (ref. ⁵⁷)) and microglia (Ctss^58,59) across platforms and segmentation tools (Fig. 5b and Supplementary Fig. 16a,b). All selected marker genes lie within this shared space, enabling cross-platform comparison.

We focused on three representative segmentation methods spanning major methodological classes: Cellpose (image-only baseline), mSAM (advanced image-based) and BIDCell (integrated transcriptomic and imaging). This analysis uncovered platform-specific strengths. Stereo-seq reliably captured astrocyte markers across tools, reflecting high segmentation fidelity for these cells. Conversely, MERFISH and, to a lesser extent, STARmap PLUS, failed to recover expected astrocytic patterns—MERFISH, in particular, showed overexpression in unrelated cell types, suggesting segmentation noise or transcript misclassification. The trend was reversed for microglial markers: MERFISH and STARmap PLUS recovered Ctss expression reliably, whereas Stereo-seq showed poor signal recovery. STARmap PLUS performed best for endothelial markers, followed by MERFISH and then Stereo-seq. While MERFISH was able to detect multiple cell types, it also showed elevated off-target expression, that is, transcripts assigned to cell types or spatial locations where the gene is not biologically expected to be expressed. In contrast, STARmap PLUS and Stereo-seq provided cleaner, more cell-type-specific expression patterns, reinforcing their advantage in generating low-noise segmentations.

To directly quantify transcript misassignment, we examined the expression of negative markers—genes not expected to be expressed in a given cell type—such as astrocyte marker Igfbp5 in endothelial cells and endothelial marker Flt1 in microglia, using the Stereo-seq dataset (Fig. 5c, top and bottom panels, respectively). These analyses revealed contamination artifacts, particularly in the spatial regions distant from a trustworthy source of the marker expression. Extending this evaluation across all segmentation tools showed that specific methods (for example, mSAM, Cellotype and Baysor) exhibited lower contamination, more faithfully assigning transcripts to the correct cell types. These findings highlight how tool choice affects expression specificity and contamination. Notably, the observed negative-marker expression levels correspond to contamination affecting up to ~10–15% of cells in some conditions, indicating substantial misassignment driven by segmentation differences rather than background noise.

To further validate these results, we constructed expression fingerprints for each cell type and quantified their enrichment across all combinations of tools and technologies (Methods). Stereo-seq reliably identified astrocytic and microglial profiles, while STARmap PLUS excelled in endothelial profile detection. MERFISH consistently underperformed in accurately capturing expected cell-type-specific signatures (Fig. 5d). These results align with the known technical attributes of the platforms. STARmap PLUS uses highly specific in situ hybridization probes and three-dimensional spatial resolution, resulting in precise transcript-to-cell assignments and reduced contamination⁶⁰. Stereo-seq, while offering high spatial resolution, is more susceptible to cross-hybridization and background noise, increasing misassignment risk, particularly in densely packed regions⁶¹. MERFISH showed the highest variability and contamination, likely because the tools did not resolve limitations in probe performance, capture efficiency or spatial resolution (Fig. 5d).

To explore how these issues propagate to downstream analysis, we computed ligand–receptor interactions using the Giotto R package⁶², focusing on a predefined set of well-established signaling pairs with known roles in cortical organization and glial–neuronal communication. Specifically, we restricted this analysis to the representative ligand–receptor pairs shown in Fig. 5e—Pdgfc–Pdgfra⁶³, Prok2–Prokr2 (ref. ⁶⁴) and Vip–Vipr2 (ref. ⁶⁵)—which were selected a priori based on extensive prior evidence of their biological relevance in the mammalian cortex. STARmap PLUS and Stereo-seq consistently placed interacting pairs in the expected cells, and MERFISH frequently misassigned ligand–receptor pairs to incorrect cell populations, reflecting higher transcript contamination and misclassification (Fig. 5e).

In particular, several representative ligand–receptor interactions illustrate these platform-specific differences. For example, the astrocyte–OPC signaling axis involving Pdgfc–Pdgfra, a pathway associated with oligodendrocyte progenitor cell (OPC) proliferation and maintenance, showed differing inferred cell-type assignments across technologies and segmentation methods. Similarly, STARmap PLUS more specifically detected Prok2–Prokr2 communication among layer-specific excitatory neurons, consistent with their known roles in cortical circuit modulation, whereas MERFISH assigned this interaction more diffusely across cell types. A comparable pattern was observed for the Vip–Vipr2 pathway, which STARmap PLUS and Stereo-seq localized primarily to Vip interneurons and their downstream excitatory partners, while MERFISH displayed broader signal dispersion across multiple cell populations.

In summary, these analyses demonstrate that choice of spatial transcriptomics technology has a substantial impact on segmentation robustness, expression specificity and downstream biological interpretation. STARmap PLUS and Stereo-seq generally exhibited more cell-type-specific expression patterns and more spatially restricted ligand–receptor interaction assignments. While MERFISH was able to detect cell types, it showed considerable background signal and nonspecific contamination from neighboring cells, which segmentation tools could not completely resolve. These findings underscore the importance of developing analytical strategies tailored to each spatial transcriptomics technology’s specific characteristics and limitations.

Nextflow: a baseline unified pipeline for spatial segmentation benchmarking

To establish a reproducible and scalable benchmarking framework, we implemented the pipeline using Nextflow, a workflow management system. Its modular design allows us to seamlessly integrate diverse tools and scripts, manage dependencies efficiently and ensure reproducibility across different computing environments. By leveraging parallel execution and containerization (for example, Anaconda, Docker and Singularity), our pipeline is scalable, thus enabling consistent performance comparisons across different datasets and platforms.

Our pipeline is organized into three blocks: data preprocessing, application of segmentation tools and evaluation of segmentation outcomes. In the initial block of data preprocessing, transcript and image files are imported, followed by transformations to standardize and optimize the data for subsequent analysis. Critical tasks include variable selection and renaming, axis alignment and transformations to the image data, such as color scale adjustment and centering. Both the image and transcript files are then divided into smaller, manageable patches of specified dimensions to facilitate segmentation.

In the second block, segmentation, the methodologies are applied to the patched files, with each method predicting cell boundaries and systematically organizing the results into designated folders corresponding to the respective segmentation techniques. Finally, the evaluation block encompasses an evaluation script designed to (1) aggregate the segmentation results from the many patched files back into the original spatial context, (2) process the reference single-cell RNA-sequencing (scRNA-seq) data, and map the cell-type annotation to the spatial transcriptomics dataset, and (3) compute benchmarking metrics and visualization. This third block computes defined metrics and visualizes all segmentation outcomes, enabling systematic benchmarking and comparison across tools.

Together, this Nextflow-based pipeline provides a unified, extensible baseline for large-scale benchmarking of spatial transcriptomics segmentation methods, facilitating reproducible evaluation and systematic comparison as new tools and technologies emerge.

Practical guidance for segmentation method and platform selection

The varied performance across datasets highlights the importance of aligning segmentation tools with specific dataset characteristics and biological questions. Image-based methods such as SAM performed well on datasets with visually distinct cells, whereas SCS performed better on datasets with more uniform cellular appearances. This context dependency suggests that researchers should select segmentation methods based not only on overall benchmark performance but also on tissue architecture, platform characteristics and the intended downstream analysis (Fig. 6a). Platform-specific effects further influence biological interpretation, particularly for cell-type detection, expression specificity and ligand–receptor inference (Fig. 6b). STARmap PLUS consistently performed well across both readily and more difficult-to-detect cell types, Stereo-seq performed robustly for easier cell types but was less reliable for more challenging ones, and MERFISH showed higher background contamination despite generally strong expression signals. Practical deployment also involves computational considerations. Methods integrating transcriptomic information generally required longer execution times than image-only approaches, adding a trade-off between segmentation accuracy, coverage and computational cost (Supplementary Table 6).

Fig. 6 — a, Decision tree illustrating key factors for selecting cell segmentation methods. Highlighted branches denote the most relevant criteria: cellular homogeneity (purple), segmentation quality (green), performance in terms of the number of cells identified and segmented (cyan), and the required user expertise to run the tool (blue). For example, if a user with only image data as input has cells with varied shapes and diameters and aims for optimal cell identification and segmentation performance, mSAM would be the recommended method. b, As in a, but illustrating key factors for selecting technology.

Source data

Discussion

Our findings suggest that segmentation uncertainty is a major, often underappreciated source of variability in spatial transcriptomics studies, potentially influencing biological conclusions as strongly as downstream analytical choices. Our benchmark shows that no universal segmentation method exists for spatial transcriptomics. Instead, segmentation performance depends on the interplay among method design, data modality, tissue architecture, platform-specific properties and the evaluation criteria used. This has practical implications for study design: the appropriate segmentation strategy should be selected according to the biological question, the spatial transcriptomics technology and the downstream analysis of interest. Our findings also highlight important trade-offs between segmentation coverage, transcriptomic fidelity, computational cost and biological interpretability. In this context, hybrid approaches that integrate image and transcriptomic information appear particularly promising for robust segmentation across diverse datasets.

Our results also show that conclusions about the ‘best’ segmentation method depend strongly on how performance is summarized. Metrics that emphasize per-cell quality can favor different tools from those that emphasize recovery of the full cellular landscape. This distinction is important because different biological applications place different demands on segmentation. Studies focused on comprehensive spatial maps, compositional analyses, rare-cell recovery or cell–cell communication may require coverage-aware evaluation, whereas analyses focused on high-confidence cellular profiles may benefit from stricter quality-oriented criteria. These considerations highlight the need for multidimensional benchmarking rather than reliance on a single global score.

The comparison across spatial transcriptomics technologies further emphasizes that segmentation cannot be considered independently from the experimental platform. Platform-specific properties, including image quality, spatial resolution, transcript detection chemistry, background signal and gene-panel design, can influence both segmentation outputs and downstream biological interpretation. These effects are especially important for analyses that depend on accurate transcript-to-cell assignment, such as cell-type annotation and ligand–receptor inference. Therefore, computational segmentation choices should be evaluated together with platform characteristics and the intended biological question.

Beyond the quantitative comparisons, our findings reinforce that image-driven, transcript-informed and multimodal approaches each have advantages under different biological and technical conditions. Tissue architecture, molecular complexity, platform-specific noise and downstream analysis goals should therefore guide the choice of segmentation. Practical deployment also involves computational considerations, creating an additional trade-off between segmentation accuracy, coverage and computational cost.

Some limitations should be considered when interpreting this benchmark. Although the study spans multiple tissues, species, platforms and segmentation strategies, it does not exhaustively cover all available spatial transcriptomics technologies or all possible parameter settings for each method. Some comparisons are also constrained by differences in platform design, image quality, gene panels and tissue composition, which can make it difficult to fully separate algorithmic performance from technology-specific effects. In addition, the gene-set sensitivity analysis was performed on a single dataset and should therefore be interpreted as a preliminary assessment rather than a general conclusion. Future replicate-focused benchmarks using matched or near-matched tissue sections would further help disentangle biological variability, platform effects and segmentation-specific differences.

Segmentation is one of the largest hidden sources of uncertainty in spatial biology. Segmentation quality sets the upper bound on the quality of all downstream representation learning. Spatial biology is rapidly moving toward building reference atlases, inferring cell–cell communication, training foundation models and reconstructing tissue organization. Our results demonstrate that these analyses are intrinsically tied to the quality and assumptions of the underlying segmentation. Notably, segmentation performance arises from the interplay among method design, data modality, tissue architecture, computational requirements and platform characteristics. This underscores the importance of aligning study design, platform choice and research question with the strengths and limitations of available segmentation tools. As spatial technologies continue to scale in resolution and throughput, standardized, biologically informed evaluation frameworks will become essential to ensure that advances in data generation translate into reliable biological discovery. Beyond the current benchmark, the standardized Nextflow pipeline provides a reproducible infrastructure for evaluating future segmentation methods, datasets and technologies under a common standard. The consistent performance of multimodal methods suggests that future advances in segmentation may emerge less from improved image processing alone and more from the principled integration of morphological and molecular information.

Methods

Datasets and experimental platforms

This study analyzed publicly available datasets from five leading subcellular spatial transcriptomics (SST) platforms. These platforms were selected to span a broad range of gene coverage and spatial resolution capabilities: Nanostring CosMx and BGI’s Stereo-seq, which profile thousands of genes with moderate spatial resolution; 10x Genomics Xenium, which focuses on a few hundred genes but offers higher spatial resolution and sensitivity; STARmap PLUS and MERFISH. This selection highlights the diversity of SST technologies and enables a comprehensive performance evaluation that accounts for trade-offs between gene coverage and spatial resolution. The chosen datasets have been used in segmentation benchmarking studies, ensuring a robust comparison across various tissues and species. Beyond the platforms, the selection criteria also account for variations in cell numbers, tissue morphologies and anatomical contexts, providing a solid framework for analyzing the interplay between image resolution and transcriptomic data under complex conditions.

Representative-sample selection and replicates. For several platforms, multiple samples or near-replicates are publicly available (for example, Xenium, CosMx and Stereo-seq). In this benchmark, we selected one representative sample per technology to ensure cross-method comparability (consistent imaging channels, probe panels and adequate molecular coverage) and to avoid method-specific exclusions. Because most publicly available ‘replicates’ correspond to adjacent/serial sections or different biological samples rather than true technical reruns of the identical tissue section, differences across such replicates can reflect biological and section-to-section variability in addition to technical effects; therefore, replicate-focused benchmarking is treated as a dedicated future direction (Discussion).

Xenium dataset

The Xenium dataset used in our study is the breast cancer dataset, obtained from 10x Genomics (https://www.10xgenomics.com/products/xenium-in-situ/preview-dataset-human-breast/). This dataset includes two replicates, each featuring 313 unique gene names. We selected the first replicate, Xenium-BreastCancer1, for our analysis to align with previous segmentation studies that primarily utilized this replicate in their main results. Focusing on Xenium-BreastCancer1 ensures that our findings are directly comparable to existing work. Xenium-BreastCancer1 has RGB image dimensions of 5,475 × 7,524 × 3. This dataset has been previously used in the segmentation analysis. We included three further human Xenium datasets obtained from the HISSTA database (https://kbds.re.kr/hissta/download/): Xenium-Kidney (sample ID: XEN026), Xenium-Colon (sample ID: XEN017) and Xenium-Melanoma-Skin (sample ID: XEN027).

CosMx datasets

The non-small cell lung cancer (NSCLC) dataset, profiled using NanoString’s CosMx technology, includes eight samples from five NSCLC tissues, spanning a range of tumor characteristics. The CosMx NSCLC Lung dataset used in this study was downloaded from https://nanostring.com/products/cosmx-spatial-molecular-imager/nsclc-ffpe-dataset/. For our analysis, we used data from Lung5-2, one of the samples in this dataset, which consists of 30 fields of view. We selected the Lung5-2 to broaden the evaluation of segmentation methods across different sections of NSCLC tissue samples. This allows us to assess the consistency and robustness of the segmentation methods in varied yet related contexts. Specifically, we analyzed one slice containing 980 gene names and with a pixel dimension of 5,472 × 3,648 × 1 (grayscale) to evaluate the segmentation methods. Additionally, we used the human pancreatic tissue dataset from CosMx, which was also obtained from https://nanostring.com/products/cosmx-spatial-molecular-imager/ffpe-dataset/cosmx-smi-human-pancreas-ffpe-dataset/. This dataset includes 18,946 gene names and has a pixel dimension of 4,255 × 4,255 × 1 (grayscale). While the NSCLC dataset has been previously used in studies on spatial transcriptomic segmentation, the pancreatic tissue dataset has not been previously used in segmentation analysis. We included it to evaluate segmentation performance across tissue types and anatomical contexts. We additionally included two human datasets from the HISSTA repository (https://kbds.re.kr/hissta/download/): CosMx-Stomach (sample ID: COS061) and CosMx-Ovarian Cancer (sample ID: COS063).

Stereo-seq datasets

The Stereo-seq datasets used in this analysis, which include DAPI images and identified gene expressions (bin 1), were obtained from https://db.cngb.org/stomics/mosta/download/ for specimens labeled as ‘whole mouse brain’ and ‘embryo E12.5_E1S3’. Each dataset includes accompanying high-resolution images, with dimensions of 26,449 × 26,455 × 1 and 11,749 × 16,299 × 1 pixels, respectively. Compared to other technologies like Xenium, these datasets encompass a significantly broader array of genes, featuring 25,877 and 18,085 distinct gene names, respectively. These datasets have been previously used in segmentation analyses.

STARmap PLUS dataset

The STARmap PLUS dataset used in our analysis corresponds to the mouse primary motor cortex and was obtained from Zenodo (https://doi.org/10.5281/zenodo.7332091), initially published by Zeng et al.⁴⁸. This dataset integrates histopathology with spatial transcriptomics, including 1,022 unique gene names across 122,442 cells. The tissue was profiled using STARmap PLUS technology, enabling the evaluation of segmentation methods in a multimodal context that combines spatial gene expression with structural tissue features. This dataset was included to extend the analysis to Alzheimer’s disease mouse models and multimodal tissue representations.

MERFISH dataset

The MERFISH dataset used in our study profiles the mouse primary motor cortex and was obtained from a study by Zhang et al.⁶⁶. This dataset includes 258 genes profiled across approximately 300,000 cells and provides high-resolution SST data. Including the MERFISH dataset enables us to assess segmentation performance on dense single-molecule data and to compare results across spatial transcriptomics platforms within the same brain region. We incorporated additional MERFISH datasets from the HISSTA database (https://kbds.re.kr/hissta/download/), including mouse prostate (sample ID: MER086) and mouse uterus (sample ID: MER088). We further analyzed a MERSCOPE Brain-MERFISH dataset from the Vizgen public repository (https://console.cloud.google.com/storage/browser/public-datasets-vizgen-merfish/), profiling mouse brain tissue at high spatial resolution, to assess segmentation performance on dense single-molecule spatial transcriptomics data generated using the MERSCOPE platform.

Reference datasets for single-cell spatial data cell-type annotation

We use publicly available reference datasets to annotate the cells in the spatial transcriptomics datasets in our analysis. For the breast cancer dataset, we utilized an scRNA-seq dataset⁶⁷. This dataset includes 12,716 cells, with expression profiles for 308 genes and 9 cell types. For the NSCLC dataset, we used an scRNA-seq dataset downloaded from the CellxGene database⁶⁸. It contains 193,108 cells across 33,145 genes and provides annotations for 60 distinct cell types.

For the human pancreatic tissue dataset, we utilized an scRNA-seq dataset⁶⁹ that includes 13,497 cells, with expression profiles for 58,482 genes across 15 cell types. We used the annotation database from the ALLEN BRAIN ATLAS⁷⁰ for the whole mouse brain dataset, focusing on neural cell types. This encompasses 12 cell types across 1,679 samples and 24,057 genes. The dataset was obtained from https://brain-map.org/. Lastly, we used a large reference dataset⁷¹ consisting of approximately 2 million cells and 26,183 genes for the embryo tissue dataset.

A uniform preprocessing pipeline for spatial transcriptomics data

Comparing methodologies requires standardizing the data. Therefore, we standardized the raw data across different spatial transcriptomics technologies by processing the transcriptomic information from all platforms into a uniform format. Each transcript file was organized with the following columns: geneID, MIDCounts, cellID and the x and y coordinates. Additionally, images were standardized and converted to TIFF format from various formats (JPEG, PNG, OME-TIFF and TIFF) to ensure compatibility with the segmentation tools. Below, we detail the considerations made during the standardization process.

HVG selection

In cases where the tested segmentation tools were either ineffective or impractical due to excessive processing times, the top 1,000 HVGs were selected according to the guidelines provided by Seurat (version 5)⁷².

Unified coordinates space

To standardize the coordinate range across different spatial transcriptomics technologies, we mapped the raw coordinates from each platform to fit within the minimum and maximum range of the corresponding image dimensions (width and height). This mapping created a unified coordinate system based on image pixels, simplifying interpretation and visualization.

Homogeneous image color scale

All images were converted to grayscale before being processed with the segmentation tools, facilitating a uniform comparison framework.

Spatial transcriptomics cell-type annotation

Uniform spatial transcriptomics analysis was conducted on raw data from various spatial transcriptomics technologies, using the Seurat v5 pipeline with the SCTransform normalization method. To annotate these datasets, we used reference datasets, as discussed in the previous section. These reference datasets were processed according to the same guidelines as in Seurat version 5. The corresponding cell-type metadata were transferred using the FindTransferAnchors and TransferData functions.

Equal patching strategy

To address the computational demands of specific models, we divided the raw data—images and transcript files—into patches, ensuring consistent input files across the models compared, thus eliminating discrepancies arising from variations in the input data. Patch sizes were selected arbitrarily to yield a manageable number of patches per dataset. For the breast cancer, NSCLC, human pancreatic tissue, whole mouse brain and embryo datasets, the chosen patch sizes were 3,600 × 3,600, 1,200 × 1,200, 1,200 × 1,200, 1,200 × 1,200 and 2,400 × 2,400, respectively.

Segmentation methods and parameter settings

Our study evaluated 11 segmentation methods, categorized into three types: methods utilizing only transcript information, those combining both image and transcript data, and methods that rely solely on images. Below, we briefly overview each method’s fundamental principles and technical specifications.

Watershed segmentation

The Watershed segmentation pipeline, implemented in Spateo⁷³, delineated nuclei from stained spatial transcriptomics images with precise parameter settings. First, the function st.cs.mask_nuclei_from_stain (adata, otsu_classes = 4, otsu_index = 1) was executed to perform a multi-class Otsu thresholding, dividing the image into four intensity categories (as specified by otsu_classes = 4). The second category, defined by otsu_index = 1 was selected to represent the foreground or nuclei, effectively distinguishing them from the background. A binary mask was then generated, highlighting the regions where potential nuclei are present. Following masking, peak detection was carried out using st.cs.find_peaks_from_mask (adata, ‘stain’, 7), where a Gaussian filter with a seven-pixel radius was applied to smooth the stain image and identify local maxima within the masked regions. These maxima served as markers for the Watershed algorithm. The final step involved running st.cs.watershed (adata), which applied the Watershed algorithm to the blurred image, expanding regions from the detected peaks to segment individual nuclei. This algorithm delineated the boundaries of each nucleus based on intensity gradients in the stain image, providing distinct, labeled regions (nuclei) for further analysis. This pipeline was used to analyze the information from the patched image.

For the Watershed baseline, segmentation outcomes are mainly influenced by the Otsu thresholding parameters (otsu_classes, otsu_index) used for nuclei masking and by the peak-detection window size controlling seed density before Watershed propagation. These parameters strongly affect foreground detection and over-segmentation versus under-segmentation, making Watershed sensitive to image contrast and staining quality.

CellPose: universal geometric model for cell segmentation

CellPose is a cell segmentation algorithm utilizing a universal geometric model to address various cellular imaging challenges. This method uses a deep-learning framework with convolutional neural networks (CNNs) to segment cells based on shape and internal structure. Instead of direct segmentation, CellPose generates vector fields that represent cell boundaries and contours, thereby refining segmentation results. The algorithm uses a U-Net-like architecture with enhancements such as residual blocks to mitigate the vanishing gradient problem and improve performance by doubling the network depth. Global average pooling is applied to the smallest convolutional maps to capture image-specific style vectors.

In this study, the denoise.CellposeDenoiseModel function was initialized with GPU acceleration (gpu = True) for efficiency. Configured with the type ‘cyto3’ for cytoplasmic structures and using a denoising function (restore_type = ‘denoise_cyto3’), this model improves image quality by reducing noise. Segmentation was performed on single-channel grayscale images.

The model.eval() function generated outputs, including masks for segmented regions, flows representing vector fields with predicted cell contours, styles for variations in appearance and diameters for cell diameters. The segmentation masks were stored, generating the region-of-interest table. This pipeline was applied to the patched images.

SCS

SCS introduces a cell segmentation method that combines staining and transcriptomics data. The process begins with the traditional Watershed algorithm to detect nuclei from staining images. Subsequently, it leverages a transformer neural network model to predict the relationship of transcriptomic spots to cell centers, determining whether spots are within cells or part of the extracellular matrix. A gradient flow-tracking algorithm then groups these spots based on their directional predictions toward nuclei centers, assigning them to specific cells. This integration of imaging and dense transcriptomic data allows SCS to facilitate subcellular analyses such as RNA localization, thus providing insights into cellular functions and interactions at the molecular level. The innovative aspect of SCS lies in its gradient flow-tracking algorithm, which groups spots into cells based on their predicted directional gradients toward the nuclei centers. This approach effectively overcomes common segmentation challenges, such as overlapping cells and unclear boundaries. SCS was executed using the ‘scs.segment_cells’ function with the align = ‘rigid’ option to ensure rigid alignment of images. This alignment method corrects for misalignment by adjusting the translation (shifting) and rotation, but does not involve scaling or shearing.

Baysor segmentation

Baysor introduces a cell segmentation method for imaging-based spatial transcriptomics. Baysor uniquely combines molecular position data with optional auxiliary staining to optimize the delineation of two-dimensional or three-dimensional cell boundaries based on the joint likelihood of transcriptional composition and cell morphology. Baysor uses a Bayesian mixture model approach coupled with a Markov random field to maintain spatial coherency among cells. Optionally, additional morphological data from stains can be incorporated to refine cell boundary delineation. In this study, Baysor was executed using transcriptomic information only, without auxiliary staining or prior image-based segmentations. This configuration was chosen to ensure a method-independent and consistent benchmarking setup across datasets, avoiding variability introduced by differences in staining quality, availability or dependence on an initial segmentation method. Importantly, this choice is supported by prior work: the original BIDCell study evaluated Baysor both with and without a prior nuclei segmentation using the CellSPA evaluation framework³⁵. Based on this evidence, we used the transcript-only Baysor configuration in our benchmark. A fixed scale value of 100 was set with a scale standard deviation allowance of 50%.

For Baysor, segmentation is most sensitive to the cell size prior (segmentation.scale, scale_std) and the molecule-count thresholds (min_molecules_per_cell, min_molecules_per_segment), which together control how readily molecules are grouped into cells versus left as background. Runtime and fit stability are mainly driven by the number of iterations, while n_clusters influences how strongly cell-type-like structure is enforced during segmentation.

BIDCell

The BIDCell cell segmentation method introduces a self-supervised deep-learning framework tailored for adequate cell delineation in SST. It utilizes a Bidirectional U-Net3⁺ architecture, integrating bidirectional connections between the encoder and decoder to capture features at multiple scales, resulting in accurate segmentation. BIDCell’s approach leverages biologically informed loss functions that utilize relationships between spatial gene expressions and cell morphology, eliminating the need for manual annotations. This framework incorporates insights from existing scRNA-seq data to guide the segmentation process, ensuring cell shape prediction and enhancing cell expression purity. During inference, the trained model generates segmentation masks, refined using morphological operations such as erosion and dilation to correct artifacts and improve mask quality. Connected component analysis is applied to accurately identify and label individual cells, facilitating the precise segmentation of complex or crowded cell structures. All experiments were conducted using the default proposed parameters for the BIDCell method.

Key BIDCell parameters affecting segmentation include spatial scaling and alignment settings (target_pix_um, base_pix_x/y, base_ts_x/y) and model configuration parameters such as patch_size and loss weights (ne_weight, os_weight, cc_weight, pos_weight, neg_weight), which together regulate cell boundary placement and transcript assignment.

SAM

The SAM paper introduces a foundation model for image segmentation, designed to generalize across a wide range of segmentation tasks using a new approach to model training and data utilization. This SAM model is capable of real-time, prompt-based segmentation, making it highly flexible for interactive use. SAM comprises three main components: an image encoder, a prompt encoder and a mask decoder. The image encoder utilizes a Vision Transformer (ViT) to convert images into comprehensive embeddings. The prompt encoder, capable of handling various input forms such as points, boxes and free-form text, generates embeddings that guide the segmentation process. The mask decoder, a lightweight component that is computationally less intensive, synthesizes these embeddings to produce accurate segmentation masks based on input prompts.

SAM’s training leverages a data engine that iterates between model-assisted data annotation and using this newly annotated data to refine the model. This cycle significantly expands the training dataset, making SAM robust against diverse data distributions. We used a pretrained ViT-H model checkpoint for initialization and the ‘SamAutonaticMaskGenerator()’ function to generate the masks for each image. The generated masks were then processed to extract cellular coordinates.

To facilitate reproducibility and practical reuse, we report the main parameters governing mask generation for SAM-based models. For SAM, we used the default configuration provided by the authors, including enabling multiple mask outputs (multimask_output = TRUE), a mask threshold of 0.0, a predicted intersection-over-union (IoU) threshold of 0.88 and a stability score threshold of 0.95. These settings prioritize high-confidence and stable segmentations in image-based contexts.

SAM2

SAM2, Segment Anything in Images and Videos, introduces an advanced model designed for video and image segmentation, addressing the challenges of spatiotemporal segmentation tasks across diverse data types. SAM2 uses a transformer architecture with streaming memory, enabling real-time video frame processing and effective memory utilization to track objects over time. A significant feature of SAM2 is its ability to utilize prompts, such as clicks, boxes or masks on any frame, guiding the segmentation process throughout the video sequence. The model is designed to be highly interactive, allowing for iterative refinements of segmentation by updating prompts in subsequent frames. SAM2 also benefits from a data engine that iteratively improves the model by using it to generate and refine training data, leading to a robust model capable of handling a broad range of segmentation tasks. This approach allows SAM2 to achieve higher accuracy and efficiency in segmenting complex video and image content compared to previous models.

In this study, we initialized the SAM2 model from a pretrained checkpoint (sam2_hiera_large) and configured it to process images. We then used the ‘SAM2AutomaticMaskGenerator()’ function to generate masks for each image.

For reproducibility and to guide users wishing to adapt SAM2 to their own data, we highlight a small set of key configuration parameters that most strongly influence mask generation in our experiments: enabling high-resolution feature extraction (use_high_res_features_in_sam = true), allowing multiple candidate masks per region (multimask_output_in_sam = true) and configuring memory encoding using a sigmoid scale of 20.0 and bias of −10.0 (sigmoid_scale_for_mem_enc = 20.0, sigmoid_bias_for_mem_enc = −10.0). These settings follow the authors’ recommended configuration for stable image segmentation and can be adjusted depending on dataset characteristics and desired segmentation granularity.

mSAM

The mSAM method extends SAM to multidimensional microscopy data, offering both automatic and interactive segmentation capabilities. By fine-tuning SAM on light and electron microscopy images, mSAM enhances segmentation quality across various imaging conditions. In our study, mSAM was used to generate segmentation masks by first computing image embeddings using a fine-tuned SAM model (vit_b_lm). Then, an instance of AutomaticMaskGenerator (AutomaticMaskGenerator()) was initialized with the original image and its precomputed embeddings to generate predictions.

To support reproducibility and user-guided customization, we report the core parameters used for mSAM mask generation. mSAM inherits SAM’s automatic mask generation framework and was executed using the default configuration recommended by the authors, including enabling multiple mask outputs (multimask_output = TRUE), a mask threshold of 0.0, a predicted IoU threshold of 0.88 and a stability score threshold of 0.95. These parameters balance segmentation sensitivity and mask stability for multidimensional microscopy images. Users may adjust the predicted IoU and stability thresholds to favor either higher recall or stricter mask confidence depending on tissue complexity and imaging quality.

CelloType: unified model for segmentation and classification

CelloType is an end-to-end transformer-based model designed for simultaneous cell segmentation and classification of microscopy images. Utilizing a Swin Transformer backbone, it extracts multiscale features, which are processed through the DINO object detection module and the MaskDINO segmentation head. This integrated approach enables precise delineation of cellular and noncellular structures while assigning cell-type labels.

Our study used a pretrained model (tissuenet_model_0019999.pth) provided by the CelloType framework. The model was initialized with the CelloTypePredictor using a confidence threshold of 0.3 and a maximum detection number of 1,000, and was run on a GPU-enabled environment (device = ‘cuda’). The inference process began by loading grayscale microscopy images and converting them to RGB format compatible with the model. Each image was processed through the model’s predict() function to generate instance segmentation masks.

The predicted masks were then post-processed to extract nonzero pixel coordinates and corresponding cell instance values. The results were saved in tab-separated files containing x–y coordinates and cell labels for each detected cell. This procedure was consistently applied across various datasets, including mouse primary motor cortex STARmap PLUS images, using batch processing pipelines to ensure reproducibility and scalability across conditions.

Swin Transformer: hierarchical vision transformer using shifted windows

Swin Transformer introduces a hierarchical vision transformer architecture that computes representations using shifted windows, enabling efficient modeling of visual data at various scales. By limiting self-attention computation to nonoverlapping local windows and shifting these windows between layers, Swin Transformer achieves linear computational complexity relative to image size while facilitating cross-window connections. This design allows the Swin Transformer to serve as a general-purpose backbone for computer vision tasks, including image classification, object detection and semantic segmentation, outperforming previous models on benchmarks like COCO and ADE20K. In this study, we used the pretrained model and inference code provided by the SegGal framework (https://boomstarcuc.github.io/SegGal/). Input images were converted to the COCO JSON format following SegGal specifications. The model was then applied to generate instance-level segmentation masks, which were post-processed to extract cellular coordinates for downstream analysis.

Cascade Mask R-CNN (Seesaw)

The Seesaw Loss method addresses the challenge of long-tailed distributions in instance segmentation by dynamically rebalancing the gradients of positive and negative samples for each category. It introduces two complementary factors: a mitigation factor that reduces the influence of head classes on tail classes based on the ratio of cumulative training instances, and a compensation factor that increases the penalty for misclassified cases to avoid false positives in tail categories. This approach enables Seesaw Loss to effectively mitigate the bias toward head classes, leading to significant performance improvements on long-tailed datasets like LVIS when integrated into frameworks such as Mask R-CNN. In this study, we used the pretrained model and inference code provided by the SegGal framework (https://boomstarcuc.github.io/SegGal/). Input images were converted to the COCO JSON format following SegGal specifications. The model was then applied to generate instance-level segmentation masks post-processed to extract cellular coordinates for downstream analysis.

Exclusion criteria

During benchmarking, both Cascade Mask R-CNN (Seesaw) and the Swin Transformer consistently failed to produce biologically meaningful cell segmentations across datasets. As illustrated in Supplementary Fig. 17, these models exhibited a pronounced grid-like artifact pattern, in which predictions were dominated by patch boundaries rather than underlying cellular structures. This failure mode resulted in systematic over-segmentation and spatially regular artifacts that are incompatible with downstream transcriptomic evaluation. Because these outputs do not represent valid cell segmentations, we excluded these two methods from the quantitative benchmarking in Fig. 2. The inclusion of Supplementary Fig. 16 provides a visual justification for this exclusion and ensures transparency in the benchmarking decision.

Evaluation metrics

To thoroughly assess the quality of our segmentation methods, we adopted the evaluation metrics from the BIDCell framework, specifically the CellSPA metrics. This framework evaluates segmentation performance across five complementary categories, providing a comprehensive view of the results’ accuracy and biological relevance. Below, we briefly describe each metric used.

Baseline metrics: overall characteristics

M1: number of cells: The total number of cells identified within the segmentation process.
M2: the proportion of transcripts assigned: The percentage of total transcripts accurately assigned within the boundaries of the identified cells.

Cell-level QC metrics

m3: number of transcripts per cell: The average number of transcripts detected per cell, providing insights into the transcriptional activity within cells.
M4: number of genes expressed per cell: The average number of genes expressed per cell reflects cellular complexity.
M5: cell area: The total spatial area occupied by each cell. It is calculated using equation (1):

Density = \frac{\sum_{i \in I} n_{i}}{A}

where $\sum_{i \in I} n_{i}$ represents the sum of all transcripts over the set I, and A represents the cell area.

Cell morphology metrics

M6: elongation: Measures the elongation of a cell based on its bounding box dimensions. It is calculated according to equation (2):

Elongation = \frac{H_{bb}}{W_{bb}}

where $H_{bb}$ and $W_{bb}$ represent the height and width of the bounding box, respectively. This metric indicates the cell’s shape elongation, which can affect cellular function and behavior interpretation.

M7: circularity: Measures how circular a cell is, emphasizing its compactness and regularity. The formula used is given in equation (3):
$Circularity = \frac{4_{π} A}{P_{convex}^{2}}$ 3
where $A$ is the area and $P_{convex}$ is the convex parameter. A circularity value of 1 indicates a perfect circle, with values decreasing as the cell shape deviates from this ideal.
M8: sphericity: Reflects the degree to which a cell resembles a sphere. It is quantified as shown in equation (4):
$Sphericity = \frac{R_{I}}{R_{C}}$ 4
where $R_{I}$ is the radius of the inscribing circle and $R_{C}$ is the radius of the circumscribing circle. A sphericity of 1 suggests a perfect sphere.
M9: compactness: Compares the area of the cell to a circle with the same perimeter. It is calculated using equation (5):
$Compactness = \frac{4_{π} A}{P_{cell}^{2}}$ 5
where $A$ is the area and $P_{cell}$ is the cell parameter. Higher compactness indicates a more efficient cellular configuration.
M10: convexity: Describes the ratio of the convex perimeter of a cell to its actual perimeter. The formula is shown in equation (6):
$Convexity = \frac{P_{convex}}{P_{cell}}$ 6
M11: eccentricity: Measures the aspect ratio of the cell’s shape. It is defined according to equation (7):
$Eccentricity = \frac{L_{major}}{L_{minor}}$ 7
where $L_{major}$ and $L_{minor}$ are the lengths of the major and minor axes, respectively.
M12: solidity: Assesses the solidity of the cell by comparing its area to the area of its convex hull. It is calculated as shown in equation (8):

Solidity = \frac{A}{A_{convex}}

where $A_{convex}$ is the area of the convex hull. Solidity close to 1 indicates a cell with minimal concavities and a smooth boundary.

Gene-level QC metrics

M13: proportion of cells expressing each gene: This metric assesses the distribution of gene expression across all segmented cells, providing insights into the transcriptional uniformity or diversity within the sample.

Segmented cell expression purity

M14: concordance between re-quantified spatial transcriptomic expression data and original spatial expression data: Measures the similarity between the gene expression profiles of re-segmented cells (using the compared segmentation methods) and original spatial segmented data from data generators. The similarity is quantified using Pearson correlation coefficients on average log-normalized gene expression data according to equation (9):
$\frac{cov (X, Y)}{σ_{X} σ_{Y}}$ 9
M15: purity of expression: Identifies the precision, recall and F1 score for expression of selected positive and negative gene markers, enhancing the assessment of expression homogeneity. The F1 score, a measure of a test’s accuracy, is calculated using the harmonic mean of precision and recall.

Spatial characteristics

M16: cell-type diversity and variability: Analyzes the variation in cell characteristics across spatially distinct regions. Each region’s cell-type diversity is calculated using Shannon entropy, indicating a higher diversity with a higher entropy value. The relationship between cell-type diversity and cell-characteristic variability is quantified using the Pearson correlation according to equation (10):

Pearson correlation = \frac{cov (D, C)}{σ_{D} σ_{C}}

where $D$ represents the diversity (entropy), and $C$ represents the characteristic’s coefficient of variation within each region.

Neighboring contamination

M17: contamination assessment: This metric assesses the effectiveness of cell segmentation in preventing signal contamination between adjacent cells of different types. The Euclidean distance is calculated between each cell in one type (for example, cell type A) and its nearest neighbor of another type (for example, cell type B). Cells are grouped based on their proximity to different cell types, and within each group, the proportion of cells that incorrectly express markers specific to neighboring cell types is measured. Precisely, the proportion of cells expressing a selected negative marker, typically associated with cell type B, is calculated. It is anticipated that methods with less contamination will show lower levels of such marker expression among cells close to different cell types, indicating higher segmentation accuracy and biological validity.

Downstream biological evaluation metrics

To evaluate the biological accuracy and signal-to-contamination ratio of each segmentation output—and to study the trade-offs between tools and technologies—we applied a set of established downstream analysis metrics⁷⁴. Together, these metrics quantify three dimensions of performance—(i) consistency, (ii) contamination and (iii) interpretability of the segmented expression profiles across strategies. Specifically, we assessed scaled expression values for overlapping genes across cell types, evaluated the presence of contamination signals by examining spatial distances to expected cell-type locations using the CellSPA framework (metric M17) and performed t-tests on marker-gene expression fingerprints to detect tool and technology-specific enrichment. Additionally, we used the Giotto R package to assess the significance of ligand–receptor interactions for each cell type, allowing a robust comparison of intercellular communication networks recovered by different segmentation methods.

Overall characteristics

Number of high-quality cells segmented (meeting QC criteria: total transcripts between 20 and 2,000, and total genes per cell greater than 20) (M1)
Proportion of transcripts assigned (M2)

Cell-level quality

Number of transcripts per cell (M3)
Number of genes expressed per cell (M4)

Gene-level quality

Average ratio of tool’s proportion of cells expressing each gene against reference values by tools (M13)

Cell morphology

Correlation between cell-type elongation values from the tool’s segmentation and the reference segmentation (M6)
Product between cell-type elongation correlations (tools versus reference) and average total transcripts per cell by tools (M6)

Expression and cell composition similarity against reference

Average diagonal value from the cell-type log₂ expression correlation matrix comparing the reference and the new tool’s segmentation
Product between log₂ cell-type average correlations and average total transcripts per cell by tools (M14)
Product between cell-type average percentage of gene expressed and average total transcripts per cell by tools
The average correlation of cell-type proportions between reference and tool’s outcomes by cell type for each tool

Expression purity

Average ratio of tool’s positive purity score obtained against reference positive purity score by tools (M15)
The average ratio of the tool’s purity score obtained against the average total transcripts per cell (M15)

Spatial characteristics diversity

Correlation between the coefficient of variation of total transcripts against entropy by tools (M16)

UMAP representation consistency with cell types

Silhouette scores derived from principal component analysis representation components and cell-type annotations for each tool

For scoring, we applied transformations to ensure comparability. The following transformations were applied:

Scaling: The values obtained for the different models across each dataset were scaled to a range of 0 to 1, where 0 represents the lowest value and 1 represents the highest, using the equation (11):

Scaled value = \frac{Original value - Min value}{Max value - Min value}

Ranking of the models

We used two averaging methods. The first calculated a global average (mean) across all scaled metrics to derive an overall ranking based on the relative rankings of each metric. The second method groups the metrics into those, as mentioned above, eight different subcategories, with averages (means) within each subcategory and then averages across the subcategories to provide a final ranking. This approach ensures that the final ranking equally and comprehensively reflects the model’s performance across different evaluation aspects.

In addition, to explicitly account for segmentation coverage, we computed a coverage-adjusted Overall score by weighting the unadjusted Overall score by the ratio of high-quality segmented cells produced by each method relative to the number of reference cells available in each dataset. This adjustment penalizes methods that achieve high metric scores while segmenting only a small fraction of cells, and rewards methods that balance segmentation quality with comprehensive cell recovery. The adjusted Overall score was used alongside the unadjusted ranking and the Times winner metric to provide a complementary, coverage-aware assessment of segmentation robustness across datasets.

Unified pipeline for spatial segmentation benchmarking

Cell segmentation in single-cell spatial transcriptomics is essential for tasks like assessing cellular heterogeneity and identifying cell types. However, accurately delineating cell boundaries can be challenging due to the complexity of cellular structures. Our pipeline simplifies the segmentation process, saving researchers time and effort. It supports various methods and ensures proper preprocessing and evaluation while helping users choose the most appropriate method for their tissue type without needing deep technical knowledge. By streamlining these steps, our pipeline improves accuracy and reduces the workload, allowing researchers to concentrate on extracting meaningful biological insights.

This pipeline was constructed using the Nextflow workflow management system version 23.10.0, designed to facilitate the compartmentalized use of components for benchmarking cellular segmentation methods in spatial transcriptomics. The pipeline is organized into three primary blocks: data preprocessing, segmentation tool application and evaluation of segmentation outcomes.

In the data preprocessing block, transcript and image files are imported and undergo a series of transformations to standardize and optimize the data for subsequent analysis. Key variables necessary across all segmented methods are selected and renamed to generate standard file formats for downstream tasks. Axis alignment to ensure uniformity between transcript and image datasets is performed through scaling transcripts axis x and y ranges to the pixel space of the corresponding image. The images are cropped to max–min transcripts available in either x and y ranges to reduce and remove black zones without cells, thereby reducing time consumption in later applying segmentation methods. Additionally, image data undergo a color scale adjustment, where multiple channel images (for example, RGB) are transformed into grayscale, to standardize downstream analysis across benchmarked tools. To facilitate the segmentation process and bypass specific tool limitations, both image and transcript-aligned files are divided into smaller, manageable patches of specified dimensions.

The second block focuses on applying segmentation methodologies to the patched files previously generated in the first block. Each method predicts cell boundaries and systematically organizes the results into designated folders corresponding to the segmentation techniques.

Finally, the evaluation stage includes an evaluation script aggregating the segmentation results from the patched files into the original spatial context. Additionally, it processes the reference scRNA-seq data and maps the cell-type annotations to the spatial transcriptomics dataset. This block also computes the benchmarking metrics and generates multiple visualizations across all segmentation outcomes. All steps are designed to run in parallel, leveraging Nextflow’s capabilities to efficiently execute downstream steps as input files from previous steps are generated.

As an open and expandable resource, the pipeline outlines essential preprocessing steps necessary for running various segmentation methods, incorporating both transcript and image data. It also provides illustrative examples for implementing segmentation methods that utilize either one or both types of input data. For instance, SCS is highlighted as a method that leverages both data types, while Cellpose relies solely on image data. These examples serve as templates for users to integrate their custom segmentation methods or utilize existing ones, enhancing the adaptability and utility of the pipeline. Each block is structured to use different scripts, operating within an Anaconda environment that includes the necessary libraries and specific versions of Python or R required for each script.

Segmentation outcomes and benchmarking reproducibility

To ensure reproducibility, we created separate Anaconda environments for each model’s specific Python version, as specified by the developers at the time of this benchmarking. Each environment includes the necessary versions of the libraries required to run the models accurately. Codes used in this benchmarking, including the baseline Nextflow unified pipeline and Anaconda environments, are available on GitHub via https://github.com/TranslationalBioinformaticsUnit/Benchmarking-SegmentationMethods-ST/.

For annotation transfer analyses from reference datasets to spatial transcriptomics data, as well as for evaluation and visualization of the results, we utilized an Anaconda environment that includes the latest available version of R. This setup guarantees that the analysis can be replicated under the same conditions and with the identical software versions used in the study.

Spatial transcriptomics technology evaluation

To assess how spatial transcriptomics technologies affect segmentation-derived molecular profiles, we performed a cross-platform comparison using Stereo-seq, MERFISH and STARmap PLUS datasets. Segmentation outputs from a panel of representative tools—including Watershed-based algorithms, Cellpose, Cellotype, SCS, SAM, mSAM, Baysor and BIDCell—were standardized into SpatialExperiment objects for uniform processing. Gene expression matrices from each tool and technology were restricted to the intersection of all detected genes, and cell-type labels were harmonized across datasets to ensure consistent grouping of astrocytes, endothelial cells and microglia. For the dot plot analysis, we used the full set of 67 genes that overlapped by name across technologies and exhibited nonzero expression in at least one tool–technology–cell-type combination; in contrast, the Venn diagram reflects only overlapping genes with nonzero expression within each individual technology/tool, illustrating differences in gene detectability. Marker-fingerprint recovery was evaluated by converting objects to Seurat format and computing average scaled expression and percentage-expressing values for canonical markers. Negative-marker contamination was quantified using CellSPA’s distance-dependent NegMarkerVsDist metric, which assesses the proportion of cells expressing inappropriate markers across spatial distance bins. Marker enrichment per segmentation was estimated using t-tests comparing curated gene sets against the full expression matrix and reporting –log₁₀(P) values. Finally, ligand–receptor communication scores were computed with Giotto by constructing spatial networks from Delaunay triangulation and k-nearest neighbor graphs and estimating interaction significance (permutation importance or PI) across tools and technologies using 500 permutations.

Computer platform

All model analyses were performed on an HPC system with a single NVIDIA Tesla V100 GPU (1.29 GHz, 32 GB HBM2) specifically for deep-learning and segmentation tasks. The system also featured an Intel Skylake CPU (2.6 GHz, 32 cores) for CPU-bound processes, supported by 383 GB of RAM for handling large datasets and memory-intensive computations. An Intel Cascade Lake CPU node was used for the evaluation pipeline. Each node consists of 48 cores, operating at a clock speed of 4.20 GHz, and delivers 32 FLOPS per core. The Cascade Lake node is equipped with 3 TB of RAM and 6 TB of local storage, allowing for efficient handling of large datasets and high-performance computations.

Statistics and reproducibility

No statistical method was used to predetermine sample size. This study is a computational benchmark of publicly available spatial transcriptomics datasets, and datasets were selected to span multiple technologies, tissues, species, image types, gene-panel sizes and cell-count ranges. For platforms with multiple publicly available samples, representative samples were selected to ensure cross-method comparability, adequate molecular coverage and compatibility with the segmentation workflows. No experimental intervention was performed, and no control group was used because the study benchmarks segmentation outputs rather than testing treatment effects.

Data were excluded only according to predefined technical and QC criteria. Segmentation outputs that did not produce biologically meaningful cell masks, including outputs dominated by grid-like patch-boundary artifacts, were excluded from quantitative benchmarking. Low-quality segmented cells were filtered using transcript-count and gene-count thresholds as described above. Missing tools or metrics indicate method failure on the corresponding dataset, unavailable data or absent cell labels. The experiments were not randomized. The investigators were not blinded to allocation during experiments and outcome assessment, because all analyses were performed on publicly available datasets with known platform, tissue and method identities.

Unless otherwise stated, statistical summaries are descriptive and were computed across segmented cells, genes, cell types, spatial regions or segmentation methods, depending on the metric. For box plots of cell-level quality metrics, the unit of analysis is the high-quality segmented cell. Correlations were computed using the Pearson or Spearman correlation as specified for each analysis. Marker-enrichment analyses were performed using t-tests, and ligand–receptor interaction significance was estimated using the Giotto framework with 500 permutations. The full benchmarking workflow, including preprocessing, segmentation, evaluation and visualization scripts, is available in the associated GitHub repository to support reproducibility.

Reporting summary

Further information on research design is available in the Nature Portfolio Reporting Summary linked to this article.

Online content

Any methods, additional references, Nature Portfolio reporting summaries, source data, extended data, supplementary information, acknowledgements, peer review information; details of author contributions and competing interests; and statements of data and code availability are available at https://doi.org/10.1038/s41592-026-03272-x.

Supplementary information

Supplementary Information^{(37.6MB, pdf)}

Supplementary Figs. 1–17 and Tables 1–6.

Reporting Summary^{(2MB, pdf)}

Peer Review File^{(2.4MB, pdf)}

Source data

Source Data Figs. 1–6^{(102.2KB, xlsx)}

Statistical source data for Figs. 1–6.

Acknowledgements

King Abdullah University of Science and Technology supported this work. We thank all members of the lab for their constructive and critical feedback on the paper.

Author contributions

X.M.-d.-M. performed the benchmarking analyses, collected, curated and processed all datasets, implemented the computational workflows, generated figures, interpreted the results and drafted the paper. S.A.K. performed benchmarking analyses, contributed to dataset collection, implemented computational workflows and contributed to drafting of the paper. A.M. contributed to dataset collection. R.L., V.L. and N.A.K. provided conceptual input and critical feedback on the analyses and paper. D.G.-C. and J.T. supervised the work and contributed to editing of the paper and final revisions. All authors reviewed and approved the final paper.

Peer review

Peer review information

Nature Methods thanks Qinghua Jiang, Michael Morgan and the other, anonymous, reviewer(s) for their contribution to the peer review of this work. Peer reviewer reports are available. Primary Handling Editor: Madhura Mukhopadhyay, in collaboration with the Nature Methods team.

Data availability

All spatial transcriptomics datasets analyzed in this study were obtained from publicly available sources associated with Stereo-seq⁴⁶, STARmap PLUS⁴⁸, MERFISH⁶⁶, Xenium⁴⁴ and CosMx^45,75. Permanent source records and associated references are provided in refs. ^{44–46,66,75–77,} and described in the Methods. Direct links to the original data sources and to the standardized transcript and image files used for benchmarking are additionally provided through the associated GitHub repository (https://github.com/TranslationalBioinformaticsUnit/Benchmarking-SegmentationMethods-ST/). Source data are provided with this paper.

Code availability

All code used for data preprocessing, segmentation, evaluation and figure generation is publicly available at https://github.com/TranslationalBioinformaticsUnit/Benchmarking-SegmentationMethods-ST/.

Competing interests

The authors declare no competing interests.

Footnotes

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

These authors contributed equally: Xabier Martínez-de-Morentin, Sumeer A. Khan.

These authors jointly supervised this work: David Gomez-Cabrero, Jesper Tegner.

Contributor Information

Xabier Martínez-de-Morentin, Email: xabier.iribarren@kaust.edu.sa.

David Gomez-Cabrero, Email: david.gomezcabrero@kaust.edu.sa.

Jesper Tegner, Email: jesper.tegner@kaust.edu.sa.

Supplementary information

The online version contains supplementary material available at https://doi.org/10.1038/s41592-026-03272-x.

References

1.Ståhl, P. L. et al. Visualization and analysis of gene expression in tissue sections by spatial transcriptomics. Science353, 78–82 (2016). [DOI] [PubMed] [Google Scholar]
2.Tian, L., Chen, F. & Macosko, E. Z. The expanding vistas of spatial transcriptomics. Nat. Biotechnol.41, 773–782 (2023). [DOI] [PMC free article] [PubMed] [Google Scholar]
3.Marx, V. Method of the year: spatially resolved transcriptomics. Nat. Methods18, 9–14 (2021). [DOI] [PubMed] [Google Scholar]
4.Arora, R. et al. Spatial transcriptomics reveals distinct and conserved tumor core and edge architectures that predict survival and targeted therapy response. Nat. Commun.14, 5029 (2023). [DOI] [PMC free article] [PubMed] [Google Scholar]
5.Huang, S. et al. Spatial transcriptomics: a new frontier in cancer research. Clin. Cancer Bull.3, 13 10.1007/s44272-024-00018-8 (2024). [DOI]
6.Cheng, M. et al. Spatially resolved transcriptomics: a comprehensive review of their technological advances, applications, and challenges. J. Genet. Genom.50, 625–640 (2023). [DOI] [PubMed] [Google Scholar]
7.Li, Y., Stanojevic, S. & Garmire, L. X. Emerging artificial intelligence applications in spatial transcriptomics analysis. Comput. Struct. Biotechnol. J.20, 2895–2908 (2022). [DOI] [PMC free article] [PubMed] [Google Scholar]
8.Wang, L., Maletic-Savatic, M. & Liu, Z. Region-specific denoising identifies spatial co-expression patterns and intra-tissue heterogeneity in spatially resolved transcriptomics data. Nat. Commun.13, 6912 (2022). [DOI] [PMC free article] [PubMed] [Google Scholar]
9.Cui, Y. et al. DenoiseST: a dual-channel unsupervised deep learning-based denoising method to identify spatial domains and functionally variable genes in spatial transcriptomics. Preprint at bioRxiv 10.1101/2024.03.04.583438 (2024). [DOI]
10.Bergenstråhle, L. et al. Super-resolved spatial transcriptomics by deep data fusion. Nat. Biotechnol.40, 476–479 (2022). [DOI] [PubMed] [Google Scholar]
11.Zeng, Y. et al. Imputing spatial transcriptomics through gene network constructed from protein language model. Commun. Biol.7, 1271 (2024). [DOI] [PMC free article] [PubMed] [Google Scholar]
12.Sun, E. D., Ma, R. & Zou, J. SPRITE: improving spatial gene expression imputation with gene and cell networks. Bioinformatics40, i521–i528 (2024). [DOI] [PMC free article] [PubMed] [Google Scholar]
13.Pham, D. et al. Robust mapping of spatiotemporal trajectories and cell–cell interactions in healthy and diseased tissues. Nat. Commun.14, 7739 (2023). [DOI] [PMC free article] [PubMed] [Google Scholar]
14.Xu, H. et al. Unsupervised spatially embedded deep representation of spatial transcriptomics. Genome Med.16, 12 (2024). [DOI] [PMC free article] [PubMed] [Google Scholar]
15.Liu, Z., Wu, D., Zhai, W. & Ma, L. SONAR enables cell type deconvolution with spatially weighted Poisson-Gamma model for spatial transcriptomics. Nat. Commun.14, 4727 (2023). [DOI] [PMC free article] [PubMed] [Google Scholar]
16.Lu, Y., Chen, Q. M. & An, L. SPADE: spatial deconvolution for domain specific cell-type estimation. Commun. Biol.7, 469 (2024). [DOI] [PMC free article] [PubMed] [Google Scholar]
17.Coleman, K., Hu, J., Schroeder, A., Lee, E. B. & Li, M. SpaDecon: cell-type deconvolution in spatial transcriptomics with semi-supervised learning. Commun. Biol.6, 378 (2023). [DOI] [PMC free article] [PubMed] [Google Scholar]
18.Hao, M., Hua, K. & Zhang, X. SOMDE: a scalable method for identifying spatially variable genes with self-organizing map. Bioinformatics37, 4392–4398 (2021). [DOI] [PubMed] [Google Scholar]
19.Zhang, K., Feng, W. & Wang, P. Identification of spatially variable genes with graph cuts. Nat. Commun.13, 5488 (2022). [DOI] [PMC free article] [PubMed] [Google Scholar]
20.Kueckelhaus, J. et al. Inferring histology-associated gene expression gradients in spatial transcriptomic studies. Nat. Commun.15, 7280 (2024). [DOI] [PMC free article] [PubMed] [Google Scholar]
21.Liang, Q., Solis Soto, L., Haymaker, C. & Chen, K. LSGI: interpretable spatial gradient analysis for spatial transcriptomics data. Genome Biol.26, 238 (2025). [DOI] [PMC free article] [PubMed] [Google Scholar]
22.Blampey, Q. et al. Sopa: a technology-invariant pipeline for analyses of image-based spatial omics. Nat. Commun.15, 4981 (2024). [DOI] [PMC free article] [PubMed] [Google Scholar]
23.Chen, H., Li, D. & Bar-Joseph, Z. SCS: cell segmentation for high-resolution spatial transcriptomics. Nat. Methods20, 1237–1243 (2023). [DOI] [PubMed] [Google Scholar]
24.Palla, G. et al. Squidpy: a scalable framework for spatial omics analysis. Nat. Methods19, 171–178 (2022). [DOI] [PMC free article] [PubMed] [Google Scholar]
25.Hu, Y. et al. Benchmarking clustering, alignment, and integration methods for spatial transcriptomics. Genome Biol.25, 212 (2024). [DOI] [PMC free article] [PubMed] [Google Scholar]
26.Yan, L. & Sun, X. Benchmarking and integration of methods for deconvoluting spatial transcriptomic data. Bioinformatics39, btac805 (2023). [DOI] [PMC free article] [PubMed] [Google Scholar]
27.Yuan, Z. et al. Benchmarking spatial clustering methods with spatially resolved transcriptomics data. Nat. Methods21, 712–722 (2024). [DOI] [PubMed] [Google Scholar]
28.Wang, Y. et al. A systematic evaluation of computational methods for cell segmentation. Brief. Bioinform.25, bbae407 (2024). [DOI] [PMC free article] [PubMed] [Google Scholar]
29.Choe, K., Pak, U., Pang, Y., Hao, W. & Yang, X. Advances and challenges in spatial transcriptomics for developmental biology. Biomolecules13, 156 (2023). [DOI] [PMC free article] [PubMed] [Google Scholar]
30.Atta, L. & Fan, J. Computational challenges and opportunities in spatially resolved transcriptomic data analysis. Nat. Commun.12, 5283 (2021). [DOI] [PMC free article] [PubMed] [Google Scholar]
31.Kesidis, A. L., Krassanakis, V., Misthos, L. M. & Merlemis, N. patchIT: a multipurpose patch creation tool for image processing applications. Multimodal Technol. Interact.6, 111 10.3390/mti6120111 (2022). [DOI]
32.Nejat, P. et al. Creating an atlas of normal tissue for pruning WSI patching through anomaly detection. Sci. Rep.14, 3932 (2024). [DOI] [PMC free article] [PubMed] [Google Scholar]
33.Chen, H. et al. Recovering single-cell expression profiles from spatial transcriptomics with scResolve. Cell Rep. Methods4, 100864 (2024). [DOI] [PMC free article] [PubMed] [Google Scholar]
34.Fang, S. et al. Computational approaches and challenges in spatial transcriptomics. Genomics Proteomics Bioinformatics21, 24–47 (2023). [DOI] [PMC free article] [PubMed] [Google Scholar]
35.Fu, X. et al. BIDCell: biologically-informed self-supervised learning for segmentation of subcellular spatial transcriptomics data. Nat. Commun.15, 509 (2024). [DOI] [PMC free article] [PubMed] [Google Scholar]
36.Ravi, N. et al. SAM 2: Segment Anything in images and videos. In International Conference on Learning Representations (ICLR) Proceedings (ICLR, 2025).
37.Kirillov, A. et al. Segment Anything. In Proc. IEEE/CVF International Conference on Computer Vision (ICCV) 4015–4026 (IEEE Computer Society, 2023).
38.Archit, A. et al. Segment Anything for Microscopy. Nat. Methods22, 579–591 (2025). [DOI] [PMC free article] [PubMed] [Google Scholar]
39.Petukhov, V. et al. Cell segmentation in imaging-based spatial transcriptomics. Nat. Biotechnol.40, 345–354 (2022). [DOI] [PubMed] [Google Scholar]
40.Liu, Z. et al. Swin Transformer: hierarchical vision transformer using shifted windows. In Proc. IEEE International Conference on Computer Vision 9992–10002 10.1109/ICCV48922.2021.00986 (Institute of Electrical and Electronics Engineers, 2021). [DOI]
41.Wang, J. et al. Seesaw Loss for long-tailed instance segmentation. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 9695–9704 (IEEE Computer Society, 2021).
42.Stringer, C. & Pachitariu, M. Cellpose3: one-click image restoration for improved cellular segmentation. Nat. Methods22, 592–599 (2025). [DOI] [PMC free article] [PubMed] [Google Scholar]
43.Pang, M., Roy, T. K., Wu, X. & Tan, K. CelloType: a unified model for segmentation and classification of tissue images. Nat. Methods22, 348–357 (2024). [DOI] [PMC free article] [PubMed] [Google Scholar]
44.Janesick, A. et al. High resolution mapping of the tumor microenvironment using integrated single-cell, spatial and in situ analysis. Nat. Commun.14, 8353 10.1038/s41467-023-43458-x (2023). [DOI] [PMC free article] [PubMed] [Google Scholar]
45.He, S. et al. High-plex imaging of RNA and proteins at subcellular resolution in fixed tissue by spatial molecular imaging. Nat. Biotechnol.40, 1794–1806 10.1038/s41587-022-01483-z (2022). [DOI] [PubMed] [Google Scholar]
46.Chen, A. et al. Spatiotemporal transcriptomic atlas of mouse organogenesis using DNA nanoball-patterned arrays. Cell185, 1777–1792.e21 10.1016/j.cell.2022.04.003 (2022). [DOI] [PubMed] [Google Scholar]
47.Androvic, P. et al. Spatial transcriptomics-correlated electron microscopy maps transcriptional and ultrastructural responses to brain injury. Nat. Commun.14, 4115 (2023). [DOI] [PMC free article] [PubMed] [Google Scholar]
48.Zeng, H. et al. Integrative in situ mapping of single-cell transcriptional states and tissue histopathology in a mouse model of Alzheimer’s disease. Nat. Neurosci.26, 430–446 10.1038/s41593-022-01251-x (2023). [DOI] [PMC free article] [PubMed] [Google Scholar]
49.Di Tommaso, P. et al. Nextflow enables reproducible computational workflows. Nat. Biotechnol.35, 316–319 (2017). [DOI] [PubMed] [Google Scholar]
50.Lee, W.-H., Wang, G.-M., Seaman, L. B. & Vannucci, S. J. Coordinate IGF-I and IGFBP5 gene expression in perinatal rat brain after hypoxia–ischemia. J. Cereb. Blood Flow Metab.16, 227–236 (1996). [DOI] [PubMed] [Google Scholar]
51.Miya, K., Keino-Masu, K., Okada, T., Kobayashi, K. & Masu, M. Expression of heparan sulfate endosulfatases in the adult mouse brain: co-expression of Sulf1 and dopamine D1/D2 receptors. Front. Neuroanat.15, 726718 (2021). [DOI] [PMC free article] [PubMed] [Google Scholar]
52.Ohayon, D. et al. Sulfatase 2 promotes generation of a spinal cord astrocyte subtype that stands out through the expression of Olig2. Glia67, 1478–1495 (2019). [DOI] [PMC free article] [PubMed] [Google Scholar]
53.Chen, Z., Gao, M., Su, Y., Liu, P. & Sun, B. Running promotes transformation of brain astrocytes into neuroprotective reactive astrocytes and synaptic formation by targeting Gpc6 through the STAT3 pathway. Front. Physiol.12, 633618 (2021). [DOI] [PMC free article] [PubMed] [Google Scholar]
54.Kim, H. Y., Choi, J. S., Cha, J. H., Choi, J. Y. & Lee, M. Y. Expression of vascular endothelial growth factor receptors Flt-1 and Flk-1 in embryonic rat forebrain. Neurosci. Lett.425, 131–135 (2007). [DOI] [PubMed] [Google Scholar]
55.Nitta, T. et al. Size-selective loosening of the blood-brain barrier in claudin-5-deficient mice. J. Cell Biol.161, 653–660 (2003). [DOI] [PMC free article] [PubMed] [Google Scholar]
56.Jia, C. et al. Vitronectin from brain pericytes promotes adult forebrain neurogenesis by stimulating CNTF. Exp. Neurol.312, 20–32 (2019). [DOI] [PMC free article] [PubMed] [Google Scholar]
57.Sladojevic, N., Yu, B. & Liao, J. K. Regulator of g-protein signaling 5 maintains brain endothelial cell function in focal cerebral ischemia. J. Am. Heart Assoc.9, e017533 (2020). [DOI] [PMC free article] [PubMed] [Google Scholar]
58.Crapser, J. D., Arreola, M. A., Tsourmas, K. I. & Green, K. N. Microglia as hackers of the matrix: sculpting synapses and the extracellular space. Cell. Mol. Immunol.18, 2472–2488 (2021). [DOI] [PMC free article] [PubMed] [Google Scholar]
59.Wu, J. et al. Genome-wide screen and validation of microglia pro-inflammatory mediators in stroke. Aging Dis.12, 786–800 (2021). [DOI] [PMC free article] [PubMed] [Google Scholar]
60.Wang, X. et al. Three-dimensional intact-tissue sequencing of single-cell transcriptional states. Science361, eaat5691 (2018). [DOI] [PMC free article] [PubMed] [Google Scholar]
61.Isnard, P. & Humphreys, B. D. Spatial transcriptomics: integrating morphology and molecular mechanisms of kidney diseases. Am. J. Pathol.195, 23–39 (2025). [DOI] [PMC free article] [PubMed] [Google Scholar]
62.Dries, R. et al. Giotto: a toolbox for integrative analysis and visualization of spatial expression data. Genome Biol.22, 78 (2021). [DOI] [PMC free article] [PubMed] [Google Scholar]
63.Cardona, H. J., Somasundaram, A., Crabtree, D. M., Gadd, S. L. & Becher, O. J. Prenatal overexpression of platelet-derived growth factor receptor A results in central nervous system hypomyelination. Brain Behav.11, e2332 (2021). [DOI] [PMC free article] [PubMed] [Google Scholar]
64.Cheng, M. Y., Leslie, F. M. & Zhou, Q. Y. Expression of prokineticins and their receptors in the adult mouse brain. J. Comp. Neurol.498, 796–809 (2006). [DOI] [PMC free article] [PubMed] [Google Scholar]
65.Millman, D. J. et al. VIP interneurons in mouse primary visual cortex selectively enhance responses to weak but specific stimuli. eLife9, e55130 (2020). [DOI] [PMC free article] [PubMed] [Google Scholar]
66.Zhang, M. et al. Spatially resolved cell atlas of the mouse primary motor cortex by MERFISH. Nature598, 137–143 10.1038/s41586-021-03705-x (2021). [DOI] [PMC free article] [PubMed] [Google Scholar]
67.Wu, S. Z. et al. A single-cell and spatially resolved atlas of human breast cancers. Nat. Genet.53, 1334–1347 (2021). [DOI] [PMC free article] [PubMed] [Google Scholar]
68.Madissoon, E. et al. A spatially resolved atlas of the human lung characterizes a gland-associated immune niche. Nat. Genet.55, 66–77 (2023). [DOI] [PMC free article] [PubMed] [Google Scholar]
69.Jones, R. C. et al. The Tabula Sapiens: a multiple-organ, single-cell transcriptomic atlas of humans. Science376, eabl4896 (2022). [DOI] [PMC free article] [PubMed] [Google Scholar]
70.Daigle, T. L. et al. A suite of transgenic driver and reporter mouse lines with enhanced brain-cell-type targeting and functionality. Cell174, 465–480 (2018). [DOI] [PMC free article] [PubMed] [Google Scholar]
71.Cao, J. et al. The single-cell transcriptional landscape of mammalian organogenesis. Nature566, 496–502 (2019). [DOI] [PMC free article] [PubMed] [Google Scholar]
72.Hao, Y. et al. Dictionary learning for integrative, multimodal and scalable single-cell analysis. Nat. Biotechnol.42, 293–304 (2024). [DOI] [PMC free article] [PubMed] [Google Scholar]
73.Qiu, X. et al. Spatiotemporal modeling of molecular holograms. Cell187, 7351–7373 (2024). [DOI] [PubMed] [Google Scholar]
74.Mitchel, J., Gao, T., Petukhov, V., Cole, E. & Kharchenko, P. V. Impact and correction of segmentation errors in spatial transcriptomics. Nat. Genet.58, 434–444 (2025). [DOI] [PubMed] [Google Scholar]
75.CosMx SMI Human Pancreas FFPE Dataset. NanoString Technologies/Bruker Spatial Biologyhttps://brukerspatialbiology.com/products/cosmx-spatial-molecular-imager/ffpe-dataset/cosmx-smi-human-pancreas-ffpe-dataset/ (2024).
76.Yu, J. et al. HISSTA: a human in situ single-cell transcriptome atlas. Bioinformatics41, btaf142 10.1093/bioinformatics/btaf142 (2025). [DOI] [PMC free article] [PubMed] [Google Scholar]
77.MERFISH Mouse Brain Receptor Map. Vizgen Data Release V1.0 (May 2021). Vizgenhttps://info.vizgen.com/mouse-brain-map (2021).

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Supplementary Information^{(37.6MB, pdf)}

Supplementary Figs. 1–17 and Tables 1–6.

Reporting Summary^{(2MB, pdf)}

Peer Review File^{(2.4MB, pdf)}

Source Data Figs. 1–6^{(102.2KB, xlsx)}

Statistical source data for Figs. 1–6.

Data Availability Statement

All code used for data preprocessing, segmentation, evaluation and figure generation is publicly available at https://github.com/TranslationalBioinformaticsUnit/Benchmarking-SegmentationMethods-ST/.

[CR1] 1.Ståhl, P. L. et al. Visualization and analysis of gene expression in tissue sections by spatial transcriptomics. Science353, 78–82 (2016). [DOI] [PubMed] [Google Scholar]

[CR2] 2.Tian, L., Chen, F. & Macosko, E. Z. The expanding vistas of spatial transcriptomics. Nat. Biotechnol.41, 773–782 (2023). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR3] 3.Marx, V. Method of the year: spatially resolved transcriptomics. Nat. Methods18, 9–14 (2021). [DOI] [PubMed] [Google Scholar]

[CR4] 4.Arora, R. et al. Spatial transcriptomics reveals distinct and conserved tumor core and edge architectures that predict survival and targeted therapy response. Nat. Commun.14, 5029 (2023). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR5] 5.Huang, S. et al. Spatial transcriptomics: a new frontier in cancer research. Clin. Cancer Bull.3, 13 10.1007/s44272-024-00018-8 (2024). [DOI]

[CR6] 6.Cheng, M. et al. Spatially resolved transcriptomics: a comprehensive review of their technological advances, applications, and challenges. J. Genet. Genom.50, 625–640 (2023). [DOI] [PubMed] [Google Scholar]

[CR7] 7.Li, Y., Stanojevic, S. & Garmire, L. X. Emerging artificial intelligence applications in spatial transcriptomics analysis. Comput. Struct. Biotechnol. J.20, 2895–2908 (2022). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR8] 8.Wang, L., Maletic-Savatic, M. & Liu, Z. Region-specific denoising identifies spatial co-expression patterns and intra-tissue heterogeneity in spatially resolved transcriptomics data. Nat. Commun.13, 6912 (2022). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR9] 9.Cui, Y. et al. DenoiseST: a dual-channel unsupervised deep learning-based denoising method to identify spatial domains and functionally variable genes in spatial transcriptomics. Preprint at bioRxiv 10.1101/2024.03.04.583438 (2024). [DOI]

[CR10] 10.Bergenstråhle, L. et al. Super-resolved spatial transcriptomics by deep data fusion. Nat. Biotechnol.40, 476–479 (2022). [DOI] [PubMed] [Google Scholar]

[CR11] 11.Zeng, Y. et al. Imputing spatial transcriptomics through gene network constructed from protein language model. Commun. Biol.7, 1271 (2024). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR12] 12.Sun, E. D., Ma, R. & Zou, J. SPRITE: improving spatial gene expression imputation with gene and cell networks. Bioinformatics40, i521–i528 (2024). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR13] 13.Pham, D. et al. Robust mapping of spatiotemporal trajectories and cell–cell interactions in healthy and diseased tissues. Nat. Commun.14, 7739 (2023). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR14] 14.Xu, H. et al. Unsupervised spatially embedded deep representation of spatial transcriptomics. Genome Med.16, 12 (2024). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR15] 15.Liu, Z., Wu, D., Zhai, W. & Ma, L. SONAR enables cell type deconvolution with spatially weighted Poisson-Gamma model for spatial transcriptomics. Nat. Commun.14, 4727 (2023). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR16] 16.Lu, Y., Chen, Q. M. & An, L. SPADE: spatial deconvolution for domain specific cell-type estimation. Commun. Biol.7, 469 (2024). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR17] 17.Coleman, K., Hu, J., Schroeder, A., Lee, E. B. & Li, M. SpaDecon: cell-type deconvolution in spatial transcriptomics with semi-supervised learning. Commun. Biol.6, 378 (2023). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR18] 18.Hao, M., Hua, K. & Zhang, X. SOMDE: a scalable method for identifying spatially variable genes with self-organizing map. Bioinformatics37, 4392–4398 (2021). [DOI] [PubMed] [Google Scholar]

[CR19] 19.Zhang, K., Feng, W. & Wang, P. Identification of spatially variable genes with graph cuts. Nat. Commun.13, 5488 (2022). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR20] 20.Kueckelhaus, J. et al. Inferring histology-associated gene expression gradients in spatial transcriptomic studies. Nat. Commun.15, 7280 (2024). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR21] 21.Liang, Q., Solis Soto, L., Haymaker, C. & Chen, K. LSGI: interpretable spatial gradient analysis for spatial transcriptomics data. Genome Biol.26, 238 (2025). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR22] 22.Blampey, Q. et al. Sopa: a technology-invariant pipeline for analyses of image-based spatial omics. Nat. Commun.15, 4981 (2024). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR23] 23.Chen, H., Li, D. & Bar-Joseph, Z. SCS: cell segmentation for high-resolution spatial transcriptomics. Nat. Methods20, 1237–1243 (2023). [DOI] [PubMed] [Google Scholar]

[CR24] 24.Palla, G. et al. Squidpy: a scalable framework for spatial omics analysis. Nat. Methods19, 171–178 (2022). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR25] 25.Hu, Y. et al. Benchmarking clustering, alignment, and integration methods for spatial transcriptomics. Genome Biol.25, 212 (2024). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR26] 26.Yan, L. & Sun, X. Benchmarking and integration of methods for deconvoluting spatial transcriptomic data. Bioinformatics39, btac805 (2023). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR27] 27.Yuan, Z. et al. Benchmarking spatial clustering methods with spatially resolved transcriptomics data. Nat. Methods21, 712–722 (2024). [DOI] [PubMed] [Google Scholar]

[CR28] 28.Wang, Y. et al. A systematic evaluation of computational methods for cell segmentation. Brief. Bioinform.25, bbae407 (2024). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR29] 29.Choe, K., Pak, U., Pang, Y., Hao, W. & Yang, X. Advances and challenges in spatial transcriptomics for developmental biology. Biomolecules13, 156 (2023). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR30] 30.Atta, L. & Fan, J. Computational challenges and opportunities in spatially resolved transcriptomic data analysis. Nat. Commun.12, 5283 (2021). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR31] 31.Kesidis, A. L., Krassanakis, V., Misthos, L. M. & Merlemis, N. patchIT: a multipurpose patch creation tool for image processing applications. Multimodal Technol. Interact.6, 111 10.3390/mti6120111 (2022). [DOI]

[CR32] 32.Nejat, P. et al. Creating an atlas of normal tissue for pruning WSI patching through anomaly detection. Sci. Rep.14, 3932 (2024). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR33] 33.Chen, H. et al. Recovering single-cell expression profiles from spatial transcriptomics with scResolve. Cell Rep. Methods4, 100864 (2024). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR34] 34.Fang, S. et al. Computational approaches and challenges in spatial transcriptomics. Genomics Proteomics Bioinformatics21, 24–47 (2023). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR35] 35.Fu, X. et al. BIDCell: biologically-informed self-supervised learning for segmentation of subcellular spatial transcriptomics data. Nat. Commun.15, 509 (2024). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR36] 36.Ravi, N. et al. SAM 2: Segment Anything in images and videos. In International Conference on Learning Representations (ICLR) Proceedings (ICLR, 2025).

[CR37] 37.Kirillov, A. et al. Segment Anything. In Proc. IEEE/CVF International Conference on Computer Vision (ICCV) 4015–4026 (IEEE Computer Society, 2023).

[CR38] 38.Archit, A. et al. Segment Anything for Microscopy. Nat. Methods22, 579–591 (2025). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR39] 39.Petukhov, V. et al. Cell segmentation in imaging-based spatial transcriptomics. Nat. Biotechnol.40, 345–354 (2022). [DOI] [PubMed] [Google Scholar]

[CR40] 40.Liu, Z. et al. Swin Transformer: hierarchical vision transformer using shifted windows. In Proc. IEEE International Conference on Computer Vision 9992–10002 10.1109/ICCV48922.2021.00986 (Institute of Electrical and Electronics Engineers, 2021). [DOI]

[CR41] 41.Wang, J. et al. Seesaw Loss for long-tailed instance segmentation. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 9695–9704 (IEEE Computer Society, 2021).

[CR42] 42.Stringer, C. & Pachitariu, M. Cellpose3: one-click image restoration for improved cellular segmentation. Nat. Methods22, 592–599 (2025). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR43] 43.Pang, M., Roy, T. K., Wu, X. & Tan, K. CelloType: a unified model for segmentation and classification of tissue images. Nat. Methods22, 348–357 (2024). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR44] 44.Janesick, A. et al. High resolution mapping of the tumor microenvironment using integrated single-cell, spatial and in situ analysis. Nat. Commun.14, 8353 10.1038/s41467-023-43458-x (2023). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR45] 45.He, S. et al. High-plex imaging of RNA and proteins at subcellular resolution in fixed tissue by spatial molecular imaging. Nat. Biotechnol.40, 1794–1806 10.1038/s41587-022-01483-z (2022). [DOI] [PubMed] [Google Scholar]

[CR46] 46.Chen, A. et al. Spatiotemporal transcriptomic atlas of mouse organogenesis using DNA nanoball-patterned arrays. Cell185, 1777–1792.e21 10.1016/j.cell.2022.04.003 (2022). [DOI] [PubMed] [Google Scholar]

[CR47] 47.Androvic, P. et al. Spatial transcriptomics-correlated electron microscopy maps transcriptional and ultrastructural responses to brain injury. Nat. Commun.14, 4115 (2023). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR48] 48.Zeng, H. et al. Integrative in situ mapping of single-cell transcriptional states and tissue histopathology in a mouse model of Alzheimer’s disease. Nat. Neurosci.26, 430–446 10.1038/s41593-022-01251-x (2023). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR49] 49.Di Tommaso, P. et al. Nextflow enables reproducible computational workflows. Nat. Biotechnol.35, 316–319 (2017). [DOI] [PubMed] [Google Scholar]

[CR50] 50.Lee, W.-H., Wang, G.-M., Seaman, L. B. & Vannucci, S. J. Coordinate IGF-I and IGFBP5 gene expression in perinatal rat brain after hypoxia–ischemia. J. Cereb. Blood Flow Metab.16, 227–236 (1996). [DOI] [PubMed] [Google Scholar]

[CR51] 51.Miya, K., Keino-Masu, K., Okada, T., Kobayashi, K. & Masu, M. Expression of heparan sulfate endosulfatases in the adult mouse brain: co-expression of Sulf1 and dopamine D1/D2 receptors. Front. Neuroanat.15, 726718 (2021). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR52] 52.Ohayon, D. et al. Sulfatase 2 promotes generation of a spinal cord astrocyte subtype that stands out through the expression of Olig2. Glia67, 1478–1495 (2019). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR53] 53.Chen, Z., Gao, M., Su, Y., Liu, P. & Sun, B. Running promotes transformation of brain astrocytes into neuroprotective reactive astrocytes and synaptic formation by targeting Gpc6 through the STAT3 pathway. Front. Physiol.12, 633618 (2021). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR54] 54.Kim, H. Y., Choi, J. S., Cha, J. H., Choi, J. Y. & Lee, M. Y. Expression of vascular endothelial growth factor receptors Flt-1 and Flk-1 in embryonic rat forebrain. Neurosci. Lett.425, 131–135 (2007). [DOI] [PubMed] [Google Scholar]

[CR55] 55.Nitta, T. et al. Size-selective loosening of the blood-brain barrier in claudin-5-deficient mice. J. Cell Biol.161, 653–660 (2003). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR56] 56.Jia, C. et al. Vitronectin from brain pericytes promotes adult forebrain neurogenesis by stimulating CNTF. Exp. Neurol.312, 20–32 (2019). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR57] 57.Sladojevic, N., Yu, B. & Liao, J. K. Regulator of g-protein signaling 5 maintains brain endothelial cell function in focal cerebral ischemia. J. Am. Heart Assoc.9, e017533 (2020). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR58] 58.Crapser, J. D., Arreola, M. A., Tsourmas, K. I. & Green, K. N. Microglia as hackers of the matrix: sculpting synapses and the extracellular space. Cell. Mol. Immunol.18, 2472–2488 (2021). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR59] 59.Wu, J. et al. Genome-wide screen and validation of microglia pro-inflammatory mediators in stroke. Aging Dis.12, 786–800 (2021). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR60] 60.Wang, X. et al. Three-dimensional intact-tissue sequencing of single-cell transcriptional states. Science361, eaat5691 (2018). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR61] 61.Isnard, P. & Humphreys, B. D. Spatial transcriptomics: integrating morphology and molecular mechanisms of kidney diseases. Am. J. Pathol.195, 23–39 (2025). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR62] 62.Dries, R. et al. Giotto: a toolbox for integrative analysis and visualization of spatial expression data. Genome Biol.22, 78 (2021). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR63] 63.Cardona, H. J., Somasundaram, A., Crabtree, D. M., Gadd, S. L. & Becher, O. J. Prenatal overexpression of platelet-derived growth factor receptor A results in central nervous system hypomyelination. Brain Behav.11, e2332 (2021). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR64] 64.Cheng, M. Y., Leslie, F. M. & Zhou, Q. Y. Expression of prokineticins and their receptors in the adult mouse brain. J. Comp. Neurol.498, 796–809 (2006). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR65] 65.Millman, D. J. et al. VIP interneurons in mouse primary visual cortex selectively enhance responses to weak but specific stimuli. eLife9, e55130 (2020). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR66] 66.Zhang, M. et al. Spatially resolved cell atlas of the mouse primary motor cortex by MERFISH. Nature598, 137–143 10.1038/s41586-021-03705-x (2021). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR67] 67.Wu, S. Z. et al. A single-cell and spatially resolved atlas of human breast cancers. Nat. Genet.53, 1334–1347 (2021). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR68] 68.Madissoon, E. et al. A spatially resolved atlas of the human lung characterizes a gland-associated immune niche. Nat. Genet.55, 66–77 (2023). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR69] 69.Jones, R. C. et al. The Tabula Sapiens: a multiple-organ, single-cell transcriptomic atlas of humans. Science376, eabl4896 (2022). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR70] 70.Daigle, T. L. et al. A suite of transgenic driver and reporter mouse lines with enhanced brain-cell-type targeting and functionality. Cell174, 465–480 (2018). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR71] 71.Cao, J. et al. The single-cell transcriptional landscape of mammalian organogenesis. Nature566, 496–502 (2019). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR72] 72.Hao, Y. et al. Dictionary learning for integrative, multimodal and scalable single-cell analysis. Nat. Biotechnol.42, 293–304 (2024). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR73] 73.Qiu, X. et al. Spatiotemporal modeling of molecular holograms. Cell187, 7351–7373 (2024). [DOI] [PubMed] [Google Scholar]

[CR74] 74.Mitchel, J., Gao, T., Petukhov, V., Cole, E. & Kharchenko, P. V. Impact and correction of segmentation errors in spatial transcriptomics. Nat. Genet.58, 434–444 (2025). [DOI] [PubMed] [Google Scholar]

[CR75] 75.CosMx SMI Human Pancreas FFPE Dataset. NanoString Technologies/Bruker Spatial Biologyhttps://brukerspatialbiology.com/products/cosmx-spatial-molecular-imager/ffpe-dataset/cosmx-smi-human-pancreas-ffpe-dataset/ (2024).

[CR76] 76.Yu, J. et al. HISSTA: a human in situ single-cell transcriptome atlas. Bioinformatics41, btaf142 10.1093/bioinformatics/btaf142 (2025). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR77] 77.MERFISH Mouse Brain Receptor Map. Vizgen Data Release V1.0 (May 2021). Vizgenhttps://info.vizgen.com/mouse-brain-map (2021).

PERMALINK

Benchmarking segmentation methods for spatial transcriptomics

Xabier Martínez-de-Morentin

Sumeer A Khan

Alberto Maillo

Robert Lehmann

Narsis A Kiani

Vincenzo Lagani

David Gomez-Cabrero

Jesper Tegner

Abstract

Main

Fig. 1. Overview of the benchmark workflow and evaluation design.

Results

Benchmarking methodology—platforms, datasets and evaluation metrics

Fig. 2. Overview of ranked methods by an overall score across seven datasets.

The performance of segmentation methods varies with dataset and metric

Segmentation performance metrics converge in biologically meaningful macro categories

Fig. 3. Metric composition.

Metrics categories show trade-offs in the choice of segmentation methods

Fig. 4. Interdependence of evaluation metrics across datasets.

Spatial transcriptomics technology impacts segmentation outcomes

Fig. 5. Biological consequences of segmentation choice across technologies.

Nextflow: a baseline unified pipeline for spatial segmentation benchmarking

Practical guidance for segmentation method and platform selection

Fig. 6. Practical decision tree for selecting a segmentation tool.

Discussion

Methods

Datasets and experimental platforms

Xenium dataset

CosMx datasets

Stereo-seq datasets

STARmap PLUS dataset

MERFISH dataset

Reference datasets for single-cell spatial data cell-type annotation

A uniform preprocessing pipeline for spatial transcriptomics data

HVG selection

Unified coordinates space

Homogeneous image color scale

Spatial transcriptomics cell-type annotation

Equal patching strategy

Segmentation methods and parameter settings

Watershed segmentation

CellPose: universal geometric model for cell segmentation

SCS

Baysor segmentation

BIDCell

SAM

SAM2

mSAM

CelloType: unified model for segmentation and classification

Swin Transformer: hierarchical vision transformer using shifted windows

Cascade Mask R-CNN (Seesaw)

Exclusion criteria

Evaluation metrics

Baseline metrics: overall characteristics

Cell-level QC metrics

Cell morphology metrics

Gene-level QC metrics

Segmented cell expression purity

Spatial characteristics

Neighboring contamination

Downstream biological evaluation metrics

Ranking of the models

Unified pipeline for spatial segmentation benchmarking

Segmentation outcomes and benchmarking reproducibility

Spatial transcriptomics technology evaluation

Computer platform

Statistics and reproducibility

Reporting summary

Online content

Supplementary information

Source data

Acknowledgements

Author contributions

Peer review

Peer review information

Data availability

Code availability

Competing interests