From spots to cells: Cell segmentation in spatial transcriptomics with BOMS

Abstract

Imaging-based Spatial Transcriptomics methods enable the study of gene expression and regulation in complex tissues at subcellular resolution. However, inaccurate cell segmentation procedures lead to misassignment of mRNAs to individual cells which can introduce errors in downstream analysis. Current methods estimate cell boundaries using auxiliary DAPI/Poly(A) stains. These stains can be difficult to segment, thus requiring manual tuning of the method, and not all mRNA molecules may be assigned to the correct cells. We describe a new method, based on mean shift, that segments the cells based on the spatial locations and the gene labels of the mRNA spots without requiring any auxiliary images. We evaluate the performance of BOMS across various publicly available datasets and demonstrate that it achieves comparable results to the best existing method while being simple to implement and significantly faster in execution. Open-source code is available at https://github.com/sciai-lab/boms.

Introduction

The development of Spatial Transcriptomics (ST) technologies in recent years has led to a huge increase in the acquisition of spatial data and its subsequent use in the study of tissue composition and function. ST methods capture genes with their spatial context and can be used for determining the cell-type composition of tissues, for exploring the spatial sources of gene expression variation, and for the analysis of cell-cell interactions and communication between various cell types. These applications are contingent on the availability of a segmentation mask to group mRNA molecules into cells and assign a transcription profile to each cell. However, this computational task remains a challenge.

The standard cell segmentation methods rely on a nucleus or membrane staining to identify cell instances and boundaries respectively. Although deep learning based methods like Cellpose perform well on the task of nuclei segmentation, the nucleus does not capture the full extent of the cell body, resulting in a lot of mRNA molecules remaining unassigned. The membrane would be more indicative of the cell boundary, but segmenting it in densely packed cells with overlaps remains an error-prone task. The Segmentation-free method SSAM [1] directly produces a cell-type map of the tissue without producing a cell-by-gene matrix, but cannot be used for downstream applications like neighborhood analysis. Some assisted cell segmentation tools like pciSeq [2], JSTA [3], SCS [4] have also been developed which utilize both the mRNA and staining data to perform segmentation. These methods generally rely on the DAPI staining to obtain an initial segmentation and then use the mRNA molecules to propagate the cell boundary. Good performance is dependent on access to a nucleus segmentation of adequate quality, and the methods might struggle in cases where not every cell has a nucleus visible in a 2D section. There have also been deep-learning methods that utilize transcript data directly, such as Bering [5], UCS [6], GeneSegNet [7], and BIDCell [8]. These methods often still require supervision, at least in the form of initial cell labels, which are typically derived from nuclei segmentation. Adapting these models to new datasets requires computational resources, and when using pre-trained models, fine-tuning is often necessary to account for batch effects and dataset heterogeneity. In the case of BIDCell, additional scRNA-seq data and prior biological knowledge, in the form of positive and negative marker genes, are required. Petukhov et al. have proposed Baysor [9] that uses Bayesian Mixture Modeling to segment the cells either completely without an auxiliary image or with the inclusion of one with a user-defined confidence level. Although it has an elegant mathematical foundation, it is difficult to diagnose the source of error if the method does not work well out of the box and the long runtimes on large datasets make it challenging to search for optimal parameters. ClusterMap [10] is an unsupervised framework based on density peak clustering to segment cells, but the incorporation of spatial distance and gene information into a single distance metric can result in cells that are physically disconnected.

To tackle these issues, we have developed BOMS—a method that performs cell segmentation in imaging-based spatial transcriptomics datasets without the requirement of an auxiliary image. BOMS is based on the classical Mean shift algorithm [11] and uses the modes of the underlying distribution in the multidimensional domain (space and gene expression) to cluster small neighborhoods together to obtain cells. BOMS is easy to understand, has few tunable parameters, and is fast to execute. It can also utilize an auxiliary image when available to further improve accuracy. We demonstrate that BOMS can be applied to segment cells in a variety of Spatial Transcriptomics datasets and compares favorably with the current state-of-the-art methods.

Materials and methods

The BOMS algorithm

BOMS is based on the assumption that a cell body is homogeneous: molecules belonging to the same cell form small local neighborhoods that are transcriptionally similar to each other. Such similar molecular neighborhoods that are in close proximity to each other will probably belong to the same cell instance.

BOMS takes as input the gene identities and their spatial locations (Fig 1). In the first step, it takes the k nearest neighbors of each molecule as a measure of the local transcriptional landscape. Each molecule is thus represented by a Neighbourhood Gene Expression (NGE) vector containing the gene counts in the immediate vicinity. We term the space in which the spatial coordinates lie as the ‘Spatial domain’ and the NGE vectors comprise the ‘Range domain’.

Fig 1. Workflow of the BOMS algorithm.

A: BOMS takes as input the gene labels and the spot locations. B: Among the k spatial Nearest Neighbors, the number of occurrences of each gene is calculated to form the Neighborhood Gene Expression (NGE) vectors. These NGE vectors can be visualized in the color space by taking PCA projection of them in three dimensions. The spatial locations together with the NGE vectors form separate clusters for individual cell instances in the joint spatial-NGE space. BOMS takes advantage of this structure and tries to find the modes in this joint domain by iteratively moving towards the maxima of the underlying (estimated) density function. C: Sample trajectories of the Meanshift procedure are shown along with the final mode locations. D: Cell segmentation labels are estimated by grouping together all the molecules that were mapped to the same mode. E: Final cell outlines are shown with the NGE vectors. F: Movement of a spot after incorporating Cellpose flows with different confidence levels, $\alpha$. The meanshift direction is marked in red and the direction of Cellpose flow is marked in blue. The final location of the spot is a convex combination of the two vectors, with $\alpha =1$ coinciding with the mean shift vector, and $\alpha =0$ coinciding with the update as per Cellpose flow.

In the second step, the NGE vectors are used to find the modes in the joint spatial-range domain. A multivariate kernel window is defined around each molecule such that all the spots that are close to it in the spatial as well as the range domain are inside the window. This proximity is defined by the Euclidean distance in the spatial domain and the cosine distance in the range domain. The width of the kernel window is regulated by two tunable parameters—spatial bandwidth h_s and range bandwidth h_r. During each iteration of the BOMS algorithm, the multivariate kernel is iteratively shifted to the centroid of the points contained inside of it. For the centroid calculation, each point is assigned a weight that decreases with increasing distance from the center of the kernel. The kernel thus moves in the direction of maximum increase in the joint density gradient and defines a path leading to the modes of the estimated joint density. After convergence, the individual cell instances are delineated by grouping together all those molecules that converged to the same mode.

BOMS can also utilize available DAPI Stainings to improve its results by using the flows obtained by applying the Cellpose model [12] to the image to adjust the direction in which the kernel moves according to a user-defined confidence level.

The naive implementation of the algorithm is costly as in every iteration it needs to find the neighbors for all points. For computational efficiency, the method has been implemented in C++ using multidimensional kd-trees.

Implementation details.

The output of the FISH-based experiments will consist of N spots with their spatial locations and gene labels. Let ${\mathbf{x}}_{\mathbf{n}}^{\mathbf{s}}$ denote the 2/3-dimensional spatial coordinates and g_n denote the gene-label for the ${\mathit{\text{n}}}^{\mathit{\text{th}}}$ observation. Given the total number of genes G in the dataset, we convert the gene labels g_n to G-dimensional 1-hot vectors ${\mathbf{x}}_{\mathbf{n}}^{\mathbf{r}}$ where the element i = g_n is 1 and rest are 0. We refer to ${\mathbf{x}}_{\mathbf{n}}^{\mathbf{r}}$ as the range vector.

For each molecule, the gene labels of the spatial k-Nearest Neighbours are taken to form the Neighbourhood Gene Expressions (NGE) vectors

$${\mathbf{y}}_{\mathbf{n}}^{\mathbf{r}}={\displaystyle \sum _{j\in N_k(n)}}{\mathbf{x}}_{\mathbf{j}}^{\mathbf{r}}$$ (1)

where N_k(n) are the indices of the k spatial nearest points to spot-n in the dataset.

If the number of genes G in the dataset is greater than 50, then we use PCA to reduce the dimensions of the NGE vectors from G to 50 for speed-up.

We store all the information about the spot-n in the vector

$${\mathbf{z}}_{\mathbf{n}\mathbf{,}\mathbf{0}}=\left[\begin{array}{c}{\mathbf{x}}_{\mathbf{n}}^{\mathbf{s}}\\ {\mathbf{y}}_{\mathbf{n}}^{\mathbf{r}}\end{array}\right].$$ (2)

Each BOMS iteration consists of the update

$${\mathbf{z}}_{\mathbf{i}\mathbf{,}\mathbf{t}\mathbf{+}\mathbf{1}}=\left[\begin{array}{c}{\mathbf{z}}_{\mathbf{i}\mathbf{,}\mathbf{t}\mathbf{+}\mathbf{1}}^{\mathbf{s}}\\ {\mathbf{z}}_{\mathbf{i}\mathbf{,}\mathbf{t}\mathbf{+}\mathbf{1}}^{\mathbf{r}}\end{array}\right]=\frac{{\sum }_{j=1}^N{\mathbf{z}}_{\mathbf{j}\mathbf{,}\mathbf{t}}f\left(d_{ij}^s;h_s\right)g\left(d_{ij}^r;h_r\right)}{{\sum }_{j=1}^{N}f\left(d_{ij}^s;h_s\right)g\left(d_{ij}^r;h_r\right)},$$ (3)

where

$$d_{ij}^s=\Vert {\mathbf{z}}_{\mathbf{i}\mathbf{,}\mathbf{t}}^{\mathbf{s}}-{\mathbf{z}}_{\mathbf{j}\mathbf{,}\mathbf{t}}^{\mathbf{s}}\Vert$$ (4)

$$d_{ij}^r=1-\frac{{\mathbf{z}}_{\mathbf{i}\mathbf{,}\mathbf{t}}^{\mathbf{r}}·{\mathbf{z}}_{\mathbf{j}\mathbf{,}\mathbf{t}}^{\mathbf{r}}}{\Vert {\mathbf{z}}_{\mathbf{i}\mathbf{,}\mathbf{t}}^{\mathbf{r}}\Vert \Vert {\mathbf{z}}_{\mathbf{j}\mathbf{,}\mathbf{t}}^{\mathbf{r}}\Vert }$$ (5)

with $\Vert \Vert$ representing the L2-norm and $·$ representing the dot product.

Various kernels can be used for the spatial and range domain, but we found that the Epanechnikov kernel for the spatial domain and disk kernel for the range domain works well in practice on multiple datasets:

$$f(x;h)=\{\begin{array}{cc}1-\frac{x^2}{h^2}\hfill & \text{if }|x|\le h\hfill \\ 0\hfill & \text{otherwise}\hfill \end{array}$$ (6)

$$g(x;h)=\{\begin{array}{cc}1\hfill & \text{if }|x|\le h\hfill \\ 0\hfill & \text{otherwise}\hfill \end{array}$$ (7)

After the last iteration of BOMS has completed, all the modes that are closer than h_s/4 in the spatial domain are grouped together using single-linkage clustering. Thus all the points that converged to the same spatial mode ${\mathbf{z}}_{\mathbf{n}}^{\mathbf{s}}$ are assigned the same cell instance label. Cell instances with very few number of molecules, less than a threshold th_bg, are assigned to the background.

When the flows fl(p) from Cellpose model are available, then the direction in which the point moves in each iteration is adjusted as follows—If flow at ${\mathbf{z}}_{\mathbf{i}\mathbf{,}\mathbf{t}}^{\mathbf{s}}$ is greater than $ϵ$, $fl({\mathbf{z}}_{\mathbf{i}\mathbf{,}\mathbf{t}}^{\mathbf{s}})>ϵ$, then

$${\mathbf{z}}_{\mathbf{i}\mathbf{,}\mathbf{t}\mathbf{+}\mathbf{1}}^{\mathbf{s}}={\mathbf{z}}_{\mathbf{i}\mathbf{,}\mathbf{t}}^{\mathbf{s}}+\alpha ({\mathbf{z}}_{\mathbf{i}\mathbf{,}\mathbf{t}\mathbf{+}\mathbf{1}}^{\mathbf{s}}-{\mathbf{z}}_{\mathbf{i}\mathbf{,}\mathbf{t}}^{\mathbf{s}})+(1-\alpha )fl({\mathbf{z}}_{\mathbf{i}\mathbf{,}\mathbf{t}}^{\mathbf{s}})$$

where $\alpha \in [0,1]$ signifies the weight given to BOMS. In case the flow is small, the update is done according to Eq 3.

Choice of parameters and troubleshooting.

BOMS has mainly three tunable parameters—the number of neighbours K to calculate the NGE vectors, the spatial bandwidth parameter h_s and the range bandwidth parameter h_r. K in [30,50] has worked well in our experiments. In general, choosing the value of K much smaller than the expected number of molecules per cell should work. h_s should be chosen to be around the radius of the cells. For the range bandwidth, a value of h_r in [0.2,0.5] works well in practice. Choosing too low a value for h_s or h_r can result in over-segmentation (fragmented cells), whereas choosing too high a value of h_s and h_r can result in under-segmentation (multiple cells merged together). If tuning h_s and h_r does not solve the problem, then K should be increased to solve oversegmentation and decreased to solve for undersegmentation.

Code availability.

The source code for BOMS is available at https://github.com/sciai-lab/boms. The current version works on both Linux and Windows OS.

Data visualisation

Visualisation of gene expression.

In order to analyse the gene expression patterns in the spatial transcriptomic data visually, we first form the NGE vectors. The number of neighbours $K_{vis}$ chosen for visualization are greater than the corresponding segmentation parameters as we want to see more global patterns. This gives us $K_{vis}$ dimensional gene expression vectors for the N spots. We use PCA to reduce the dimensions to 3 yielding $N\times 3$ dimensional matrix C. In order to increase the contrast, these reduced dimensions are clipped below at –1 and above at 1.5. We then perform min-max scaling to scale the values in C between 0 and 1. These 3-dimensional vectors are then interpreted as RGB colors, with each spot having its individually mapped color.

Cell boundary visualization.

We use Python’s library SciPy to draw convex hulls around the molecules belonging to the same cell. As some molecules can sometimes lie a bit further from the main molecule cloud, we can end up with big oddly shaped hulls. In order to achieve a nicer visualisation we do some filtering beforehand as follows. For all cells

• For all the molecules belonging to a single cell we calculate the spatial density for the ${\mathit{\text{i}}}^{\mathit{\text{th}}}$ molecule $\mathbf{d}\mathbf{e}{\mathbf{n}}_{\mathbf{i}}$ with the epanechnikov kernel which has a bandwidth equal to h_s

  $$\mathbf{d}\mathbf{e}{\mathbf{n}}_{\mathbf{i}}={\displaystyle \sum _{j:label(j)=label(i)}}(1-\frac{{d_{ij}^s}^2}{h_s^2})𝐈({d_{ij}^s}^2\le h_s^2)$$ (8)

  where label(i) is the cell instance label for ${\mathit{\text{i}}}^{\mathit{\text{th}}}$ molecule and I is the indicator function.
• We calculate the mean density of molecules $\overline{{\mathbf{d}\mathbf{e}\mathbf{n}}_i}$ belonging to the same cell.

  $$\overline{{\mathbf{d}\mathbf{e}\mathbf{n}}_i}=\frac{{\displaystyle {\displaystyle \sum _{j:label(j)=label(i)}}{\mathbf{d}\mathbf{e}\mathbf{n}}_j}}{{\displaystyle {\displaystyle \sum _{j:label(j)=label(i)}}1}}$$ (9)

• If ${\mathbf{d}\mathbf{e}\mathbf{n}}_i<\frac{\overline{{\mathbf{d}\mathbf{e}\mathbf{n}}_i}}{t}$, then we filter out the ${\mathit{\text{i}}}^{\mathit{\text{th}}}$ molecule.

We use t = 1.5 for all datasets except osmFISH for which we use t = 2. Note that this filtering is only for visualizing the polygons. In the figures where we have colored different cells with different colors, these filtered molecules will be plotted with the color of the cell that they belong to, even though they might lie outside the polygon boundary.

Datasets and preprocessing

We use four publicly available datasets to evaluate BOMS against existing methods.

1. Allen smFISH VISp Dataset—The data has 22 genes and 1074778 spots. The data is available at https://github.com/spacetx-spacejam/data. There is also a JSON file containing the results of DAPI segmentation, which we call the Silver Standard. The raw image data for DAPI is available via the Amazon S3 bucket https://s3.amazonaws.com/starfish.data.spacetx/smFISH/mouse/formatted_with_DAPI/experiment.json. These images were processed using the Starfish package https://spacetx-starfish.readthedocs.io/en/latest/ and manually stitched together. The resulting image was downsampled for easy handling. The ${17}^{\mathit{\text{th}}}$ slice was used in all the visualizations. To bring the spot data to the same scale for processing with BOMS, the x and y values were translated by 3000 and 2500 respectively, and both were then multiplied by 2.5.

2. MERFISH Dataset [13]—The data has 135 genes and 3728169 spots. The readings corresponding to the blank controls were removed. The cell boundaries detected by the authors of the original publication are also available which serve as our Silver Standard. The raw DAPI/Poly(A) image for this dataset is not available. This data is available at https://zenodo.org/records/3478502.

3. osmFISH Dataset [14]—The data has 33 genes and 1802589 spots. The spots corresponding to genes considered as low-quality by [14] were removed from the data. The segmentation mask of Poly(A) signal, which serves as our Silver Standard, is also available. The low-resolution versions of DAPI and Poly(A) images are available at https://zenodo.org/records/3478502. The x values for the spots are translated by –25 and y values are translated by 10 and subsequently converted to ‘pixel’ units using the ‘Cell area in number of pixels’ parameter provided by the Linnarsson Lab to register the spots with the DAPI and Poly(A) images. The data is available under https://linnarssonlab.org/osmFISH/availability/.

4. STARmap Dataset [15]—The data has 1020 genes and 949505 spots. The dataset has a lot of noisy spots in the background region. They were filtered using the steps described in the previous section using a bandwidth value of 60 and a t value of 85. The data is available at http://starmapresources.org/data. The DAPI images are available at https://github.com/wanglab-broad/ClusterMap/tree/main/datasets/STARmap_V1_1020

5. Xenium Dataset—This dataset includes 40,204,978 spots and 480 genes, obtained using the 10x Genomics Xenium In Situ platform for FFPE human lung cancer profiling with the human immuno-oncology panel and a custom add-on. The spatially resolved gene expression data is accompanied by imaging data, including DAPI. The DAPI image was resampled to match the spatial resolution of the spots. Based on details from the dataset’s documentation, the Silver Standard segmentation appears to be derived by expanding nuclear boundaries, as no membrane-specific stains were provided. This dataset provides a high-resolution view of gene expression across the tissue. The dataset is available at https://www.10xgenomics.com/datasets/ffpe-human-lung-cancer-data-with-human-immuno-oncology-profiling-panel-and-custom-add-on-1-standard.

We used the Cellpose algorithm [12] with default parameters on the DAPI/Poly(A) images to segment them.

Segmentation parameters

The segmentation parameters used for the various datasets are summarized in Table 1.

Table 1. Segmentation parameters for the different datasets.

Dataset	$\mathit{K}$	hs	${\mathit{h}}_{\mathit{r}}$	${\mathit{t}\mathit{h}}_{\mathit{b}\mathit{g}}$	${\mathit{K}}_{\mathit{v}\mathit{i}\mathit{s}}$
Allen smFISH	30	17.5	0.4	30	80
MERFISH	30	7	0.5	30	80
osmFISH	30	6.5	0.2	30	50
STARmap	300	80	0.5	30	300

Segmentation parameters for Xenium dataset.

For the Xenium dataset, we used K = 50, h_s = 5, h_r = 0.4 and $\alpha =0.6$. Due to the large size of the dataset, BOMS was run in 18 spatially defined patches, ensuring efficient processing without loss of data continuity.

Performance metrics

Mutual information.

We use the normalized mutual information for comparing the segmentations A and B obtained from two different methods respectively. The normalization is done using the joint entropy of A and B.

Correlation metric [9].

The Correlation metric was proposed by Petukhov et al. [9]. This metric takes two Segmentations — A and B and performs the following steps :

1. Take A as the source segmentation and B as the Target segmentation.

2. All the cells with the number of molecules below a threshold b₁ are taken out of consideration.

3. For each source cell s_i, we find all the overlapping target cells t_j.

4. Among the overlapping target cells, select the one with the largest number molecules in the overlapping region.

5. Calculate the overlapping fraction f_i as the number of molecules in the overlapping region divided by the number of molecules in the source cell.

6. Only consider those pairs for which $0.3\le f_i\le 0.7$. This is because if the $f_i\le 0.3$, then there are not enough molecules in the overlapping region to compare with the rest of the source cell. Similarly, if $f_i\ge 0.7$, there are not enough molecules in the non-overlapping part of source cell to compare with the overlapping part.

7. Eliminate the pairs for which the number of molecules in overlapping part or the number of molecules in the non-overlapping part is less than b₂.

8. Form gene expression vectors for the molecules in overlapping and non-overlapping regions.

9. Calculate the correlation value between the two gene expression vectors. If the two parts indeed belong to the same molecular celltype, then this correlation value will be high.

10. Take B as the source segmentation and A as the Target segmentation and repeat steps 2–9. The segmentation containing more homogenous transcriptional signatures per cell will have higher scores.

To compare BOMS with other methods, we use $b_1=b_2=30$ for all datasets except osmFISH for which we use $b_1=b_2=15$. For demonstrating the failure cases of this metric we have used $b_1=30,b_2=0$.

Runtime performance.

All the experiments were performed on a Dell XPS laptop with Intel(R) Core(TM) i7 Processor and 32 GB of RAM.

Compared methods

We compared the performance of BOMS with Baysor [9], pciSeq [2] and the original published Segmentations, which we term ‘Silver Standard’

Baysor.

The datasets were segmented using Baysor (v0.6.2) Command Line Interface with the parameter values taken from the supplementary table provided by Petukhov et al. [9]. To run Baysor with prior, we used the segmentation mask obtained from Cellpose.

pciSeq.

We used the python package pciSeq (v0.0.59). As inputs, we used the spot matrix and the segmentation masks from Cellpose. As the MERFISH dataset didn’t have any published stains, it was excluded from the comparison of pciSeq with BOMS.

Downstream analysis

The clustering and DEG (Differentially Expressed Genes) analysis were performed using the ScanPy package (version 1.10.3). Cells segmented by each method were filtered to exclude those with fewer than 10 or more than 900 transcripts. The dataset was normalized to a target sum of 10⁶, log-transformed, and the top-200 highly variable genes (HVGs) were selected. After this step, default parameter settings of ScanPy were used for all subsequent analyses. Dimensionality reduction was performed using PCA, followed by construction of a k-nearest neighbor (kNN) graph and Leiden clustering. The clustering resolution was fixed across datasets and segmentation methods to ensure comparability.

Differential expression analysis was conducted using the Wilcoxon rank-sum test. To evaluate the similarity of DEGs across methods, the top-5 DEGs for each cluster were identified for the Silver Standard and the other segmentation methods. The Jaccard similarity of the top-5 DEGs was then computed between clusters in the Silver Standard and the corresponding clusters in BOMS, Baysor, and pciSeq.

The ScanPy function ‘ingest’ with PCA embeddings was employed to transfer cell type labels from the Silver Standard to cells segmented using BOMS and other methods. The joint UMAP embeddings were generated using the widely used umap-learn Python package (https://github.com/lmcinnes/umap), with parameters set to min_dist = 0.1, spread = 2, and n_epochs = 1000.

Results and discussion

Fig 2 illustrates the results of applying the BOMS method on the various publicly available datasets. The results for Allen smFISH data (Fig 2A) indicate that methods like BOMS and Baysor, which work independently of DAPI data, can potentially detect some cells that would be challenging to identify solely from the DAPI staining due to illumination artifacts arising during data acquisition (these cells are highlighted with red circles in the Silver Standard panel to indicate missed detections, and green circles in the BOMS and Baysor panels to indicate successful detections). The figure also underscores the variability in success between these methods, revealing instances where one method might succeed while the other fails (e.g., locations highlighted with red circles in BOMS and green in Baysor, and vice versa). Additionally, both BOMS and Baysor can fail to distinguish between adjacent cells evident in the DAPI staining, leading to undersegmentation where two distinct cells may be merged into a single instance. This is particularly likely when the adjacent cells share a similar molecular profile. Examples of such instances are highlighted with red circles in Fig 2A, including cases where both methods fail (green circles in Silver Standard, red in both BOMS and Baysor) or where one method performs better (e.g., green circles in Silver Standard and BOMS, red in Baysor, and vice versa).

Fig 2. Examples of BOMS results on the published datasets.

A: BOMS segmentation results on the Allen smFISH dataset. All molecules are colored by taking a PCA projection of the NGE vectors. Cell boundaries are shown by black contours. The right column shows a zoomed-in version of BOMS, Baysor, and the original published segmentation overlaid on the DAPI image. Green colored circles indicate that the method has correctly detected cell boundaries whereas red colored circles indicate incorrect segmentation. B: BOMS result on the STARmap dataset [15] overlaid on the DAPI image, C: BOMS result on the osmFISH dataset [14] overlaid on the poly(A) image and D: BOMS result on the MERFISH [13] dataset.

Fig 3 demonstrates the effect of incorporating auxiliary information in the form of Cellpose flows into the BOMS segmentation process for the Allen smFISH and osmFISH datasets. The results indicate that incorporating Cellpose flows as auxiliary information allows for better alignment with the boundaries identified by Cellpose, particularly in cases with substantial intracellular transcriptional variation. As the influence of Cellpose flows increases, the boundaries of BOMS-segmented cells become more consistent with the Cellpose results, highlighting the utility of auxiliary stains in improving segmentation accuracy.

Fig 3. Effect of Cellpose flows on BOMS segmentation.

The figure demonstrates the impact of incorporating Cellpose flows into BOMS segmentation for the Allen smFISH (A) and osmFISH (B) datasets. For the Allen smFISH dataset, Cellpose was applied to the DAPI image, while for the osmFISH dataset, it was applied to the poly(A) image. Cyan boundaries represent Cellpose segmentations, while black outlines depict the resulting BOMS segmentations. All molecules are colored based on a PCA projection of their NGE vectors, where spots with similar colors have a similar molecular neighborhood. Transcriptional variation within cells is reflected by differences in the coloring of spots. As the influence of Cellpose flows increases, BOMS segmentation aligns more closely with Cellpose boundaries. Lower $\alpha$ values are particularly necessary in cases with substantial intracellular transcriptional variation to ensure faithful alignment with Cellpose results.

The difficulty of establishing a groundtruth in spatial transcriptomic imaging data makes evaluating the performance of different methods challenging. The most common auxiliary stain acquired in the spatial experiments are the DAPI nuclei images, which are then segmented to get cell boundaries. However, this approach leaves a lot of transcripts outside the boundary. These so-called ‘dangling’ transcripts are difficult to assign to an individual cell [16]. Moreover, multinucleate cells would be difficult to identify, and cells without a nucleus would not be detected at all. Hence, even a perfectly segmented DAPI cannot serve as a groundtruth. Cell membrane staining would be better for segmentation, but membrane markers often generate low signals, lack specificity for the cytoplasmic membrane, or are not applicable to all cell types [17]. Consequently, we resort to a number of imperfect evaluation strategies which in tandem can provide a means to compare different methods for cell segmentation.

We evaluated BOMS against a set of related methods [9] [2] for cell segmentation and spot assignment on a collection of published datasets [13] [14] [15]. A summary of the results is presented in Table 2, with visual comparisons provided in Fig 4. BOMS identifies a higher number of molecules as part of a cell compared to the other methods (Baysor and pciSeq) and with reference to the original published segmentations (‘Silver Standard’) (Fig 4E). The number of cells detected by BOMS is similar to that of Baysor and pciSeq (Fig 4D). MERFISH and osmFISH datasets show the largest change compared to the Silver Standard, since a lot of transcripts were unassigned in them.

Table 2. Summary of benchmarking results across datasets and methods. The table compares BOMS with other segmentation methods, including Baysor, pciSeq, and the Silver Standard, based on key metrics: runtime, mutual information, correlation score, number of detected cells, and fraction of molecules assigned. ‘S’ and ‘T’ refer to Source and Target segmentations, respectively. See Fig 4 for visual representations of these comparisons.

Method	STARmap	Allen smFISH	osmFISH	MERFISH
Median correlation score
BOMS (S) $\to$ SS (T)	0.75	0.9	0.88	0.75
SS (S) $\to$ BOMS (T)	0.67	0.86	0.78	0.39
BOMS (S) $\to$ Baysor (T)	0.45	0.77	0.45	0.45
Baysor (S) $\to$ BOMS (T)	0.9	0.92	0.95	0.73
BOMS (S) $\to$ pciSeq (T)	0.5	0.89	0.9	—
pciSeq (S) $\to$ BOMS (T)	0.75	0.85	0.68	—
Normalized MI
BOMS	0.7017	0.5976	0.4104	0.4577
BOMS (w/ DAPI)	0.7146	0.6065	0.3738	—
Baysor	0.3554	0.6084	0.4276	0.5098
Baysor (w/ DAPI)	0.4029	0.6101	0.4302	—
pciSeq (w/ DAPI)	0.4318	0.6189	0.4399	—
Number of cells
BOMS	1033	5360	14055	17009
BOMS (w/ DAPI)	1009	5320	16139	—
Baysor	867	5652	14398	9085
Baysor (w/ DAPI)	785	4007	13822	—
pciSeq (w/ DAPI)	789	4831	4940	—
Silver Standard	975	3450	5834	6406
Fraction of molecules assigned
BOMS	0.6196	0.9423	0.9360	0.9862
BOMS (w/ DAPI)	0.6197	0.9470	0.8884	—
Baysor	0.2796	0.9188	0.7487	0.5063
Baysor (w/ DAPI)	0.3145	0.7897	0.7513	—
pciSeq (w/ DAPI)	0.3275	0.8098	0.7160	—
Silver Standard	0.6326	0.6184	0.4449	0.4977
Execution time (min)
BOMS	1.42	0.58	1.17	4.42
Baysor	7.50	11.00	38.00	30.00

Fig 4. Comparison of BOMS with related methods.

A: The runtime performance of BOMS vs. Baysor—BOMS produces results of similar quality to Baysor while being 10 times faster. B: Mutual Information Scores with respect to the Silver Standard (original published segmentations). The scores are similar to those of Baysor and pciSeq, except on the STARmap dataset. C: Schematic showing the calculation of correlation score for comparing a source and target Segmentation. For each cell in the source segmentation, the target cell with the maximum overlap is computed. Correlation score between the molecules in the overlapping region and the remaining molecules in the source cell is then estimated. If the source segmentation is correct, the corresponding correlation scores should be high. D: The number of detected cells reported by different methods, showing BOMS is able to recover more cells than the other methods. E: The fraction of molecules assigned to cells by different methods, showing least number of unassigned transcripts by BOMS. F: a. Correlation score for BOMS vs. Baysor, b. BOMS vs. pciSeq, c. BOMS vs. Silver Standard showing a higher performance of BOMS with respect to pciSeq and the original published segmentation.

Next, we use the Correlation metric proposed by Petukhov et al. [9] to evaluate BOMS against other methods. The Correlation metric compares the performance of one method relative to another without requiring a groundtruth. Like the underlying methods, the metric assumes a homogeneous cell body. The method that better explains the areas of mismatch between two candidates gets a higher score. The correlation metric is computed briefly as follows—for each cell in a ‘Source’ Segmentation, we find all the overlapping cells in the ‘Target’ Segmentation (Fig 4C). The target cell with the largest such overlap is then selected, and we compute the correlation between the Gene Expression of the overlapping region and the Gene Expression of the remaining region of the source cell. If the source segmentation is reasonably correct, then this correlation value will be high as two partitions of the same cell should be transcriptionally similar as per our assumption. If on the other hand the correlation value is low, then it implies that the target segmentation did a better job by assigning different instance labels to these two molecular regions. The same calculation is done after switching the source and target. The method demonstrating superior performance will have higher correlation values when it is the source.

BOMS gets a higher average Correlation score when compared with pciSeq (except on STARmap dataset) and the original published segmentations (Fig 4F). Baysor shows a higher correlation score compared to BOMS across all protocols. However, it is essential to interpret these results cautiously. The correlation metric tends to reward under-segmentation across the same celltype—if the source segmentation merges two cells of similar transcriptional signature but the target does not, then the corresponding correlation score for the source will be high. The metric also penalizes the target when the source is over-segmented—if the source cells split a single cell and the two parts have slightly different gene expression profiles, then the target gets a low correlation score. Baysor gets a higher score because of these reasons. It shows a tendency towards over-segmenting single cells, potentially capturing some subcellular localization of mRNA molecules, leading to higher correlation scores for Baysor even when results from BOMS method better adhere to the DAPI image.

This shortcoming of the correlation metric is demonstrated in Fig 5. When BOMS is run with the parameters that we consider optimal for the Allen smFISH data, BOMS gets a lower correlation score compared to Baysor (segmentation shown in Fig 5B). When we increase the spatial bandwidth parameter in BOMS so that it will merge cells leading to undersegmentation, the correlation metric indicates that BOMS is performing better than Baysor despite visual inspection contradicting this (Fig 5C). Similarly, when we decrease the range bandwidth parameter making BOMS over-segment individual cells, the correlation score would lead one to conclude that these results are qualitatively similar to Baysor when, in fact, they are much worse (Fig 5F). Interestingly, when under-segmentation is achieved with the help of range bandwidth parameter (Fig 5D) instead of the spatial bandwidth parameter or if over-segmentation is done with the spatial bandwidth parameter (Fig 5E), then the correlation metric does the intuitively right thing by assigning low scores to BOMS.

Fig 5. Breakdown of the correlation metric proposed by [9].

The figure illustrates the behavior of the Correlation metric when comparing Baysor with BOMS at varying settings, inducing under-segmentation or over-segmentation, on the Allen smFISH dataset. The Silver Standard segmentation is depicted with red contours in A–F. A: Baysor segmentation. B: BOMS with optimal parameters (h_s = 17.5, h_r = 0.4). C: BOMS with a high spatial bandwidth to induce under-segmentation (h_s = 30, h_r = 0.4). D: BOMS with a high range bandwidth to induce under-segmentation (h_s = 17.5, h_r = 0.9). E: BOMS with low spatial bandwidth to cause over-segmentation (h_s = 10, h_r = 0.4). F: BOMS with low range bandwidth to cause over-segmentation (h_s = 17.5, h_r = 0.08). G: Correlation scores depicting higher values implying good performance when BOMS under-segments because of high spatial bandwidth or when BOMS over-segments because of low range bandwidth in contrast to bad visual results. H: Normalized Mutual information values with respect to the Silver Standard make it evident that the results are actually worse despite good correlation scores.

We also analyze the Mutual information of the different methods with respect to the Silver Standard (published segmentations) (Fig 4B). Except for the STARmap dataset, the performance of BOMS is similar to Baysor and pciSeq. We also observe an increase in performance when the Cellpose flows are taken into consideration to improve the segmentation results.

Lastly, we compare the computation time of the different methods (Fig 4A). BOMS outperforms Baysor significantly, demonstrating 5–10x increase in speed while also being more memory efficient.

Comparison of BOMS and cellpose.

Fig 6 demonstrates the results obtained from BOMS and Cellpose for adjacent cells in the Allen smFISH and osmFISH datasets. For the Allen smFISH dataset (Fig 6A), Cellpose was applied to the DAPI image, while BOMS relied solely on transcriptomic information. BOMS not only matches the results of Cellpose but also identifies larger cell boundaries by incorporating transcripts outside the nuclei. This ability is particularly advantageous when only DAPI, the most commonly available stain, is used, as it allows BOMS to assign spots outside the nuclear region, which Cellpose cannot assign with DAPI alone. For the osmFISH dataset (Fig 6B), where Cellpose was applied to the poly(A) image, BOMS provides results comparable to Cellpose without auxiliary image information.

Fig 6. Comparison of cells obtained from BOMS and Cellpose.

The figure shows results from Cellpose and BOMS on adjacent cells in the (A) Allen smFISH dataset, where Cellpose was applied to the DAPI image, and (B) osmFISH dataset, where Cellpose was applied to the poly(A) image. For the Allen smFISH dataset, BOMS identifies larger cell boundaries compared to Cellpose as it also includes transcripts outside the nuclei, which Cellpose cannot detect when only DAPI staining is available. For osmFISH, the results from BOMS align closely with Cellpose boundaries.

Fig 7 highlights instances from the osmFISH dataset where Cellpose, applied to the poly(A) image, failed to detect cells due to imaging artifacts or lack of signal. In contrast, BOMS successfully identified these cells based on transcriptomic data. This demonstrates the robustness of BOMS in cases where auxiliary images fail, providing an important advantage in cell segmentation.

Fig 7. Detection of cells missed by Cellpose but identified by BOMS.

This figure presents examples from the osmFISH dataset where Cellpose, applied to the poly(A) image, failed to detect cells, while BOMS successfully segmented them using transcriptomic data. For each instance, the DAPI image, poly(A) image, and transcriptomic data are shown with Cellpose boundaries (cyan) and BOMS boundaries (black) overlaid. These examples highlight BOMS’s ability to segment cells in cases where auxiliary images are insufficient due to imaging artifacts or lack of signal.

Table 3 summarizes the comparisons between BOMS and other segmentation methods, including Baysor, pciSeq, and Auxiliary Stain Segmentation. This table provides a qualitative overview, highlighting the unique strengths and limitations of each method to complement the quantitative comparisons presented earlier.

Table 3. Comparison of the advantages and disadvantages of different cell segmentation methods. The table summarizes the strengths and limitations of BOMS and other methods (Baysor, pciSeq, and Auxiliary Stain Segmentation), providing qualitative insights into their performance characteristics.

Method	Advantages	Disadvantages
BOMS	• Fast runtime. • Simple to understand with only three tunable parameters. • Supports auxiliary stains for improved performance.	• Slightly lower correlation scores on some datasets. • Can produce suboptimal segmentation in regions with homogeneous transcription or subcellular variation without auxiliary stains.
Baysor	• High correlation scores. • Can incorporate compartment-specific gene information to handle intracellular variation. • Supports auxiliary stains for improved performance.	• Slower runtime. • Can lead to over-segmentation. • Complex methodology, making it difficult to tune if results are unsatisfactory. • Can produce suboptimal segmentation in regions with homogeneous transcription or subcellular variation without auxiliary stains.
pciSeq	• Simple implementation with good runtime.	• Requires a cell mask, making it reliant on auxiliary stains. Cannot segment based on transcriptional data alone. • Lacks a mechanism to assign confidence levels to segmentation quality of the cell mask.
Auxiliary Stain Segmentation (No Gene Data)	• Effective for regions with homogenous transcription or dense gene presence.	• Limited by tissue type and imaging quality. • DAPI-based segmentation results in loss of signal from molecules outside boundaries. • Misses cells not present in the imaging plane.

Application of BOMS to high-resolution Xenium dataset.

The Xenium dataset poses challenges due to high transcript density, with many spots located outside nuclear boundaries. Visual inspection (Fig 8) shows differences in transcriptional signatures within and beyond nuclei, reflected in the coloring of NGE vectors.

Fig 8. Application of BOMS to the Xenium dataset.

A: Segmented cells using BOMS. B: Segmented cells using the Silver Standard. In (A) and (B), the spots are colored by taking a PCA projection of the NGE vectors. C: Cellpose boundaries overlaid on the DAPI image, where Cellpose was applied to define nuclear boundaries. D: Correlation score of BOMS with respect to the Silver Standard.

The Silver Standard segmentation relies heavily on nuclear boundaries, which may not accurately capture cell morphology, especially in regions beyond the nuclei. While the normalized mutual information (NMI) between BOMS and the Silver Standard is 0.786, transcriptional assignments outside nuclei differ significantly, leading to visual discrepancies. BOMS achieves higher average correlation score, suggesting better capture of transcriptomic relationships (Fig 8D). However, this metric has limitations, and it remains unclear which method more accurately reflects the biology.

Incorporating auxiliary information from Cellpose flows was crucial for BOMS performance, highlighting the importance of combining transcriptomic and morphological data in challenging datasets like Xenium.

Impact of segmentation on clustering and differential expression analysis.

The downstream effects of segmentation on clustering and DEG (Differentially Expressed Gene) analysis were evaluated using the Allen smFISH and MERFISH datasets (Figs 9 and 10). For the Allen smFISH dataset, the analysis revealed that the number of clusters identified was consistent across the Silver Standard, BOMS, Baysor, and pciSeq segmentations, as shown in Fig 9 (A–D). The number of clusters for each segmentation method is indicated in the title of the respective UMAP plot. However, the overlap of the top-5 DEGs for clusters from the Silver Standard and those from other methods was limited, as illustrated by the Jaccard similarity heatmaps in Fig 9 (E–G).

Fig 9. Results of clustering and DEG analysis on the Allen smFISH dataset.

A–D: UMAP embeddings of cells segmented by (A) Silver Standard, (B) BOMS, (C) Baysor, and (D) pciSeq, with clusters represented by distinct colors. The number of clusters is indicated in the title of each subplot. E–G: Heatmaps showing the Jaccard similarity of the top-5 DEGs identified for clusters in the Silver Standard and clusters from (E) BOMS, (F) Baysor, and (G) pciSeq.

Fig 10. Results of clustering and DEG analysis on the MERFISH dataset.

A–D: UMAP embeddings of cells segmented by (A) Silver Standard, (B) BOMS, (C) Baysor, and (D) pciSeq, with clusters represented by distinct colors. The number of clusters is indicated in the title of each subplot. E–G: Heatmaps showing the Jaccard similarity of the top-5 DEGs identified for clusters in the Silver Standard and clusters from (E) BOMS, (F) Baysor, and (G) pciSeq.

For the MERFISH dataset, segmentation methods yielded different numbers of clusters (Fig 10 (A–D)). Baysor identified the highest number of clusters and achieved the greatest separation in UMAP, while BOMS and pciSeq produced UMAP embeddings that appeared more similar to one another. Interestingly, preprocessing by removing molecules identified as background by Baysor before applying BOMS increased the number of clusters identified by BOMS.

These results highlight the sensitivity of downstream analyses to both the segmentation algorithm and intermediate preprocessing steps. Each choice or substep in the upstream segmentation workflow—such as background removal—can significantly affect downstream clustering and DEG analysis. Accurate and consistent segmentation is therefore critical to ensure reliable downstream analyses, as differences in segmentation propagate through to key biological insights.

Cell type label transfer from silver standard.

To further assess the impact of BOMS on biologically meaningful downstream analysis, we performed a cell type label transfer from the Silver Standard to cells segmented using BOMS. This analysis was conducted on the osmFISH and MERFISH datasets, where the largest discrepancies between methods were reported by Petukhov et al. [9].

Both BOMS and Baysor recover a greater number of cells across various cell types compared to the Silver Standard (Fig 11A). Notably, BOMS identifies additional astrocytes that were missed by the Silver Standard due to the absence of detectable poly(A) signal (Fig 11C). This highlights the ability of BOMS to capture cell types that may otherwise remain unannotated when relying on auxiliary staining alone.

Fig 11. Cell type label transfer for the osmFISH dataset.

A: Bar plot showing the number of cells assigned to each cell type by the Silver Standard, BOMS, Baysor, and pciSeq. B: Example of an astrocyte cell identified both in the Silver Standard and by BOMS. C: Example of an astrocyte detected by BOMS but not present in the Silver Standard due to lack of poly(A) signal. The grayscale background shows poly(A) staining. D-E Joint UMAP embedding of cells from BOMS and the Silver Standard colored by (D) cell type and (E) dataset, illustrating overlap and unique contributions.

Results for the MERFISH dataset are provided in the S1 Appendix. These findings demonstrate that segmentation approaches leveraging transcriptomic data, such as BOMS, can enhance the completeness of cell-type maps, offering valuable biological insights.

Conclusion

Accurate cell segmentation can increase the number of detected cells and decrease the number of unassigned transcripts in in-situ transcriptomics data. It can help in identifying correct celltype signatures, complete cell-type maps and missing rare cell types. We describe a methodology to perform the essential preprocessing step of cell segmentation in in-situ transcriptomics data. BOMS is based on the classical Meanshift method and is very simple to interpret. It contains only three tunable parameters that have an intuitive effect on the output—the number of nearest neighbors K to form NGE vectors, the spatial bandwidth parameter h_s and the range bandwidth h_r. We have included a guideline to choose these parameters in the Methods section. BOMS exhibits a fast runtime, enabling researchers to test different parameters for their specific research goals. We showed that BOMS is applicable to a variety of spatial datasets including MERFISH, osmFISH, STARmap and shows a good performance on them.

There are some cell segmentation cases that can be challenging for BOMS. The first step in BOMS is to compute the NGE vectors by taking the k nearest neighbors for each spot. This can smooth the signal at the cell boundary excessively making the subsequent steps unable to resolve the distinct cells accurately. Performing segmentation in transcriptionally homogeneous areas in dense tissues is also difficult, which might be improved by the inclusion of DAPI/Poly(A) information. BOMS can make use of Cellpose flows to improve its segmentation in such a case. Segmenting cells in datasets with a high spatial resolution showing subcellular molecular localization also remains an outstanding challenge as the underlying assumption of the method that a cell body is homogeneous is invalid.

A critical challenge for the field is the lack of a reasonable metric to judge the different segmentation methods. DAPI/Poly(A) cannot be considered as the groundtruth due to their limitations. The Correlation metric proposed by [9] does not paint an accurate picture and can give high scores to a method with worse performance. Therefore there is a need for the development of a principled metric. This can encourage the discovery of new scientific methods and the adoption of already existing methods in automated pipelines for the analysis of spatial transcriptomic data.

Overall, the key strengths of the method are its good segmentation accuracy, conceptual simplicity and concomitant interpretability as well as computational efficiency. BOMS is available as a plugin in python and the code is available at https://github.com/sciai-lab/boms.

Supporting information

S1 Appendix. Additional text and figures for the manuscript.

(PDF)

Data Availability

All relevant data are within the manuscript and its Supporting Information files.

Funding Statement

The work was supported by “Informatics for Life” funded by the Klaus Tschira Foundation, but no grant numbers were issued.