Bayesian Hidden Mark Interaction Model for Detecting Spatially Variable Genes in Imaging-Based Spatially Resolved Transcriptomics Data

Abstract

Recent technology breakthroughs in spatially resolved transcriptomics (SRT) have enabled the comprehensive molecular characterization of cells whilst preserving their spatial and gene expression contexts. One of the fundamental questions in analyzing SRT data is the identification of spatially variable genes whose expressions display spatially correlated patterns. Existing approaches are built upon either the Gaussian process-based model, which relies on ad hoc kernels, or the energy-based Ising model, which requires gene expression to be measured on a lattice grid. To overcome these potential limitations, we developed a generalized energy-based framework to model gene expression measured from imaging-based SRT platforms, accommodating the irregular spatial distribution of measured cells. Our Bayesian model applies a zero-inflated negative binomial mixture model to dichotomize the raw count data, reducing noise. Additionally, we incorporate a geostatistical mark interaction model with a generalized energy function, where the interaction parameter is used to identify the spatial pattern. Auxiliary variable MCMC algorithms were employed to sample from the posterior distribution with an intractable normalizing constant. We demonstrated the strength of our method on both simulated and real data. Our simulation study showed that our method captured various spatial patterns with high accuracy; moreover, analysis of a seqFISH dataset and a STARmap dataset established that our proposed method is able to identify genes with novel and strong spatial patterns.

1. INTRODUCTION

Recent advancements in spatially resolved transcriptomics (SRT) technology have fundamentally transformed our capacity to study cellular behavior at a molecular level, while preserving their spatial and gene expression contexts. This technological leap has opened new avenues for exploring complex biological systems at unprecedented levels of detail and accuracy. Efremova et al. (2020) and Liao et al. (2021) found that the positional context of gene expression is important to understanding tissue functionality and pathology changes, which highlights the pivotal role of SRT techniques. Broadly, SRT technologies are categorized into sequencing-based and imaging-based methods based on differences in RNA profiling: sequencing-based and imaging-based. Spatial transcriptomics, one of the next-generation sequencing (NGS) technologies, resolves gene expression profiles at a resolution of 100 μm. Spatial transcriptomics implemented by the 10x Visium platform achieved 55μm resolution, allowing for a detailed study of spatial organization. On the other hand, imaging-based technologies have revolutionized the field of transcriptomics by achieving single-cell resolution, with prominent examples such as sequential fluorescence in situ hybridization seqFISH (Ståhl et al., 2016), seqFISH+ (Eng et al., 2019), and multiplexed error-robust FISH (MERFISH) (Moffitt et al., 2018). Datasets profiled by SRT technologies have inspired the exploration of the spatial organization of gene expression within tissues. Cohorts with single-cell resolution motivate more biological analysis, such as cell-cell communication analysis via CellChat (Jin et al., 2021), characterization of ligand-receptor interactions between different cell types (Efremova et al., 2020) and so on. Hence, spatial information provided by imaing-based SRT data makes it more feasible to identify and quantify gene expression in specific regions of a tissue.

One of the most interesting questions arising along the development of SRT techniques is the identification of spatially variable genes (SVGs) whose expressions display spatially correlated patterns. Studies have found that SVGs demarcate clear spatial substructure, and are relevant to disease progression (Svensson et al., 2018; Hu et al., 2021). Various methods across different fields have been developed to identify SVGs, each capitalizing on distinct strengths. Trendsceek (Edsgärd et al., 2018) is built upon marked point processes to rank and evaluate the spatial pattern of each gene; however, it yields unsatisfactory performance (Sun et al., 2020) and is inhibited from scaling to large-scale data due to the expensive computational cost (Sun et al., 2020; Dries et al., 2021). SpatialDE (Edsgärd et al., 2018), SPARK (Sun et al., 2020), and BOOST-GP (Li et al., 2021) capture spatial correlation patterns by utilizing the Gaussian process. Specifically, SpatialDE models normalized gene expression levels via a multivariate Gaussian model with a spatial covariance function characterizing linear and periodic spatial patterns. SPARK models raw counts using a generalized linear spatial model with different periodic and Gaussian kernels. BOOST-GP models raw counts with a Bayesian zero-inflated negative binomial (ZINB) model with a squared exponential kernel covariance matrix. However, the performance of these kernel-based methods relies heavily on the resemblance between the underlying spatial expression patterns and the predefined kernel functions (Jiang et al., 2022). BinSpect (Dries et al., 2021), a non-model based method, identifies SVGs through statistical enrichment analysis of spatial network neighbors with binarized gene expression states. SpaGCN (Hu et al., 2021) defines SVGs as genes as those exhibiting differential expression among spatial domains and employs a deep learning model to identify these domains. BOOST-MI (Jiang et al., 2022) utilizes an energy-based modified Ising model to identify SVGs exclusively for sequencing-based SRT data, with the limitation that the spatial position of measured spots needs to be on the regular lattice grid. Compared to kernel-based models, energy-based interaction characterization enables the detection of broader types of gene spatial expression patterns.

As mentioned, gene expressions resolved by imaging-based SRT have single-cell resolution, which potentially unearths more biological insights. We aimed to develop a model that can identify SVGs with higher accuracy to be used on data from imaging-based SRT platforms, and uncover more biological mechanisms. Drawing inspiration from the success of energy-based models over kernel-based approaches (Jiang et al., 2022), we propose a novel joint Bayesian framework model, BOOST-HMI. This model utilizes a recently proposed energy function for mark interaction (Li et al., 2019). In particular, we adopt a ZINB mixture model to handle the unique data characteristics of SRT, including excess zeros and unknown mean-variance structures. Additionally, our method introduces a latent binary gene expression indicator to distinguish high and low expression states at the cellular level, thereby enhancing the model’s robustness against noise. Unlike BOOST-MI, our proposed BOOST-HMI is not constrained by the spatial distribution requirements of sequencing-based SRT data, making it versatile for imaging-based datasets where cells are randomly distributed. Furthermore, BOOST-HMI directly models raw counts within a joint Bayesian framework, addressing uncertainties associated with dichotomization. Our comprehensive simulation studies, covering various scenarios, demonstrate the superior accuracy of BOOST-HMI in detecting SVGs. We also applied our model to two real datasets: a mouse hippocampus seqFISH dataset and a mouse visual cortex STARmap dataset, where it successfully detected more spatial patterns and layer-specific SVGs, potentially unveiling novel biological insights.

The rest of the paper is organized as follows: section 2 introduces our ZINB mixture model for identifying SVGs from SRT count data and discusses the extension of the Bayesian mark interaction model to SRT data. In section 3, we describe the Markov chain Monte Carlo (MCMC) algorithms for posterior sampling and the resulting posterior inference. Finally, section 4 presents our method’s performance on simulated and real SRT datasets, compared with five other methodologies.

2. METHODS

In this section, we introduce a ZINB mixture model for directly modeling the imaging-based SRT count data, and a hidden mark interaction model to quantify the spatial dependency of latent binary gene expression levels. The schematic diagram of BOOST-HMI is shown in Figure 1.

Figure 1.

The schematic diagram of the proposed BOOST-HMI model.

Before introducing the models, we summarize the SRT data notations as follows. We denote the gene expression raw counts as a $n$-by-$p$ matrix $\mathit{Y}$ with each entry $y_{ij}\in \mathbb{N}$ denoting the number of read counts for gene $j$ at cell $i$. Every column ${\mathit{y}}_{\cdot j}$ in $\mathit{Y}$ denotes the expression counts across all measured cells for gene $j$, while each row ${\mathit{y}}_i$. denotes the counts of all genes on cell $i$ where $i=1,\cdots ,n$, $j=1,\cdots ,p$. As to geospatial profile, let a $n$-by- 2 matrix $T$ be the matrix for the spatial location of cells, where each row ${\mathit{t}}_{i\cdot }=\left(t_{i1},t_{i2}\right)\in {\mathbb{R}}^2$ records the coordinates of cell $i$ in the 2D Cartesian plane.

2.1. A ZINB mixture model for modeling gene expression count data

For the majority of SRT techniques, gene expression measurements obtained are in the form of counts. In the context of for imaging-based SRT platforms, gene expressions are collected as the count of barcoded mRNA corresponding to a particular transcript within a single cell (Zhao et al., 2022). Due to the characteristics of these measurements, observed count data often suffers from over-dispersion and zero-inflation. The negative binomial distribution can effectively account for the mean-variance relationship in the raw counts. Moveover, the gene expression count matrix $\mathit{Y}$ is characterized by an inflated number of zeros, resulting from imaging sensitivity and hybridization efficiency (Zhao et al., 2022); therefore, we generalized the negative binomial (NB) model to the ZINB model to account for both the over-dispersion and the high sparsity level, i.e.,

$$y_{ij}\mid {\pi }_i,{\nu }_{ij},{\varphi }_j\sim {\pi }_{i}I\left(y_{ij}=0\right)+\left(1-{\pi }_i\right)\mathrm{N}\mathrm{B}\left(s_i{\nu }_{ij},{\varphi }_j\right),\text{or},y_{ij}\mid {\pi }_i,{\nu }_{ij},{\varphi }_j\sim \mathrm{ZINB}\left({\pi }_i,s_i{\nu }_{ij},{\varphi }_j\right),$$ (1)

where parameter ${\pi }_i\in [0,1]$ represents the false zero proportion measured on cell $i.\mathrm{N}\mathrm{B}(\nu ,\varphi )$ denotes a negative binomial distribution with mean $\nu$ and dispersion parameter $\varphi$. Consequently, the variance is $\nu +{\nu }^2/\varphi .1/\varphi$ controls the overdispersion scaled by the square of mean ${\nu }^2$. The probability mass function is $\frac{\mathrm{\Gamma }(y+\varphi )}{y!\mathrm{\Gamma }\left(\varphi \right)}{\left(\frac{\varphi }{\nu +\varphi }\right)}^{\varphi }{\left(\frac{\nu }{\nu +\varphi }\right)}^y$. Given our particular circumstances, the NB mean is decomposed into two multiplicative effects, the size factor $s_i$ and the expression level ${\nu }_{ij}$. The collection of $\mathit{s}={\left(s_1,\cdots ,s_n\right)}^{\mathrm{\top }}$ reflects nuisance effects across cells. We follow SPARK Sun et al. (2020), setting $s_i$ proportional to the summation of the total number of read counts across all genes for cell $i$, and combine it with a constraint of ${\prod }_i s_i=1$, which gives $s_i={\sum }_j y_{ij}/{\prod }_i {\sum }_j y_{ij}$. By setting the constraint for $s_i$ ’s, we avoid the identifiability problem between $s_i$ ’s and ${\nu }_{ij}$ ’s.

To denoise the relative expression levels, we aim to partition ${\nu }_{ij}$ into two groups by introducing the ZINB mixture model. Dichotomization has been widely applied as a step in the analysis of SRT data. BinSpect (Dries et al., 2021) and BOOST-MI (Jiang et al., 2022) discretize the normalized expression levels for each gene into two groups for more robust SVG detection results. Here, we introduce a latent binary gene expression level indicator vector ${\mathit{\xi }}_{·j}$ to denote the dichotomized expression profiles of each gene $j$. If ${\xi }_{ij}=1$, gene $j$ is highly expressed at cell $i$, and if ${\xi }_{ij}=0$, gene $j$ has low expression at cell $i$. A mixture model is suggested to allow different ZINB model parametrizations for high and low expression levels for gene $j$, in which we assume the raw expression count $y_{ij}$ is generated one of two independent ZINB distributions with different means given the underlying binary indicator ${\xi }_{ij}$,

$$y_{ij}\mid {\xi }_{ij},{\pi }_i,{\mu }_{0j},{\varphi }_{0j},{\mu }_{1j},{\varphi }_{1j}\sim \left(1-{\xi }_{ij}\right)\mathrm{ZINB}\left({\pi }_i,s_i{\mu }_{0j},{\varphi }_{0j}\right)+{\xi }_{ij}\mathrm{ZINB}\left({\pi }_i,s_i{\mu }_{1j},{\varphi }_{1j}\right),$$ (2)

where ${\mu }_{1j}$ and ${\mu }_{0j}$, respectively, denote the group mean of read count for highly and lowly expressed genes. To guarantee that the mean expression level for a highly expressed gene is higher than a lowly expressed gene, we set a constraint for NB distribution mean across two expression levels: ${\mu }_{1j}>{\mu }_{0j}.{\varphi }_{1j}$ and ${\varphi }_{0j}$ respectively represent the dispersion parameters of the NB model for the highly and lowly expressed gene.

To complete the model, we specify the following prior distributions: ${\mu }_{0j},{\mu }_{1j}\sim \mathrm{G}\mathrm{a}\left(a_{\mu },b_{\mu }\right)$, s.t. ${\mu }_{1j}>{\mu }_{0j}>0$ and ${\varphi }_{0j},{\varphi }_{1j}\sim \mathrm{G}\mathrm{a}\left(a_{\varphi },b_{\varphi }\right)$. To create an environment conducive to model fitting, we introduce a latent variable ${\eta }_{ij}$ to indicate whether a zero count $y_{ij}$ is from the zero or NB component in Equation (1), and impose a Bernoulli prior ${\eta }_{ij}\sim \mathrm{B}\mathrm{e}\mathrm{r}\mathrm{n}\left({\pi }_i\right)$, which can be further relaxed by formulating a $\mathrm{B}\mathrm{e}\mathrm{t}\mathrm{a}\left(a_{\pi },b_{\pi }\right)$ hyperprior on ${\pi }_i$, leading to a Beta-Bernoulli prior for ${\eta }_{ij}$ with expectation $a_{\pi }/\left(a_{\pi }+b_{\pi }\right)$.

2.2. A brief review of the Bayesian mark interaction model

Marked point interaction models are statistical models for spatial point pattern analysis with applications across diverse fields such as geostatistics, ecology, material physics, and so on (Edsgärd et al., 2018; Li et al., 2019). These models are designed to study the interactions among points with numerical or categorical marks in a planar region. Marked point models are receiving greater and greater focus in biology: for instance, Trendsceek applies the marked point process to identify SVGs (Edsgärd et al., 2018). The Bayesian mark interaction model, proposed by Li et al. (2019), is a full Bayesian model that characterizes spatial correlations among cell types from tumor pathology images.

Let $\left(t_{i1},t_{i2}\right)\in {\mathbb{R}}^2$, $i=1,\dots ,n$ be the $x$- and $y$-coordinates of point $i$. Let $G=(V,E)$ denote an interaction network with a finite set of points $V$ and a set of direct interactions $E$. In the introduced Bayesian mark interaction model, we assume points have categorical marks. Here, we denote $\mathit{\xi }={\left({\xi }_1,\cdots ,{\xi }_n\right)}^{\mathrm{\top }}$ as the categorical marks of $n$ points on the plane. ${\xi }_i\in (1,\cdots ,Q)$, $Q\ge 2$ are the marks of point $i$.

The Bayesian mark interaction model formulates the pattern of marks $\mathit{\xi }$ via the energy function, which is first introduced in statistical mechanics. The energy function has terms to account for both first- and second-order properties of the marked point data. Specifically, to model the interaction energy between two points, an exponential decay function with respect to the distance between the two points is used. Moreover, the Bayesian mark interaction model neglects interaction terms of point pairs from $E$ when the corresponding distance is beyond a threshold $c$. Consequently, the model focuses on a sparse network $G^{\prime }=\left(V,E^{\prime }\right)$, where $E^{\prime }$ includes edges joining pairs of points $i$ and $i^{\prime }$ with distance $d_{i{i}^{\prime }}<c$. The setting of the distance threshold is added to avoid the high computation cost incurred when summing over $n$ data points and $(n-1)n/2$ interacting pairs of points with large $n$. Then, the potential energy of $G^{\prime }$ is measured by two addictive terms,

$$V\left(\xi |\omega ,\mathbf{\Theta },\lambda \right)=\sum _q\left({\omega }_q\sum _{i}I\left({\xi }_i=q\right)\right)+\sum _q\sum _{q^{\prime }}\left({\theta }_{q{q}^{\prime }}\sum _{\left(i\sim i^{\prime }\right)\in E^{\prime }}\exp \left(-\lambda d_{i{i}^{\prime }}\right)I\left({\xi }_i=q,{\xi }_{i^{\prime }}=q^{\prime }\right)\right),$$ (3)

where $q,q^{\prime }\in \{1,\cdots ,Q\}$ are the categories of marks. $\mathit{\omega }={\left({\omega }_1,\cdots ,{\omega }_Q\right)}^{\mathrm{\top }}$ and $\mathbf{\Theta }={\left[{\theta }_{q{q}^{\prime }}\right]}_{Q\times Q}$ are defined as first- and second-order intensities. $\left(i\sim i^{\prime }\right)$ denotes the collection of interacting pairs of cells in $G^{\prime }.d_{i{i}^{\prime }}=\sqrt{{\left(t_{i1}-t_{i^{\prime }1}\right)}^2+{\left(t_{i2}-t_{i^{\prime }2}\right)}^2}$ denotes the Euclidean distance between point $i$ and $i^{\prime }.$ $\lambda$ is the decay parameter of the distance between two points in the exponential decay function, where a larger $\lambda$ makes energy diminish quickly with respect to the increase in point pair distance.

By restricting the interaction effect within radius $c_d$, the Bayesian mark interaction model defines a local energy. According to the fundamental Hammersley-Clifford theorem (Clifford, 1990), a probability measure with a Markov property exists if we have a locally defined energy, called a Gibbs measure. This measure gives the probability of observing categorical marks associated with their locations in a particular state. We can write the joint probability on marks $\mathit{\xi }$ as,

$$\pi (\mathit{\xi }\mid \mathit{\omega },\mathbf{\Theta },\lambda )=\frac{\mathrm{e}\mathrm{x}\mathrm{p}(-V(\mathit{\xi }\mid \mathit{\omega },\mathbf{\Theta },\lambda \left)\right)}{\sum _{{\mathit{\xi }}^{\prime }\in \mathrm{\Xi }} \mathrm{e}\mathrm{x}\mathrm{p}\left(-V\left({\mathit{\xi }}^{\prime }\mid \mathit{\omega },\mathbf{\Theta },\lambda \right)\right)}$$ (4)

which is proportional to the exponential of the negative energy of marks $\mathit{\xi }$ calculated by Equation (3). The denominator is a normalizing constant that needs to sum over the entire space $\mathrm{\Xi }$ of marks combination consisting of $Q^n$ states, which is intractable even for a small size model.

The joint probability (Equation (4)) can be considered as the full data likelihood. To interpret the parameters clearly, we write the probability of observing point $i$ having mark category $q$ conditional on its neighborhood configuration,

$$\pi \left({\xi }_i=q\mid {\xi }_{-i},\omega ,\mathbf{\Theta },\lambda \right)\propto \exp \left(-{\omega }_q-\sum _{q^{\prime }}\left({\theta }_{q{q}^{\prime }}\sum _{\left\{i^{\prime }:\left(i\sim i^{\prime }\right)\in E^{\prime }\right\}}\exp \left(-\lambda d_{i{i}^{\prime }}\right)I\left({\xi }_{i^{\prime }}=q^{\prime }\right)\right)\right),$$ (5)

where ${\mathit{\xi }}_{-i}$ denotes the collection of all marks, except the $i\mathrm{t}\mathrm{h}$ one. Equation (5) shows that the probability of point $i$ with mark $q$ depends on parameter ${\omega }_q$, ${\theta }_{q{q}^{\prime }}$, and the decay parameter $\lambda$. Parameters in Equation (5) are interpreted below. Suppose there is no interaction between any two points in the space, i.e., ${\theta }_{q{q}^{\prime }}=0$; then, the conditional probability of point $i$ with ${\xi }_i=q$ is $\pi \left({\xi }_i=q\mid \cdot \right)\propto \mathrm{e}\mathrm{x}\mathrm{p}\left(-{\omega }_q\right)$. Therefore, the model parameter ${\omega }_q$ is related to the abundance of points with mark $q$. Fixing $\mathit{\omega }$ as equal values, we obtain the conditional probability of point $i$ with ${\xi }_i=q$ is $\left({\xi }_i=q\mid \cdot \right)\propto \mathrm{e}\mathrm{x}\mathrm{p}(-{\sum }_{q^{\prime }} [{\sum }_{\left\{i^{\prime }:\left(i\sim i^{\prime }\right)\in E^{\prime }\right\}} \mathrm{e}\mathrm{x}\mathrm{p}\left(-\lambda d_{i{i}^{\prime }}\right)I\left({\xi }_{i^{\prime }}=q^{\prime }\right)])$. The second-order intensity ${\theta }_{q{q}^{\prime }}$ quantifies the dependency of mark $q$ with the nearby points, with mark $q^{\prime }$ scaling by the distance decay function. A detailed parameter interpretation is provided by Li et al. (2019).

2.3. A hidden mark interaction model for identifying SVGs

In Section 2.1, we describe how our ZINB mixture model is used to convert read counts ${\mathit{y}}_{\cdot j}$ for each gene $j$ into their corresponding hidden binary states ${\mathit{\xi }}_{\cdot j}$. This dichotomization process allows us to represent gene expression in a binary format. We then treat the spatial distribution of cells as a two-dimensional point process, with the binary gene expressions ${\mathit{\xi }}_{·j}$ serving as the markers of these points. This setup enables us to effectively employ the mark interaction model to assess the spatial correlations among these markers. For simplicity, as outlined in Section 2.2, we treat the decay parameter $\lambda$ as a pre-defined hyperparameter ${\lambda }_0$. Within this framework, the energy function can be expressed as follows:

$$V\left({\mathit{\xi }}_{\cdot j}\mid {\omega }_{0j},{\omega }_{1j},{\theta }_j\right)={\omega }_{0j}\sum _{i}I\left({\xi }_{ij}=0\right)+{\omega }_{1j}\sum _{i}I\left({\xi }_{ij}=1\right)+\sum _{{}_{i\sim i^{\prime }}}I\left(d_{i{i}^{\prime }}<c_d\right){\theta }_j\text{exp}\left(-{\lambda }_0{d}_{i{i}^{\prime }}\right)I\left({\xi }_{ij}\ne {\xi }_{i^{\prime }j}\right)$$ (6)

To interpret the model parameters, we provide the conditional probability of observing a high-expression level ${\xi }_{ij}=1$ of gene $j$ at cell $i$, given the expression levels of other cells for gene $j$:

$$\pi \left({\xi }_{ij}=1\mid {\mathit{\xi }}_{-i,j},{\mathit{\omega }}_j,{\theta }_j\right)\propto \text{exp}\left(-{\omega }_{1j}-{\theta }_j\sum _{i\sim i^{\prime }}I\left(d_{i{i}^{\prime }}<c_d\right)\text{exp}\left(-{\lambda }_0{d}_{i{i}^{\prime }}\right)I\left({\xi }_{i^{\prime }j}=0\right)\right)$$ (7)

As introduced in Section 2.2, the model parameters ${\mathit{\omega }}_j={\left({\omega }_{0j},{\omega }_{1j}\right)}^{\mathrm{\top }}$ represent the first-order property, and ${\theta }_j$ reflects the second-order property of the spatial distribution of marks. Specifically, model parameters ${\omega }_{0j}$ and ${\omega }_{1j}$ are related to the abundance of cells with low and high expression levels of gene $j$, respectively. From Equation (7), if we assume ${\theta }_j=0$, then the distribution of ${\xi }_{ij}$ for cell $i$ is independent of the values of other cells and the proportion of high expression level $\pi \left({\xi }_{ij}=1\mid \cdot \right)=\mathrm{e}\mathrm{x}\mathrm{p}\left({\omega }_{1j}\right)/\left(\mathrm{e}\mathrm{x}\mathrm{p}\left({\omega }_{0j}\right)+\mathrm{e}\mathrm{x}\mathrm{p}\left({\omega }_{1j}\right)\right)$. Assuming ${\mathit{\omega }}_j$ are fixed, if the interaction parameter ${\theta }_j$ between lowly and highly expressed cells is positive, a larger probability of cell $i$ belonging to the highly expressed group corresponds to fewer lowly expressed cells in the surrounding area (i.e., fewer terms $I\left({\xi }_{i^{\prime }j}=0\right)$ ). In other words, if ${\theta }_j>0$, the gene expression level at cell $i$ tends to be concordant with the majority of its neighboring cells, resulting in a repulsion pattern. If the interaction ${\theta }_j$ is negative, we can further infer that there is a spatial pattern in which cells with different expression levels are clustered, resulting in an attraction pattern. Consequently, we identified the association between cell interactions, which potentially results in a spatial pattern, and the second-order intensity parameter ${\theta }_j$. Parameter ${\lambda }_0$ controls the change of interaction strength between a pair of points with respect to their distance. A larger ${\lambda }_0$ causes the interaction between two cells to diminish faster, resulting in a smaller interactive neighborhood for each cell. As a hyperparameter, ${\lambda }_0$ needs to be set appropriately to reflect the interaction neighborhood for cells.

An identifiability problem arises when adding a nonzero constant to ${\omega }_{0j}$ and ${\omega }_{1j}$, as it causes the joint probability $\pi \left({\mathit{\xi }}_{\cdot j}\mid \cdot \right)$ to remain invariant. Therefore, we constrain ${\omega }_{1j}=1$ and establish prior distributions for ${\omega }_{0j}$ and ${\theta }_j$ to complete the parameter model settings for the hidden mark interaction model: ${\omega }_{0j}\sim \mathbf{N}\left({\mu }_{\omega },{\tau }_{\omega }^2\right)$ and ${\theta }_j\sim \mathbf{N}\left({\mu }_{\theta },{\tau }_{\theta }^2\right)$.

3. MODEL FITTING

In this section, we introduce the MCMC algorithm for model fitting and posterior inference. Our model space consists of $\left(\mathit{M},\mathbf{\Phi },\mathit{H},\mathbf{\Xi },{\mathit{\omega }}_0,\mathit{\theta }\right)$ with the underlying grouped gene expression levels $\mathit{M}={\left[{\mu }_{kj}\right]}_{2\times p}$, the dispersion parameters $\mathbf{\Phi }={\left[{\varphi }_{kj}\right]}_{2\times p}$, the extra zero indicators $\mathit{H}={\left[{\eta }_{ij}\right]}_{n\times p}$, the binary expression level indicators $\mathbf{\Xi }={\left[{\xi }_{ij}\right]}_{n\times p}$, the first-order intensity parameter ${\mathit{\omega }}_0={\left({\omega }_{01},\dots ,{\omega }_{0p}\right)}^{\mathrm{\top }}$ and the interaction parameter $\mathit{\theta }={\left({\theta }_1,\dots ,{\theta }_p\right)}^{\mathrm{\top }}$ in the mark interaction model. Each gene is examined independently by BOOST-HMI. We give the full posterior distribution for gene $j$ as,

$$\begin{array}{l}\pi \left({\mathit{\mu }}_{\cdot j},{\varphi }_j,{\mathit{\eta }}_{\cdot j},{\mathit{\xi }}_{\cdot j},{\omega }_{0j},{\theta }_j\mid {\mathit{y}}_{\cdot j}\right)\propto \left[\prod _i\left(y_{ij}\mid {\xi }_{ij},{\eta }_{ij},{\mathit{\mu }}_{\cdot j},{\varphi }_j\right)\right]\times \pi \left({\xi }_{\cdot j}\mid {\omega }_{0j},{\omega }_{1j}=1,{\theta }_j\right)\times \\ \pi \left({\mu }_{0j}\right)\times \pi \left({\mu }_{1j}\right)\times \pi \left({\varphi }_{0j}\right)\times \pi \left({\varphi }_{1j}\right)\times \left[\prod _{i=1}^n\pi \left({\eta }_{ij}\right)\right]\times \pi \left({\omega }_{0j}\right)\times \pi \left({\theta }_j\right)\end{array}.$$ (8)

Our primary aim was to infer ${\omega }_{0j}$, ${\theta }_j$ and ${\mathit{\xi }}_{\cdot j}$, which define the Gibbs probability measure based on the local energy function. We provide estimation and inference on first-order intensity ${\omega }_{0j}$, which represents the abundance of lowly expressed levels of gene $j$, and the second-order intensity ${\theta }_j$, which captures the spatial correlation between two expression levels. The estimated latent gene expression level indicator provides a robust estimation of the spatial organization of marks.

3.1. MCMC algorithms

We estimate ${\mu }_{0j}$, ${\mu }_{1j}$, ${\varphi }_{0j}$ and ${\varphi }_{1j}$ using the random walk Metropolis-Hastings (RWMH) algorithm. ${\mathit{\eta }}_{\cdot j}$ and ${\mathit{\xi }}_{.j}$ are estimated with a Gibbs sampler. The Gibbs probability measure for the distribution of latent gene expression indicator ${\mathit{\xi }}_{\cdot j}$ in Equation (7) omits an intractable normalizing constant $\left({\omega }_{0j},{\omega }_{1j},{\theta }_j\right)={\sum }_{{\mathit{\xi }}_{\cdot j}^{\prime }} \mathrm{e}\mathrm{x}\mathrm{p}\left(-H\left({\mathit{\xi }}_{\cdot j}^{\prime }\mid {\omega }_{0j},{\omega }_{1j},{\theta }_j\right)\right)$, which makes the Metropolis-Hastings algorithm infeasible. For instance, to model a gene expression profile with $n=257$ cells, we need to traverse $2^{257}\approx 2.3\times {10}^{77}$ different arrangements of $\mathit{\xi }$ for every gene, which is a heavy computational burden. To overcome this issue, we use the double Metropolis-Hastings (DMH) algorithm proposed by Liang et al. Liang (2010) to estimate ${\omega }_{0j}$ and ${\theta }_j$ by canceling the intractable normalizing constant. The DMH is an efficient auxiliary variable MCMC algorithm. In contrast to other auxiliary MCMC algorithms, it does not require drawing the auxiliary variables from a perfect sampler, which usually increases computational cost (Møller et al., 2006). The full details of MCMC algorithms is described in the supplementary materials.

3.2. Posterior inference

Posterior inference of parameters ${\mu }_{0j}$, ${\mu }_{1j}$, ${\varphi }_{0j}$, ${\varphi }_{1j}$, ${\omega }_{0j}$, and ${\theta }_j$ is obtained by averaging the MCMC posterior samples after burn-in. We are interested in identifying the SVGs by summarizing the interaction parameter $\mathit{\theta }$. As stated in Section 2.3, investigating whether ${\theta }_j$ is positive or negative is of great importance to inferring the spatial expression pattern of gene $j$. To check the sign of ${\theta }_j$, we applied hypothesis testing ${\mathcal{M}}_0:{\theta }_j\ge 0$ versus ${\mathcal{M}}_1:{\theta }_j\le 0$. If there is strong evidence to reject the null hypothesis ${\mathcal{M}}_0$, we conclude that gene $j$ is an SVG. The Bayes factor (BF) is computed to infer whether ${\theta }_j$ is positive or negative with statistical significance from the MCMC algorithm results. The Bayes factor measures the favorability of ${\mathcal{M}}_1$ as

$${\mathrm{B}\mathrm{F}}_j=\frac{\pi \left(\mathit{y}\cdot j\mid {\mathcal{M}}_1\right)}{\pi \left(\mathit{y}\cdot j\mid {\mathcal{M}}_0\right)}=\frac{\pi \left({\mathcal{M}}_1\mid {\mathit{y}}_{\cdot j}\right)}{\pi \left({\mathcal{M}}_0\mid {\mathit{y}}_{\cdot j}\right)}\frac{\pi \left({\mathcal{M}}_0\right)}{\pi \left({\mathcal{M}}_1\right)}\approx \frac{{\sum }_u I({\theta }_j^{\left(u\right)}<0)}{{\sum }_u I({\theta }_j^{\left(u\right)}\ge 0)},$$ (9)

where $u$ index the iteration and $U$ is the total number of iterations after burn-in. The larger the ${\mathrm{B}\mathrm{F}}_j$, the more likely gene $j$ is an SVG with an attraction pattern. The smaller the ${\mathrm{B}\mathrm{F}}_j$, the more likely gene $j$ is an SVG with a repulsion pattern.

Another important parameter in our model is the latent gene expression level indicator ${\mathit{\xi }}_{\cdot j}$. We summarize the posterior distribution of ${\mathit{\xi }}_{\cdot j}$ via maximum-a-posteriori (MAP) estimates, which is the mode of the posterior distribution. A more comprehensive summary of ${\mathit{\xi }}_{\cdot j}$ ’s is based on their marginal posterior probabilities of inclusion (PPI), where ${\mathrm{P}\mathrm{P}\mathrm{I}}_{ij}={\sum }_{u=1}^U {\xi }_{ij}^{\left(u\right)}/U$. Then, the latent expression indicator indicates a high expression spot when PPIs are greater than a threshold $c_p$:

$${\mathit{\xi }}_{\cdot j}^{\mathrm{P}\mathrm{P}\mathrm{I}}={\left(I\left({\mathrm{P}\mathrm{P}\mathrm{I}}_{1j}\ge c_p\right),\cdots ,I\left({\mathrm{P}\mathrm{P}\mathrm{I}}_{nj}\ge c_p\right)\right)}^{\mathrm{\top }}.$$

4. RESULTS

4.1. Simulation Study

We generated simulated data to evaluate the ability of BOOST-HMI to identify SVGs and provide a comparison with four competing methods: SpatialDE (Svensson et al., 2018), SPARK (Sun et al., 2020), BinSpect-kmean (Dries et al., 2021), and BinSpect-rank (Dries et al., 2021).

Spatial locations of simulated data were from the geospatial profile of the mouse hippocampus dataset field 43 (Shah et al., 2016) with $n=257$ cells, which we present in Section 4.2. To generate the expression counts for gene $j$, the latent gene expression level indicators ${\mathit{\xi }}_{\cdot j}$ ’s were first generated based on Equation (7) with three different values of ${\omega }_{0j}\in \{1.4,1,0.6\}$ and the fixed value of ${\omega }_{1j}=1$. These three values of ${\mathit{\omega }}_j$ correspond to approximately 60%, 50%, and 40% lowly-expressed cells in ${\mathit{\xi }}_{\cdot j}$. Additionally, we set four different values of ${\theta }_j\in \{-2.5,-1.2,1.9,3.2\}$ to generate SVGs with various patterns of attraction or repulsion. These values correspond to strong attraction, weak attraction, weak repulsion, and strong repulsion patterns, respectively. For the non-SVG, ${\theta }_j=0$ which indicates complete randomness and no spatial correlation. The distance threshold $c_d$ and decay parameter ${\lambda }_0$ were set as 0.15 and 20, respectively. We then simulated gene expression data from a ZINB model with three different group-mean ratios $r\in \{2,5,10\}$ between high and low expression:

$$y_{ij}\mid {\xi }_{ij},{\eta }_{ij},{\mu }_{0j},{\mu }_{1j},{\varphi }_{0j},{\varphi }_{1j}\sim \left\{\begin{array}{ll}\mathrm{N}\mathrm{B}{\left(y_{ij};s_i{\beta }_{0}r,{\varphi }_{1j}\right)}^{{\xi }_{ij}}\mathrm{N}\mathrm{B}{\left(y_{ij};s_i{\beta }_0,{\varphi }_{0j}\right)}^{1-{\xi }_{ij}},& {\eta }_{ij}=0\\ I\left(y_{ij}=0\right)& {\eta }_{ij}=1\end{array}\right.,$$

where the underlying baseline expression levels ${\beta }_0=10$. In the simulation study, size factors $\mathit{s}={\left(s_1,\cdots ,s_n\right)}^{\mathrm{\top }}$ were generated from $\mathrm{l}\mathrm{o}\mathrm{g}-\mathrm{N}\left(0,{0.2}^2\right)$, and dispersion parameters ${\varphi }_{0j}$, ${\varphi }_{1j}$ in the NB model are generated from an exponential distribution $\mathrm{E}\mathrm{x}\mathrm{p}(1/10)$. Further, to imitate high sparsity and account for medium sparsity in real SRT data, we created three sets of sparsity levels, 0–10%,10–20%, and 30–40%, and generated extra zero parameters ${\mathit{\eta }}_{\cdot j}$ ’s correspondingly. Extra zeros were randomly selected and imputed into the generated gene expression count data. Thus, we considered three group-mean ratios and three sparsity levels, which is 3 × 3 = 9 scenarios in total. For each scenario, we simulated 30 replicates with $p=100$ genes in each replicate, 10 out of which were SVGs.

Before estimating the parameters using BOOST-HMI, we specified the prior distributions. Non-informative gamma priors were specified for ${\mu }_{0j}$, ${\mu }_{1j}$, ${\varphi }_{0j}$ and ${\varphi }_{1j}$, i.e., ${\mu }_{0j}\sim \mathrm{G}\mathrm{a}\left(a_{\mu },b_{\mu }\right)$ and ${\varphi }_{0j}\sim \mathrm{G}\mathrm{a}\left(a_{\varphi },b_{\varphi }\right)$. We set $a_{\mu }$, $b_{\mu }$, $a_{\varphi }$, and $b_{\varphi }$ to 0.01, which produced a gamma distribution with mean 1 and variance 100. Priors for ${\omega }_{0j}$ and ${\theta }_j$ were set to control the gene expression abundance and gene expression pattern, ${\omega }_{0j}\sim \mathrm{N}\left(1,{\tau }_{\omega }^2\right)$ and ${\theta }_j\sim \mathrm{N}\left(0,{\tau }_{\theta }^2\right)$. In the simulation study and real data analysis, we set ${\tau }_{\omega }=0.5$, where the prior distribution of ${\omega }_{0j}$ indicates that the latent proportion of low expression cells ranges from 1% to 100% with a probability of 95%. ${\tau }_{\theta }$ was set to 3.5 such that the prior for ${\theta }_j$ guarantees that ${\theta }_j$ falls within −6 to 6 with a probability of 92%. For hyperparameters in the energy function, we set the distance threshold $c_d=0.15$ and expected the relationship of decay parameter ${\lambda }_0$ and $c_d$ to be $\mathrm{e}\mathrm{x}\mathrm{p}\left(-{\lambda }_0{c}_d\right)=0.05$, which specifies the range of exponential decay function $\mathrm{e}\mathrm{x}\mathrm{p}\left(-{\lambda }_0{d}_{i{i}^{\prime }}\right)$ to be [0.05,1] as $c_d\ge d_{i{i}^{\prime }}\ge 0$; therefore, we set ${\lambda }_0$ as 20 correspondingly. As for the setting of the MCMC algorithm, we implemented BOOST-HMI in a gene-wise fashion. For each gene, we initialized model parameters by randomly drawing from their prior distributions. The MCMC algorithm is iterated $U=10,000$ times after 10,000-iteration burn-in. The algorithm was implemented in R and Rcpp. As mentioned in Section 3.2, BOOST-HMI identifies SVGs based on Bayes Factors (BFs). In our study, we select a BF threshold of 10, which indicates strong evidence in favor of the ${\mathcal{M}}_1$ (Kass and Raftery, 1995).

We implemented the other four competing methods with their default settings. SpatialDE, BinSpect and SPARK use $P$-values to select SVGs. To control type-I error rate, the Benjamini-Hochberg (BH) (Benjamini and Hochberg, 1995) procedure was used to adjust $P$-values from SpatialDE and BinSpect. We specifically avoided adjusting $P$-values from SPARK since its raw $P$-values are calibrated by the Cauchy combination rule (Sun et al., 2020; Liu et al., 2019). For $P$-values, the threshold was set to 0.05.

Our task is to evaluate the ability of each method to correctly identify underlying SVGs from the simulated dataset, which can be defined as a binary classification problem; therefore, to evaluate the performance of the five methods, we employed two performance metrics for binary classification problems: First, we used the area under the curve (AUC) (Bradley, 1997) of the receiver operating characteristic (ROC) (Fukunaga, 2013). The ROC is a plot of the true positive rate against the false positive rate for different classification thresholds. The AUC is a single value ranging from 0 to 1, with a higher value indicating better classification performance.

Figure 2 displays a boxplot of AUCs calculated by the aforementioned five methods over 30 replicates across nine scenarios. It clearly suggests that BOOST-HMI achieves superior performance compared to the other methods, especially for when there was high sparsity. BinSpect-kmeans and BinSpect-rank showed competitive performance when there was no zero-inflation, i.e., when the sparsity level was between 0% to 10%, regardless of the different group ratios; however, these methods showed decreasing AUCs as sparsity level increased. SPARK and SpatialDE suffered from a limited ability to detect SVGs from low expression variability or high zero-inflation scenarios. Between SPARK and SpatialDE, the simulation study showed that SPARK has better SVG detection power over SpatialDE, which is consistent with the conclusion drawn by Sun et al. (2020) and Jiang et al. (2022). In summary, BOOST-HMI achieved satisfactory performance and is robust against different group expression level ratios and sparsity levels.

Figure 2.

Simulation study: The boxplots of AUCs achieved by BOOST-HMI, SPARK, SpatialDE, BinSpect-kmeans, and BinSpect-rank across nine scenarios.

The second metric we used is the Matthews correlation coefficient (MCC). MCC is a summary value that examines the binary classification performance under a specific cutoff, i.e., BF or $P$–value thresholds for our study. It has values ranging from −1 to 1, incorporating true positives, true negatives, false positives and false negatives. A larger MCC value, such as 1, corresponds to an excellent classifier, while a negative MCC indicates a strong disagreement between prediction and observation. Table 1 summarizes the average MCCs obtained in the simulation study across the five methods. The result is consistent with our conclusion from our analysis of the AUC: BOOST-HMI achieved the highest power under high zero-inflation, while all other four methods suffered from the high number of false zeros. In the scenario without zero inflation, BinSpect-kmeans stood out, and BinSpect-rank, SPARK, and BOOST-HMI showed competitive performance in identifying SVGs.

Table 1.

Simulation study: The averaged MCCs, with standard deviations in parentheses, achieved by BOOST-HMI, BinSpect-kmeans, BinSpect-rank, SPARK, and SpatialDE across nine scenarios.

	0 – 10% zeros
	R = 2	R = 5	R = 10
BinSpect-kmeans	0.182 (0.159)	0.653 (0.118)	0.714 (0.085)
BinSpect-rank	0.199 (0.193)	0.476 (0.123)	0.488 (0.139)
SPARK	0.255 (0.162)	0.502 (0.164)	0.524 (0.153)
SpatialDE	0.009 (0.111)	−0.009 (0.063)	0.027 (0.109)
BOOST-HMI	0.231 (0.141)	0.449 (0.084)	0.510 (0.063)
	10% - 20% zeros
	R = 2	R = 5	R = 10
BinSpect-kmeans	0.007 (0.092)	0.121 (0.134)	0.601 (0.142)
BinSpect-rank	0.173 (0.166)	0.482 (0.120)	0.537 (0.153)
SPARK	0.113 (0.146)	0.271 (0.153)	0.373 (0.123)
SpatialDE	−0.039 (0.043)	0.005 (0.093)	0.031 (0.120)
BOOST-HMI	0.160 (0.176)	0.433 (0.091)	0.473 (0.065)
	30% - 40% zeros
	R = 2	R = 5	R = 10
BinSpect-kmeans	0.002 (0.086)	0.005 (0.083)	−0.006 (0.105)
BinSpect-rank	0.060 (0.139)	0.243 (0.157)	0.226 (0.142)
SPARK	0.014 (0.103)	0.133 (0.150)	0.144 (0.155)
SpatialDE	−0.016 (0.071)	−0.001 (0.097)	0.016 (0.110)
BOOST-HMI	0.098 (0.158)	0.406 (0.088)	0.427 (0.088)

4.2. Application to the mouse hippocampus seqFISH data

The mouse hippocampus dataset is a public seqFISH dataset with 21 field replicates collected on a third coronal section (Shah et al., 2016). Following SpatialDE and SPARK protocols, we analyzed the field 43 dataset, which contains $p=249$ genes measured on 257 cells with spatial location preserved. Out of 249 genes, 214 were selected from a list of transcription factors and signaling pathway components, and the remaining 35 were selected from cell identity markers. Quality control was performed following SPARK protocol (Sun et al., 2020) and the original study. We filtered out cells with $x$ - or $y$-axis values exceeding 203–822 pixels to tackle border artifacts. After filtering, $n=131$ cells were included for the following analysis. We excluded SpatialDE due to its unsatisfactory performance in SVG detection in the simulation study. Prior settings, MCMC algorithm implementation, and significance criteria were identical to what was described in Section 4.1. Five independent MCMC chains were sampled and diagnosed for algorithm convergence. Algorithm convergence was checked based on the BF vector. BFs from five chains were highly correlated with Pearson correlation coefficients ranging from 0.90 to 0.99. We further checked algorithm convergence using the potential scale reduction factor (PSRF) (Gelman and Rubin, 1992; Brooks and Gelman, 1998) on posterior samples of ${\theta }_j$ ’s and ${\omega }_{0j}$ ’s. If multiple chains converge to the target posterior distribution, the PSRF will be close to one. In our analysis, the PSRFs were below 1.2, suggesting convergence of the MCMC algorithms. Posterior samples obtained from the quintuplet of MCMC chains were amalgamated for subsequent analysis.

Among the $p=249$ genes analyzed in the mouse hippocampus dataset, SPARK identified 17 SVGs, while BOOST-HMI detected 22 SVGs. Notably, BOOST-HMI successfully detected 16 cell identity markers previously presented by Shah et al. (2016), whereas SPARK identified 14 markers. In comparison, BinSpect-kmeans and BinSpect-rank were more aggressive, respectively identifying 38 and 44 SVGs. A Venn diagram in Figure 3(D) showcases the overlap of SVGs identified by the four methods. Among them, BOOST-HMI and SPARK shared 12 SVGs in common. Only one SVG detected by BinSpect-kmeans overlapped with that from BOOST-HMI, and none overlapped with that from SPARK. None of the SVGs detected by BinSpect-rank were detected by either SPARK or BOOST-HMI. We further visualized the spatial pattern for each SVG using the marginal PPI of the posterior samples of the hidden gene expression indicator ${\mathit{\xi }}_{\cdot j}$. Figure 3(A) displays the spatial patterns of SVGs detected by SPARK and BOOST-HMI, while Figures S1 and S2 in the supplementary materials depict the spatial patterns for SVGs detected by BinSpect-kmeans and BinSpect-rank, respectively. Among the 12 common SVGs identified by SPARK and BOOST-HMI, strong spatial repulsion patterns between high- and low expression genes are evident across 131 cells. Notably, a larger Bayes factor (shown in parentheses) indicates a stronger spatial pattern; genes Foxol, sst, mog, myll4, and ndnf exhibited clear spatial patterns between polarized estimated hidden indicators. SPARK detected five unique SVGs, as Figure 3 (B) shows, for which the PPIs of the estimated hidden expression indicators are close to 0.5. BOOST-HMI identified five unique SVGs, as displayed in Figure 3(C). Among these, seven genes, such as gene Zfp423, slc5a7 and palvb, demonstrated either high or low expression in the majority of cells. The remaining three unique SVGs, Zic3, Mnat1, and slc 17a7 delineated three distinct patterns, which may be related to novel biological mechanisms. Lim et al. (2007) previously highlighted the crucial role of Zic3 in preserving pluripotency in embryonic stem cells, while Herman and El-Hodiri (2002) demonstrated that mutations in Zic3 are linked to developmental abnormalities such as laterality defects, congenital heart disease, and neural tube defects.

Figure 3.

Mouse hippocampus seqFISH data: (A) Spatial distribution of hidden gene expressions ${\mathit{\xi }}^{\text{MAP}}$ of the 12 SVGs detected by both SPARK and BOOST-HMI. (B) Spatial distribution of hidden gene expressions ${\mathit{\xi }}^{\mathrm{M}\mathrm{A}\mathrm{P}}$ of the five SVGs detected by SPARK only. (C) Spatial distribution of hidden gene expressions ${\mathit{\xi }}^{\mathrm{M}\mathrm{A}\mathrm{P}}$ of the ten SVGs detected by BOOST-HMI only. (D) Venn diagram of the overlap across SVGs identified by all four methods. (E) Enriched GO terms associated with SVGs detected by BOOST-HMI. (F) Boxplot of $\mathrm{M}\mathrm{o}\mathrm{r}\mathrm{a}\mathrm{n}\mathrm{’}\mathrm{s}I$ and $\mathrm{G}\mathrm{e}\mathrm{a}\mathrm{r}\mathrm{y}\mathrm{’}\mathrm{s}C$ values for SVGs across the four methods.

In addition to visualizing spatial patterns, we quantified the degree of spatial attraction or repulsion pattern in gene expression across different cells using the spatial autocorrelation tests $\mathrm{M}\mathrm{o}\mathrm{r}\mathrm{a}\mathrm{n}\mathrm{’}\mathrm{s}I$ and $\mathrm{G}\mathrm{e}\mathrm{a}\mathrm{r}\mathrm{y}\mathrm{’}\mathrm{s}C.\mathrm{M}\mathrm{o}\mathrm{r}\mathrm{a}\mathrm{n}\mathrm{’}\mathrm{s}I$ quantifies the spatial clustering or dispersion by standardizing the spatial autocovariance, yielding a correlation coefficient ranging from −1 to 1. A positive $\mathrm{M}\mathrm{o}\mathrm{r}\mathrm{a}\mathrm{n}\mathrm{’}\mathrm{s}I$ value corresponds to a spatial clustering pattern where the variable tends to have similar values to its neighboring cells. A $\mathrm{M}\mathrm{o}\mathrm{r}\mathrm{a}\mathrm{n}\mathrm{’}\mathrm{s}I$ value close to 0 suggests a random spatial distribution of the data, while a negative value corresponds to a dispersion pattern, where the variable value tends to be dissimilar from its neighbors. To assess the spatial patterns exhibited by the SVGs, $\mathrm{M}\mathrm{o}\mathrm{r}\mathrm{a}\mathrm{n}\mathrm{’}\mathrm{s}I$ and $\mathrm{G}\mathrm{e}\mathrm{a}\mathrm{r}\mathrm{y}\mathrm{’}\mathrm{s}C$ values were calculated for each SVG identified by at least one of the four methods. $\mathrm{M}\mathrm{o}\mathrm{r}\mathrm{a}\mathrm{n}\mathrm{’}\mathrm{s}I$ for each gene $j$ was calculated by the following formula:

$$\text{Moran’s}I=\frac{n}{\sum _i \sum _h w_{ih}}\frac{\sum _i \sum _h w_{ih}\left(y_{ij}-{\bar{y}}_j\right)\left(y_{hj}-{\bar{y}}_j\right)}{\sum _i {\left(y_{ij}-{\bar{y}}_j\right)}^2},$$

where $w_{ij}=A/{\left(d_{ih}\right)}^m$ is the connectivity spatial weight between cell $i$ and $h$. Spatial weight is a decay factor of the distance between two cells; in our study, we set $A=1$, $m=1.$ $y_{ij}$ and $y_{hj}$ are the gene expression count of cell $i$ and cell $h$, and ${\bar{y}}_j$ is the mean expression of gene $j$. Similar to $\mathrm{M}\mathrm{o}\mathrm{r}\mathrm{a}\mathrm{n}\mathrm{’}\mathrm{s}I,\mathrm{G}\mathrm{e}\mathrm{a}\mathrm{r}\mathrm{y}\mathrm{’}\mathrm{s}C$ measures the spatial similarity or dissimilarity between neighboring cells, and is calculated with the following formula:

$$\text{Geary’s}C=\frac{n-1}{2\sum _i \sum _h w_{ih}}\frac{\sum _i \sum _h w_{ih}{\left(y_{ij}-y_{hj}\right)}^2}{\sum _i {\left(y_{ij}-{\bar{y}}_j\right)}^2}.$$

$\mathrm{G}\mathrm{e}\mathrm{a}\mathrm{r}\mathrm{y}\mathrm{’}\mathrm{s}C$ ranges from 0 to 2, where a value close to 0 indicates a spatial attraction pattern, 1 corresponds to complete randomness, and 2 implies a spatial repulsion pattern. To ensure uniform interpretation of $\mathrm{M}\mathrm{o}\mathrm{r}\mathrm{a}\mathrm{n}\mathrm{’}\mathrm{s}I$ and $\mathrm{G}\mathrm{e}\mathrm{a}\mathrm{r}\mathrm{y}\mathrm{’}\mathrm{s}C$, following Hu et al. (2021), we scaled $\mathrm{G}\mathrm{e}\mathrm{a}\mathrm{r}\mathrm{y}\mathrm{’}\mathrm{s}C$ to the range [−1, 1]. The distributions of these values are depicted in Figure 3(F). Remarkably, over 75% of $\mathrm{M}\mathrm{o}\mathrm{r}\mathrm{a}\mathrm{n}\mathrm{’}\mathrm{s}I$ and $\mathrm{G}\mathrm{e}\mathrm{a}\mathrm{r}\mathrm{y}\mathrm{’}\mathrm{s}C$ values were positive, compellingly indicating the presence of spatial patterns associated with the SVGs across the four methods implemented. Moreover, SVGs from BOOST-HMI and SPARK exhibited the highest $\mathrm{M}\mathrm{o}\mathrm{r}\mathrm{a}\mathrm{n}\mathrm{’}\mathrm{s}I$ values, while SVGs from SPARK demonstrated the highest $\mathrm{G}\mathrm{e}\mathrm{a}\mathrm{r}\mathrm{y}\mathrm{’}\mathrm{s}C$ values.

To explore the relevant biological functions of identified SVGs, we conducted gene ontology (GO) enrichment analysis using the R package clusterProfiler (Yu et al., 2012). As mentioned, 214 genes were selected from a list of transcription factors and signaling pathway components. As a result, genes in the background set enriched 4, 622 GO terms and 10, 285 relations. Figure Figure 3(E) depicts the biological processes enriched by SVGs that were detected by BOOST-HMI, such as a smoothened signaling pathway (GO: 0007224), regulation of neural precursor cell proliferation (GO: 20000177), and cellular response to stress (GO: 0033554). Moreover, gene Foxo1 enriched three significant GO terms, which may inspire further research work on Foxo1 regulation in the mouse hippocampus. Mnat1, one of the SVGs, enriched cellular response to stress. Several studies have found that Mnat1 is associated with various disease progression and regulation. Qiu et al. (2020) found that Mnat1, which was detected only by BOOST-HMI, contributes to the progression of osteosarcoma, and Zou et al. (2020) reported that decreased Mnat1 expression induces degradation of an important regulator of necroptosis in endothelial cells from samples with Alzheimer’s disease.

4.3. Application to the mouse visual cortex STARmap data

The second real dataset we analyzed is a STARmap dataset, which profiles the mouse visual cortex from the hippocampus to the corpus callosum, spanning six neocortical layers at single-cell resolution (Wang et al., 2018). The STARmap dataset measures the expression of 1, 020 genes in 1, 549 cells, including non-neuron cells such as endothelial, oligodendrocytes, astrocytes, and neuron cells, i.e., parvalbumin-expressing, vasoactive intestinal peptide-expressing, and somatostatin-expressing interneurons. Figure 4(A) depicts the layer structure and distribution of cell types within the tissue section as presented in the original study (Wang et al., 2018). Moreover, the STARmap dataset is highly sparse with nearly 79% zero counts. To address potential sources of variability, we performed three quality control steps: 1) cells with fewer than 100 read counts detected were filtered out; 2) genes with more than 90% zero counts were filtered out; 3) genes whose maximum count is smaller than ten were removed. After quality control, the gene expression profile measured the expression of $p=107$ genes in $n=1,523$ cells.

Figure 4.

Mouse visual cortex STARmap data: (A) Voronoi diagrams of layer structures and cell type distribution. (B) Spatial distribution of the average hidden gene expressions ${\mathit{\xi }}^{\mathrm{M}\mathrm{A}\mathrm{P}}$ of the six SVG patterns detected by SPARK. (C) Spatial distribution of the average hidden gene expressions ${\mathit{\xi }}^{\text{MAP}}$ of the five SVG patterns detected by BOOST-HMI. (D) Venn diagram of the overlap across SVGs identified by all four methods. (E) Enriched GO terms associated with SVGs detected by BOOST-HMI. (F) Boxplot of $\mathrm{M}\mathrm{o}\mathrm{r}\mathrm{a}\mathrm{n}\mathrm{’}\mathrm{s}I$ and $\mathrm{G}\mathrm{e}\mathrm{a}\mathrm{r}\mathrm{y}\mathrm{’}\mathrm{s}C$ values for SVGs across the four methods.

SPARK, BinSpect-kmeans, and BinSpect-rank were implemented with the same parameter settings as the simulation study. As for BOOST-HMI, we set the distance threshold $c_d=0.05$ and the decay parameter ${\lambda }_0=60$ to satisfy the dependency $\mathrm{e}\mathrm{x}\mathrm{p}\left(-{\lambda }_0{c}_d\right)=0.05$. We ran four independent MCMC chains with the same prior specifications and parameter settings as the simulation study, and made posterior inferences after integrating the posterior samples across the four chains.

As Figure 4(D) shows, SPARK detected 99 SVGs, while BOOST-HMI detected 64 SVGs. Both BinSpect-kmeans and BinSpect-rank detected 101 SVGs. SPARK, BinSpect-kmeans, and BinSpect-rank detected 94 SVGs in common. Compared to other methods, BOOST-HMI was conservative, detecting 59 common SVGs with the other three methods. To further investigate the spatial patterns of detected SVGs, we visualized the estimated hidden gene expression indicator for each SVG, annotated with the corresponding Bayes factor, in Figures S3, S4, S5, and S6 of the supplementary materials. Analysis of the detected genes reveals a noteworthy observation: all four methods identify SVGs associated with layer structures, such as Apod, Apoe, and Egr1, which exhibit high expression in layer L6 and HPC. In contrast, SVGs exclusively detected by the other three methods either lack a clear layer structure or are estimated to be lowly expressed across the entire tissue section, which suggests that BOOST-HMI can detect SVGs with clear spatial patterns and address potential falsification. This conclusion is strongly supported by corroborating evidence from both calculations of $\mathrm{M}\mathrm{o}\mathrm{r}\mathrm{a}\mathrm{n}\mathrm{’}\mathrm{s}I$ and $\mathrm{G}\mathrm{e}\mathrm{a}\mathrm{r}\mathrm{y}\mathrm{’}\mathrm{s}C$. Figure 4(F) demonstrates that SVGs detected by BOOST-HMI show stronger spatial autocorrelation than those identified by the other three methods. To delve deeper into the identified spatial patterns, we performed agglomerative hierarchical clustering on the SVGs detected by SPARK and BOOST-HMI, as shown in Figure 4(C). SVGs detected by SPARK were grouped into six clustered, while those detected by BOOST-HMI formed five clusters. Pattern I, II, III and IV from SPARK and BOOST-HMI demonstrate a similar spatial pattern. Spatial pattern V from BOOST-HMI delineates layers L1, L2/3, and L4, while pattern V from SPARK is associated with layer L6. Pattern VI from SPARK highlights a fraction of cells in layer L1.

To gain insights into biological processes, molecular function, and cellular components, GO enrichment analysis was performed on SVGs identified by BOOST-HMI. Figure Figure 4(E) shows that SVGs detected by BOOST-HMI are implicated in biological processes such as the cellular developmental process and anatomical structure development. Additionally, these SVGs are associated with cellular components such as protein-containing complexes as well as molecular functions such as DNA binding. Notably, the gene Bcl6 significantly enriched the cellular developmental process. Nurieva et al. (2009) found that Bcl6 functions as a regulator of $\mathrm{T}$ follicular helper cell differentiation and B cell-mediated immunity. Our findings have the potential to inspire further novel biological insights.

5. CONCLUSION

This paper introduces BOOST-HMI, a novel method for identifying SVGs in imaging-based SRT datasets. By integrating gene expression data with spatial location, BOOST-HMI employs a ZINB mixture model to effectively handle the excessive zeros typical in SRT data. Additionally, it uses a hidden Bayesian mark interaction model to accurately quantify spatial dependencies in gene expressions.

Our approach is adaptable for analyzing sequencing-based SRT data. We validated BOOST-HMI through a simulation study and analysis of two real datasets, demonstrating its effectiveness across various SRT technologies and tissue sections. The simulation results showed that BOOST-HMI is particularly adept at identifying SVGs in data with high sparsity levels, between 30% and 40%. When analyzing the mouse hippocampus seqFISH seqFISH data, BOOST-HMI achieved comparable results to SPARK, with the identified SVGs exhibiting stronger spatial patterns as quantified by $\mathrm{M}\mathrm{o}\mathrm{r}\mathrm{a}\mathrm{n}\mathrm{’}\mathrm{s}I$ and $\mathrm{G}\mathrm{e}\mathrm{a}\mathrm{r}\mathrm{y}\mathrm{’}\mathrm{s}C$. Moreover, the SVGs identified were enriched in biologically relevant GO terms, such as smoothened signaling pathways and regulation of neural precursor cell proliferation, offering avenues for further biological investigation.

Further analysis of the mouse visual cortex STARmap dataset revealed that BOOST-HMI can identify SVGs with spatial patterns aligning with the underlying cell structure of the tissue. Additionally, GO enrichment analysis indicated that these SVGs are linked to cellular developmental processes, underscoring the potential for novel biological insights.

While our method assumes homogeneity in spatial patterns across tissue sections, this may not hold true for all cases. Future work will aim to generalize BOOST-HMI to accommodate heterogeneous spatial patterns, enhancing its practicality. Another focus will be on scaling the model to accommodate the growing size of SRT datasets, such as those generated by advanced technologies like Slide-seqV2, which can resolve over 19, 600 genes from around 23, 000 cells (Stickels et al., 2021). Enhanced scalability will enable BOOST-HMI to analyze datasets from various technologies like HDST, Slide-seqV2, and others, potentially leading to more groundbreaking biological discoveries.

DATA AVAILABILITY STATEMENT

The R code and related datasets for this study can be found on GitHub at https://github.com/libedeutch/BOOST-HMI/.