BayesTME: An end-to-end method for multiscale spatial transcriptional profiling of the tissue microenvironment

Abstract

Spatial variation in cellular phenotypes underlies heterogeneity in immune recognition and response to therapy in cancer and many other diseases. Spatial transcriptomics holds the potential to quantify such variation, but existing analysis methods are limited by their focus on individual tasks such as spot deconvolution. We present BayesTME, an end-to-end Bayesian method for analyzing spatial transcriptomics data. BayesTME unifies several previously distinct analysis goals under a single, holistic generative model. This unified approach enables BayesTME to deconvolve spots into cell phenotypes without any need for paired single-cell RNA-seq. BayesTME then goes beyond spot deconvolution to uncover spatial expression patterns amongst coordinated subsets of genes within phenotypes, which we term spatial transcriptional programs. BayesTME achieves state-of-the-art performance across myriad benchmarks. On human and zebrafish melanoma tissues, BayesTME identifies spatial transcriptional programs that capture fundamental biological phenomena like bilateral symmetry and tumor-associated fibroblast and macrophage reprogramming. BayesTME is open source (https://github.com/tansey-lab/bayestme).

Blurb:

BayesTME is a method for comprehensive analysis of spatial transcriptomics. BayesTME removes spatial bias and deconvolves spots into cell phenotypes without paired single-cell RNA-seq reference. After deconvolution, BayesTME models cell phenotypes as dynamic processes and uncovers spatial transcriptional programs executed by phenotypes in response to their local tissue microenvironment.

The tissue microenvironment (TME) comprises a heterogeneous mixture of cell phenotypes, subtypes, and spatial structures. The composition of the TME impacts disease progression and therapeutic response. For instance, the composition of immune cells in the tumor microenvironment is a determinant of response to immunotherapy (IO)¹⁷. More recent work suggests that it is not cellular composition but rather the spatial organization of the microenvironment that determines IO response^14,12,35,9. Spatially-unaware approaches, such as single-cell RNA and DNA sequencing (scRNA-seq and scDNA-seq), are able to capture the presence and abundance of different cell types and phenotypes (hereon referred to as simply types)²⁶, but are unable to characterize their spatial organization. Spatial measurements and spatial modeling of the tissue microenvironment in situ present an opportunity to fully uncover and understand the role that spatial structure plays in determining disease progression and therapeutic response.

Spatial transcriptomics (ST) technologies, such as Visium¹, HDST⁴⁴, and Slide-seq³³ enable biologists to measure spatially-resolved gene expression levels at thousands of spots in individual tissue. Each tissue is divided into a grid or lattice of spots, with each spot in the grid typically $50-100\mu m$ wide, typically covering 10–60 cells. The tissue is permeabilized to release mRNAs to capture probes with spot-specific barcodes. Bulk RNA-seq is then run on the captured mRNAs tagged with spatial barcodes. The result is a high-dimensional, spatially-localized gene expression count vector for each spot, representing an aggregate measurement of the gene expression of the cells in the spot.

Modeling spot-wise aggregate measurements is challenging as it requires disentangling at least four sources of spatial variation present in the raw signal. First, technical error, also known as spot bleeding, causes mRNAs to bleed to remote spots and contaminates the raw spatial signal. Second, variation in cell counts changes the absolute number of unique molecular identifiers (UMIs) per spot. Since UMI counts scale with the number of cells in each spot, conventional pre-processing methods like log-normalization break this linear relationship. Third, differences in the cell type proportions in each spot conflate signal strength with cell type prevalence. This complicates analysis as it necessitates performing a difficult deconvolution of each spot into its constituent cell type composition. These three sources of variation obscure the fourth, namely the spatial variation in gene expression within each cell type in response to the microenvironment. Teasing out these different sources of spatial variation in ST data is necessary to obtain a full understanding of the spatial architecture of the tissue microenvironment.

Several methods have been developed that specialize in subsets of these four sources of spatial variation. SpotClean²⁸ corrects spot bleeding by fitting an isotropic Gaussian model to raw UMI counts in order to map them back to their most likely original location. Spatial clustering methods^46,30,8 fuse spots together to effectively capture regions of constant cell type proportion with varying cell counts. Spot deconvolution methods^25,21,19 separate the aggregate signals into independent component signals with each attributable to a different cell type. Spatial differential expression methods^37,38 assess the aggregate spot signal to detect regions where individual genes or gene sets follow a spatial pattern. While each of these methods has moved the field of ST analysis forward, they each have shortcomings, such as making incorrect parametric assumptions, requiring perfect reference scRNA-seq data, or only capturing aggregate signals rather than phenotype-specific ones.

Notably, existing methods assume cells of a given type have a static distribution of gene expression. This assumption is at conflict with the biological knowledge that cells change their behavior in response to their local microenvironment under mechanisms including proliferation, invasion, and drug resistance³². The microenvironment regulates cell behavior and therefore alters gene expression profiles of specific cell phenotypes¹⁸. These microenvironmental influences are particularly relevant in disease contexts. For example, the microenvironment affects each phase of cancer progression and invasion-metastasis cascade⁴³. Chronic inflammation is able to induce tumor initiation, malignant conversion, and invasion¹³. Recent research also shows cancer cells in the interior of a tumor behave differently than cancer cells at the interface with healthy cells¹⁶. Existing methods are unable to accurately capture spatial expression variation within cell types and thus modeling ST data to understand the spatial structure of transcriptomic diversity in each cell type remains an important open problem.

In this paper, we present BayesTME, a holistic Bayesian approach to end-to-end modeling of ST data that goes beyond existing techniques and captures spatial differential expression within cell types. BayesTME uses a single generative model to capture the multiscale and multifaceted spatial signals in ST data. At the highest level, BayesTME models the global pattern of spatial technical error present in raw ST data. As we demonstrate, ST data contain anisotropic technical error, with UMIs bleeding toward a specific direction in each sample. At the intermediate level, BayesTME places spatial fusion priors between spots, adaptively fusing tissue regions together to reveal cellular community structure. This also enables BayesTME to pool statistical strength across spots, enabling it to perform spot deconvolution without single-cell RNA-seq reference. Graph smoothing priors are simultaneously used to capture the spatial heterogeneity of within-phenotype gene expression. These priors enable BayesTME to discover spatial transcriptional programs (STPs), coordinated spatial gene expression patterns among groups of genes within a phenotype. Through an efficient empirical Bayes inference procedure, BayesTME infers all of the latent variables in the generative model with full quantification of uncertainty. Thus, BayesTME provides statistical control of the false discovery rate for marker genes, cell counts, expression profiles, and spatial transcriptional programs.

BayesTME offers a robust, accurate, and unified one-stop-shop for ST analysis that enables the discovery of spatial transcriptional programs—undiscoverable with other approaches—that are vitally important to fully characterizing the TME. However, BayesTME can also be used as a toolbox for ST analyses. To this end, we demonstrate that each component of BayesTME outperforms existing methods on benchmarks, including bleed correction (SpotClean²⁸), reference-free cell type identification (STdeconvolve²⁵), spot deconvolution (cell2location¹⁹, DestVI²¹, RCTD⁴, CARD²²), tissue segmentation (BayesSpace⁴⁶, Giotto⁸, STLearn³⁰, cell2location¹⁹), and within-phenotype spatial gene expression (SpatialDE³⁸, Spark³⁷). We further demonstrate BayesTME’s singular ability to identify spatial transcriptional programs with high power while maintaining tight control over the false discovery rate on the reported spatially-varying genes in each cell type. On real tissues from human melanoma and zebrafish melanoma models, BayesTME identifies spatial transcriptional programs that capture core biological concepts like bilateral symmetry and differential expression between the surface and interior tumor cells. The tumor STPs reveal insight into how malignant cells interface with immune cells at the tumor border, demonstrating the value gained by BayesTME’s ability to reveal this type of spatial information. BayesTME is open source¹, does not require reference scRNA-seq, and all hyperparameters are auto-tuned without the need for any manual user input.

Results

A holistic generative model for spatial transcriptomics.

BayesTME models spatial variation at multiple scales in ST data using a single hierarchical probabilistic model. At the top-level, spot bleeding is modeled via a semi-parametric spatial contamination function. This bleeding model allows for any arbitrary spot bleeding process to be modeled, under the constraint that UMIs are less likely to bleed to spots that are farther away. By leveraging the non-tissue regions as negative controls (spots where the UMI counts should be zero), BayesTME learns this function and then inverts it to estimate the true UMI counts for each in-tissue spot.

At the spot level, BayesTME models true UMI counts in each spot using a carefully specified negative binomial distribution. The spot convolution effects due to cell aggregation are captured in the rate parameter. This ensures that a linear increase in the number of a particular cell type yields a linear increase in the UMIs from that cell type. The success probability parameter in the negative binomial likelihood is used to capture spatial variation within each cell type. These latter spatial parameters allow cell types to up- or down-regulate genes in each spot, enabling BayesTME to capture dynamic phenotypic behavior at spatially localized regions in the TME. This careful separation enables BayesTME to capture the within-phenotype spatial variation of gene expression, a more nuanced signal than currently recoverable by existing methods. Further, the uncertainty quantification provided by posterior inference enables BayesTME to detect significantly varying genes in each cell type with control of the false discovery rate.

Hierarchical priors in BayesTME encode heavy-tailed Bayesian variants^5,2 of the graph-fused group lasso prior³⁹ and the graph trend filtering prior¹¹. The fused lasso prior enforces that the prior probability distribution over cell types follows a piecewise constant spatial function, encoding the biological knowledge that groups of cell phenotypes form spatially contiguous communities. The graph trend filtering prior allows gene expression to vary within cell types, encoding the biological knowledge that cells execute gene sets in a coordinated fashion, known as transcriptional programs. Spatial transcriptional programs extend this concept by identifying and quantifying the activation level of different programs in space. Identification of the BayesTME parameter values is achieved through an empirical Bayes inference algorithm that enables Bayesian quantification of uncertainty over each parameter of interest in the decontaminated data. See the Methods for the detailed hierarchical specification of the generative model and for details on parameter estimation.

BayesTME accurately corrects previously-unreported directional spot bleeding in ST data.

Plots of raw UMI counts in real ST data (Figure 2A–C) show the UMI signal bleeds to background spots with a gradient of intensity. These plots also suggest, unlike the Gaussian assumption in previous preprocessing methods²⁸, or the uniform background noise model in other models¹⁹, bleeding error varies in magnitude in different directions. Such phenomena may be the result of cell-free DNA from dead cells, mRNA binding capacity limitation of spatial barcodes, or technical artifacts of tissue permeabilization.

Figure 2: BayesTME recovers UMI reads from bleeding contamination and preserves the spatial pattern of interest.

A-C. Bleed correction of selected marker genes in two zebrafish melanoma model samples (A and B) and a human dorsolateral prefrontal cortex sample²⁴ (C), with comparison to SpotClean. Bleeding patterns consistently show directional, anisotropic skew towards one corner. SpotClean UMI corrections are therefore expected to be biased towards the tissue boundary whereas BayesTME is more diffuse and better recapitulates the true signal. D. BayesTME performs similarly to SpotClean when the bleeding pattern is isotropic and not skewed (e.g. Gaussian or Student’s $t$); BayesTME substantially outperforms SpotClean when bleeding skews UMIs toward one direction as observed in real tissues. Box plots show median (center line), upper and lower quartiles (box limits), and 1.5x interquartile range (whiskers). The difference between BayesTME and SpotClean is quantified by $p$-value calculated using binomial tests ($n=1000$ for each bleeding scenario). E-G. Examples of simulated bleeding patterns show how BayesTME is able to learn and correct for the direction of the bleeding pattern.

BayesTME corrects bleeding while preserving the true signal. To do this, BayesTME learns a semi-parametric anisotropic bleeding model to correct directional ST bleed and map UMIs to their most likely origin in the tissue. The BayesTME correction only assumes that UMI bleeding decays monotonically as a function of distance. Non-tissue regions are leveraged by BayesTME as a form of negative control, enabling the method to identify the underlying spatial error function from the data via a maximum likelihood estimation procedure.

To evaluate the performance of the BayesTME bleed correction, we constructed synthetic datasets simulating three different bleeding mechanisms: Gaussian, heavy-tailed multivariate-t, and realistic (anisotropic) direction-biased bleeding (Figure 2E–G). The last simulation (Figure 2G) was constructed to resemble real ST data, with bias towards a specific corner of the slide. We compared BayesTME with SpotClean²⁸ (Figure 2D), an existing ST error correction technique that assumes Gaussian technical error. While both methods perform comparably in Gaussian (SpotClean ${\mu }_{MSE}=1170.08$, BayesTME ${\mu }_{MSE}=1263.66$, $p$-value = 0.06) and multivariate-t (SpotClean ${\mu }_{MSE}=1210.06$, BayesTME ${\mu }_{MSE}=1305.31$, $p$-value = 0.69) bleeding scenarios, BayesTME significantly outperformed SpotClean in the realistic bleeding scenario (SpotClean ${\mu }_{MSE}=10437.48$, BayesTME ${\mu }_{MSE}=3048.92$, $p$-value = 1.87×10⁻³⁰¹), where the $p$-values are calculated by a binomial test ($n=1000$ independent trials for each bleeding scenario) to test the statistical significance of deviations.

We found that cell typing and deconvolution were robust to this spatial error. However, bleed correction was critical to preventing genes from falsely registering as spatially varying in real ST data. These results suggest that ST experimental workflows should take care to allow ample non-tissue space in each direction of the slide. If the tissue section exceeds the fiducial markers substantially in a given direction, the technical error function will be statistically unidentifiable. In such cases, it will be impossible to distinguish technical error from true spatial variation, potentially leading to false conclusions when assessing spatially-varying gene expression within-phenotypes.

BayesTME outperforms a suite of existing methods for phenotype inference, spot deconvolution, and tissue segmentation.

We benchmarked BayesTME against other methods: BayesSpace⁴⁶, cell2location¹⁹, DestVI²¹, CARD²², RCTD⁴, STdeconvolve²⁵, stLearn³⁰, and Giotto⁸ on simulated data based on real single-cell RNA sequencing (scRNA-seq) data We randomly sampled $K^{*}$ cell types from a previously-clustered scRNA-seq dataset¹⁹; we conducted experiments for $K^{*}$ from 3 to 8. For each given $K^{*}$, we constructed spatial layouts consisting of 25 cellular communities, defined as spatially-contiguous regions of homogeneous mixtures of cell types. We randomly generated the total cell number for each spot with cellular-community-specific priors. After dividing the total cell number into $K^{*}$ cell types, we randomly sampled cells from the scRNA-seq data of the selected cell types and mapped them on top of the spot pattern from a human melanoma tissue sample⁴¹; see the Methods for details. We compared the performance of BayesTME to the above existing methods on selecting the correct number of cell phenotypes, deconvolving spots, segmenting tissues into spatial communities, and detecting groups of spatially-varying genes within phenotypes. As shown in Figure 3, BayesTME outperformed existing methods across all benchmark tasks.

Figure 3: BayesTME outperforms existing methods in semi-synthetic benchmarks.

A. BayesTME outper-forms the reference-free method STdeconvolve in expression profile inference for each cell type, measured by the coefficient of determination ($r^2$), for semi-synthetic data with ground truth number of cell types $K^{*}=3,4,5,6,7,8$ B. BayesTME outperforms all other methods when segmenting the tissue into cellular communities, measured by adjusted Rand index (ARI). C. BayesTME outperforms existing methods in robustness benchmarks. We measure the performances by the coefficient of determination ($r^2$) of the inferred cell type proportion. Reference-based methods are vulnerable to imperfect scRNA reference as demonstrated by the decline in spot deconvolution accuracy; x-axis: reference contains a subset (< 0), exact match (= 0), or superset (> 0) of the true reference. The existing reference-free method is not reliable in picking the correct number of cell types. BayesTME simultaneously detects the optimal number of cell types from the data and accurately deconvolves the spots. D. BayesTME’s estimation of expression profile compared against the ground truth ($n=10584$), featuring the expression profile of L4 excitatory neurons (red) with a Pearson’s correlation coefficient of 0.98 ($n=882$). E. Both BayesTME and its scRNA-seq guided version (BayesTME*) outperform all other existing methods in the spot deconvolution task on the simulated mouse cortex data, measured by AS (aggregated score from PCC, SSIM, RMSE, and JS).

We conducted an additional segmentation benchmark on the human dorsolateral prefrontal cortex dataset from Maynard et al.²⁴. We clustered the tissue layers and compared them against the pathologist-annotated ground truth using the adjusted Rand index ($ARI$) metric following the procedure from Chidester et al.⁶. We compared BayesTME’s performance with existing tissue segmentation methods. The raw $ARI$ of BayesTME’s segmentation is 0.542, which is comparable to the state-of-the-art tissue segmentation methods (BayesSpace $ARI=0.5552$, SpiceMix $ARI=0.5515$)⁶, where SpiceMix’s result requires ad-hoc refinement to remove extra detail revealed by the HMRF model. BayesTME similarly recovers finer-grained structure; merging the inferred fine structures in the same way as SpiceMix, BayesTME achieves an $ARI$ of 0.572; see Figure S6 for details.

BayesTME accurately identifies the correct number of cell phenotypes and each phenotype expression signature.

A core modeling task in ST analysis is the deconvolution of the spots into their constituent cell phenotype proportions. Most existing methods require a scRNA-seq reference for deconvolution and cell type mapping. As has been noted¹⁹, these methods may be brittle when a cell type is missing from the reference. This vulnerability is particularly problematic in cancer where many subclones may exist and non-overlapping sets of subclones occur between different tissue samples. BayesTME learns the cell phenotypes–both the number of types and their signatures–directly from ST data without the need for scRNA-seq. Thus, BayesTME is robust to the natural spatial heterogeneity of phenotypes in cancer and other disease tissues. To evaluate the robustness and performance of BayesTME, we compared it to both an existing reference-free method and to existing reference-based methods with different degrees of scRNA missingness. To focus purely on the deconvolution and reference-free capabilities of BayesTME, our simulations did not apply any spot bleeding.

There are two tunable hyperparameters in BayesTME: $K$, the number of cell types, and $\lambda$, the global degree of spatial smoothness. BayesTME uses a spatial cross-validation approach to automatically select both variables without the need for user input. The cross-validation procedure creates $m$ non-overlapping folds each with $\kappa \%$ of spots held out; we set $m=5$ and $\kappa =5\%$. For each fold, BayesTME enumerates $K=2,\dots ,K_{\text{max}}$ and $\lambda ={10}^1,\dots ,{10}^6$; in all of our experiments we set $K_{\text{max}}=15$. For each ($K,\lambda$), we fit BayesTME on the in-sample data. Graph smoothing priors enable BayesTME to fill in missing spots during cross-validation. BayesTME uses these imputed posterior estimates to evaluate the likelihood of the held-out data. BayesTME integrates out $\lambda$ in order to select $K$ then chooses the $\lambda$ value closest to the mean held out the likelihood for the chosen $K$; see the Methods for details.

We first evaluated how well the BayesTME recovers the true gene expression profiles of each cell type in each of our $K^{*}$ (true number of cell types) settings. We compared the BayesTME result to STdeconvolve, a reference-free alternative method based on latent Dirichlet allocation³ that provides three different approaches to estimating the number of cell types; we picked the closest estimation out of the three candidates that STdeconvolve provided. Reference-based methods assume access to ground truth cell type information from scRNA annotation, making them unavailable for comparison. In each simulation, BayesTME achieved a higher correlation with the true gene expression levels as measured by the coefficient of determination ($r^2$) of the estimated expression profile (Figure 3A). Further, STdeconvolve over- or underestimated the true number of cell types whereas BayesTME selected the correct number of cell types in each setting (Figure 3C, left).

We next evaluated the robustness of reference-based methods DestVI, CARD, cell2location, and RCTD to reference mismatch. We found that while all methods performed well when the reference was perfectly matched, reference mismatch was problematic for all four reference-based methods (Figure 3C, right). Specifically, DestVI and RCTD were sensitive to the reference being a superset of the true number of cell types (x-axis values 1 and 2) and all four were sensitive to missing cell types (x-axis values −1 and −2). By not relying on any reference scRNA-seq, BayesTME retained high accuracy across all simulations (Figure 3C, left).

Finally, we evaluated the ability of different methods to segment the tissue into spatial regions representing cellular communities. In community detection benchmarks, BayesTME ($ARI=0.99$) surpassed all other currently available alternatives (Figure 3B, Figure S1C), including both spatial clustering (BayesSpace, STLearn, Giotto) and spot deconvolution (cell2location, DestVI, STdeconvolve) methods. For cell2location ($ARI=0.31$), we used its built-in Leiden clustering; when inserting the BayesTME spatial clustering, cell2location improved to $ARI=0.93$, suggesting the BayesTME clustering provides an independent benefit even for accurate deconvolution methods.

Established benchmarks further validate BayesTME’s performance on spot deconvolution and expression signature.

We evaluated BayesTME on recently published ST benchmarks²⁰. The benchmark takes single-cell RNA sequencing data with known spatial coordinates and partitions it into a grid. Each cell in the grid simulates an ST spot. The ST gene count matrix is generated by taking the sum of the expression profile of all the cells in each spot. Following Li et al.²⁰, we measured the performance of BayesTME using the accuracy score (AS), which is an aggregated score of Pearson’s correlation coefficient (PCC), structural similarity index measure (SSIM), root mean square error (RMSE), and Jensen–Shannon divergence (JS). The final score is the normalized average rank of the 4 metrics (with the highest AS score of 1). Li et al.²⁰ provide two benchmark datasets, termed “Dataset 4” and “Dataset 10.” The authors previously found that no method dominates the performance metrics across either task and within each task the top-ranked method for each metric is variable among existing methods.

We tested both BayesTME and a scRNA-reference guided version on this benchmark. We run the scRNA-reference guided version by prespecifying the expression profile in BayesTME, and fixing it to the mean expression of the single-cell reference of each cell type. Both BayesTME and its guided version outperform the existing methods in the Dataset 10 spot deconvolution task (Figure 3E); as anticipated, the scRNA-reference based BayesTME variant slightly outperforms the reference-free version. Additionally, we compared the reference-free BayesTME’s estimation of the expression signatures with the mean expression of single-cell reference of each cell type (Figure 3D). As an example, we show the estimated expression signature of L4 excitatory neurons in comparison to the ground truth. BayesTME’s estimation of expression profile has a Pearson’s correlation coefficient $r=0.98$ for L4 excitatory neurons. For better visualization, we plotted the expressions in the log 10 space. BayesTME’s relative performance on Dataset 4 varied substantially across metrics (Figure S7). BayesTME performed comparably to most methods, with only two methods (SpatialDWLS and Tangram) dominating the results. Overall, the results suggest that BayesTME has dominant spot deconvolution performance on a subset of real data scenarios, but that no method is able to achieve superior performance in all settings.

BayesTME identifies within-phenotype spatial transcriptional programs with tight control of the false discovery rate.

In addition to bleed correction, deconvolution, and cell typing, BayesTME detects gene expression levels of each phenotype that vary in space. To do this, the generative model for BayesTME uses a negative binomial likelihood where spatially-invariant expression levels parameterize the rate and spatially-dependent expression levels parameterize the success rate. Hierarchical spatial shrinkage and clustering priors on the success rate parameters enable BayesTME to discover genes within each phenotype that spatially vary in coordination with other genes. We call these gene sets and spatial patterns spatial transcriptional programs (STPs). The STP construction in BayesTME is flexible: it allows for genes to be negatively spatially correlated within the same program, makes no assumption on the shape or pattern of spatial variation, and adaptively discovers how many genes are in each program. After inference, we use the posterior uncertainty to select STPs with control of the Bayesian false discovery rate (see Methods); we set the FDR target to 5% by default.

To benchmark BayesTME, we constructed a simulation dataset with spatial transcriptional programs by randomly sampling cells from the scRNA data following the same fashion as in the previous experiments. We used the spatial layout from a zebrafish melanoma sample as it is a large tissue containing more than 2000 spots, enabling a rich set of spatial patterns to be imprinted. We chose $K^{*}=3$ cell types and designed 2 spatial programs for each cell type, where 10 genes were randomly sampled and assigned to each of the STPs (Figure 4C). After selecting these 60 spatial genes, we reordered their sampled reads by the spot intensity of their respective spatial programs to simulate the spatial differentiation while preserving the mean expression. Thus, while the gene expression patterns are spatially informative in these simulations, clustering by scRNA-seq analysis would remain unchanged.

Figure 4: BayesTME discovers spatial transcriptional programs with high power and tight control of the false discovery rate.

A. Accuracy of the closest spatial pattern discovered by each method to the ground truth. B. True positive rate (orange) and false discovery rate (gray) for each method when predicting which genes belong to each spatially varying pattern; intervals on bars show 95% confidence intervals; the dashed line is the target (5%) false discovery rate. C. Ground truth spatial patterns used in the benchmark simulations; top: cell type proportion probabilities; bottom: spatial pattern followed by the genes in each spatial program. D. Spatial programs found by BayesTME at the 5% FDR level. E-F. Spatial patterns found by other methods; both SpatialDE and Spark are unable to disentangle phenotype proportions from spatial gene expression within phenotypes.

We benchmarked BayesTME against spatial differential expression methods^38,37 that enable control of the false discovery rate. BayesTME identified all 6 spatial transcriptional programs with on average 0.88 Pearson’s $r$ correlation to the ground truth (Figure 4A,C,D). In contrast, we found SpatialDE and Spark could only detect phenotype proportion patterns instead of meaningful within-phenotype variation in spatial gene expression (Figure 4E,F). We also evaluated the DestVI spatial expression detection mechanism and found the results to be uncorrelated with the ground truth (Figure S2). Quantitatively, BayesTME achieved an average false discovery proportion of 14% where the 95% confidence interval covers the 5% target FDR, and TPR of 94% for selecting spatially varying genes (Figure 4B).

BayesTME discovers spatial programs of immune infiltration and response in human melanoma.

We applied BayesTME to a published human melanoma dataset⁴¹ generated using first-generation ST technology, with a spot diameter of $100\mu m$ and center-to-center distance between spots of $200\mu m$³⁶. The selected sample contained visible tumor, stromal, and lymphoid tissues as annotated by a pathologist based on H&E staining (Figure 5A). Despite the relatively low resolution of the data, the cell types identified by BayesTME successfully recapitulated the histology of the tissue (Figure 5B).

Figure 5: BayesTME discovers spatial programs of immune-tumor interaction in human melanoma.

A. Pathologist-annotated H&E slide; yellow: immune cells, red: stroma, black: tumor. B. BayesTME recovers 4 cell types which map closely to the pathologist annotations. C. BayesTME recovers 5 spatial programs representing fibroblasts, immune cells, and two programs covering tumor subtypes related to transcription (left) and stress responses (right). D. Top marker genes selected by BayesTME to describe each cell type.

Five spatial transcriptional programs were identified by BayesTME (Figure 5C). Two programs were tumor-specific, and displayed somewhat distinct expression patterns, suggesting a spatially-segregated pattern of tumor heterogeneity (Figure 5C). As expected, melanoma marker genes such as PMEL and SOX10 were highly upregulated within the tumor programs (Figure 5D). Similar to the pathologist annotations, the model also detected spatial programs corresponding to stromal (fibroblast) and lymphoid tissues (Figure 5C), which marker genes including COL1A1 (fibroblast-specific, Figure 5C,D) and CXCL13 (lymphoid-specific, Figure 5D). Notably, MYL9 was one of the most highly expressed genes within the fibroblast expression signature (Figure 5D), which is a marker of tumor-associated myofibroblasts³¹, indicating that the fibroblast program identified by BayesTME represents a subpopulation of fibroblasts reprogrammed by their proximity to the tumor. In the fibroblast-related spatial program, immune-related hub genes like IGLL5 and IGJ displayed an enrichment at the tumor boundary (Figure 5C). The model also identified a macrophage-related spatial program (Figure 5C), which had not been detected by the pathologist. One of the top macrophage marker genes, CXCL9 (Figure 5C,D) is a marker of tumor-associated macrophages²³, which have an important role in anti-tumor immunity⁷. Taken together, our results show that BayesTME can successfully not only recapitulate, but also improve the detection of tumor and tumor-associated cell types that are difficult to identify purely by histology.

BayesTME discovers spatial programs capturing muscular bilateral symmetry and tumorimmune interaction in a zebrafish melanoma model.

We expanded upon our human melanoma results by applying BayesTME to our recently published dataset of zebrafish BRAF^V600E-driven melanoma¹⁶, generated using the 10X Genomics Visium technology with approximate spot resolution of $55\mu m$. Both samples contained tumor and TME tissues (muscle, skin) (Figure 6, Figure 7).

Figure 6: BayesTME identifies sharp boundaries and tumor interface programs in a zebrafish melanoma model.

A. Histology of zebrafish sample A; cutout: zoom in on the tumor interface region; bottom: zoom in on the recovered tumor/not-tumor proportions show BayesTME captures the sharp tissue change point. B-C. BayesTME discovers 5 cell types with their proportions and biologically plausible marker genes. D. 9 spatial transcriptional programs discovered at a 5% FDR; muscle programs illustrate BayesTME captures bilateral symmetry without prior knowledge; interface and tumor programs capture differences between interior and exterior tumor behavior.

Figure 7: BayesTME reveals gradual tumor invasion and confirms interface programs in a second zebrafish melanoma model.

A. Histology of the zebrafish B sample; cutout: tumor interface with the gradual invasion of tumor cells into the muscle region; bottom: corresponding tumor-muscle interface with tumor/non-tumor proportions capturing the gradient of tumor invasion present in histology. B-C. BayesTME discovers 10 cell types with their proportions and biologically plausible marker genes. D. 16 spatial transcriptional programs discovered at a 5% FDR.

Within Sample A, BayesTME identified cell types corresponding to tumor, skin, and muscle (Figure 6B, C). Each cell type upregulated expected marker genes, such as myosins and parvalbumins in muscle (myhc4, myl10, pvalb1, pvalb2, pvalb3, pvalb4), BRAF^V600E in tumor, and keratins in skin (krt5, krt91, krt15) (Figure 6C). Two cell types (“Tumor” and “Interface”) were detected within the tumor, both expressing BRAF^V600E (Figure 6A–C). Although the tumor region of Sample A bordered adjacent muscle with little mixing of the two tissue types visible on the H&E-stained section, the interface cell type appeared to infiltrate into the neighboring TME, reminiscent of the interface cell we identified in our recent work¹⁶ (Figure 6A). Many of the interface marker genes were the same as interface marker genes we previously identified, including stmn1a, tubb2b, and hmga1a¹⁶ (Figure 6C). Both spatial programs corresponding to the interface type were enriched at the tumor boundary (Figure 6D). In addition to the interface marker genes we previously identified, BayesTME uncovered a number of genes related to remodeling of the extracellular matrix (ECM) that displayed a spatial enrichment at the tumor boundary, including several collagen-related genes (col1a1a, col1a2, col1a1b; Figure 6D), consistent with a role for the interface cell state in melanoma invasion. Immune genes were also enriched at the tumor-muscle interface, including ilf2 and grn1 (Figure 6C,D).

Sample B contains a wider variety of tissue types, including heart, brain, gills, tumor, and muscle (Figure 7A–C). Mixing of tumor and muscle tissues at the tumor boundary was visible by histology (Figure 7A). Notably, BayesTME again uncovered an “interface” cell state specifically enriched at the tumor boundary (Figure 7A,B). Similar to Sample A, a number of immune-related genes were spatially patterned and/or enriched in the interface region, including lygl1, grn1, cd74a/b, and b2m (Figure 7C,D). Melanoma is a highly immunogenic cancer whose interaction with immune cells in the TME significantly influences tumor progression²⁹. Whether the enrichment of immune genes at the tumor-TME interface represents pro-inflammatory tumor cells at the tumor boundary, or a type of tumor-associated immune cell type will be an exciting topic of future investigation.

In both samples, we uncovered a significant degree of spatially-patterned tumor heterogeneity. BayesTME identified spatial programs characterized by up-regulation of classical melanoma markers such as pmela and tyrp1b (Sample A “Tumor”, Figure 6d) and BRAF^V600E and sox10 (Sample B “Tumor 3”, Figure 7D). Other spatial programs identified in the tumor likely represent other facets of tumor biology. Hypoxia-related genes (hsp70, hif1an, egln3; Figure 6D) were spatially enriched within the tumor region of Sample A, which may indicate hypoxic regions of the tumor due to lack of oxygen supply. Hypoxia has been linked to melanoma progression¹⁵. We also identified spatially-patterned signatures of metabolism, which could represent different metabolic pathways active within the tumor. One of the spatial programs identified within the tumor region of Sample B up-regulated several genes corresponding to ATP synthase subunits (atp5a1, atp5e, atp5b) and other metabolic genes (gpia, tpi1b) (Figure 7D). Determining how different metabolic pathways are spatially organized and regulated within the tumor will be an interesting area of further study. Taken together, our results indicate that BayesTME identifies complex spatial patterns of transcriptional heterogeneity within melanoma and the melanoma microenvironment, and uncovers a potentially pro-inflammatory cell state present at the tumor boundary.

Discussion

This paper has presented BayesTME, a reference-free Bayesian method for end-to-end analysis of spatial transcriptomics data. Compared with existing scRNA-seq referenced methods, BayesTME applies to a wider variety of tissues for which scRNA-seq may not be obtainable due to economic, technical, or biological limitations. Even when references are available, highly heterogeneous and diseased tissues may contain different subsets of cell types between consecutive samples. However, BayesTME is adaptable to scRNA-seq reference if a reliable one is available. With reference data, one can obtain the empirical estimation of the expression signature ϕ, which is invariant to sequencing depth batch effects. Computationally, access to pre-clustered scRNA-seq significantly accelerates the inference by removing the need to perform cross-validation to select the cell phenotypes. Similarly, BayesTME also supports using scRNA-seq reference data from atlases to form informed priors for samples. Our software documentation provides walkthroughs for both use cases.

On the other hand, unlike most reference-free methods, BayesTME does not rely on dimension reduction like PCA. This advantage enables BayesTME to draw individual gene-level inferences, including expression signatures, phenotype markers, and spatial transcriptional programs which current methods miss. Our comparison to 11 other ST data analysis methods highlighted BayesTME’s advance in bleed correction, spot deconvolution, tissue segmentation, and within-cell-type spatial variation in gene expression.

Advances in ST technology promise to soon enhance the resolution to near-single cell levels, dramatically increasing the number of spots. We have carefully designed the computational inference routines in BayesTME to meet this challenge. BayesTME scales sub-linearly with the number of spots, with a 100x increase in the number of spots leading to only a 10x increase in computational runtime (Figure S3A). To further speed up inference, one can place an informative prior on the cell count in a given spot using the H&E slide as the reference.

Understanding how cells alter their expression levels as a function of their spatial location in tissue is necessary for a complete characterization of the cellular architecture of the tissue microenvironment. BayesTME captures these expression level changes in the form of spatial transcriptional programs. Our results showed BayesTME is able to capture biologically meaningful spatial programs which hint at cell-cell interaction in tissue microenvironments. To further facilitate our understanding of cell-cell interaction mechanisms, future versions of BayesTME will introduce an additional cell-type interaction term in the success rate formulation in our negative binomial model. This interaction term will model the total influence of cell type $k$ in spot $i$ as the sum of the interactions between cell type $k$ and all possible cell types $k\prime$. We also plan to explore extending this formulation to all spots within a reasonable neighboring of spot $i$ for global interactions triggered by paracrine, synaptic, or endocrine signaling. This process is computationally expensive under the current ST technology. However, with single-cell resolution, such inference becomes tractable as we only need to look at the individual cells of different cell types within the reasonable neighborhood of cell $i$. Increased ST resolution will significantly drop the computation cost by a factor of $K$, which can also be vectorized to further speed up this process. Thus, BayesTME is well-positioned to make future computational advances in ST modeling, in step with the coming technological advances in ST methods.

STAR Methods

RESOURCE AVAILABILITY

Lead contact

Further information and requests for resources and reagents should be directed to and will be fulfilled by the lead contact, Wesley Tansey (tanseyw@mskcc.org).

Materials availability

This study did not generate new unique reagents.

Data and code availability

All data used in this manuscript is publicly available through their associated original publications. The human melanoma sample is available at https://www.spatialresearch.org/resources-published-datasets/doi-10-1158-0008-5472-can-18-0747/. The zebrafish melanoma data is available at https://doi.org/10.5281/zenodo.5512629. The human dorsolateral prefrontal cortex data is available from the Globus endpoint ‘jhpce#HumanPilot10x’, also listed at http://research.libd.org/globus. Raw data used in each simulation is available for download through the BayesTME repository page (see Code availability). All original code has been deposited at Github (https://github.com/tansey-lab/bayestme) and is publicly available as of the date of publication. DOIs are listed in the key resources table.

Key resources table

REAGENT or RESOURCE	SOURCE	IDENTIFIER
Deposited data
Human melanoma dataset	Thrane et al.	https://doi.org/10.1158/0008-5472.CAN-18-0747
Zebrafish melanoma dataset	Hunter et al.	GEO: GSE159709
Human dorsolateral prefrontal cortex dataset	Maynard et al.	Globus: jhpce#HumanPilot10x
Spatial and single-cell transcriptomics integrated benchmark dataset	Li et al.	https://doi.org/10.1038/s41592-022-01480-9
Synthetic benchmark	This study	https://doi.org/10.5281/zenodo.7986824
Software and algorithms
Python	https://www.python.org	N/A
Polya-Gamma	https://github.com/zoj613/polyagamma	N/A
matplotlib	https://matplotlib.org	N/A
scikit-learn	https://scikit-learn.org	N/A
BayesTME	This study	https://doi.org/10.5281/zenodo.7986824

Any additional information required to reanalyze the data reported in this paper is available from the lead contact upon request.

METHOD DETAILS

Data generation

Semi-syntheic data for spot deconvolution

We took the mouse brain scRNA-seq data from cell2location¹⁹ as the single-cell reference and picked $K=\mathrm{3,4},\mathrm{5,6},\mathrm{7,8}$ cell types at random. We used the spatial layout of the zebrafish A1 sample¹⁶ as the template and partitioned the tissue template into 14 blocks in order to simulate the cellular communities. We drew the cell type proportions for each cellular community (i.e. block) from a Dirichlet distribution with concentration parameter $\alpha$ equal to $1/K$, where $K$ is the number of cell types. This concentration induces sparsity in the subset of cell types in each spot, which replicates real tissues where cell types are often present only in a subset of the tissue. We then sampled the blockwise mean total cell count per spot ${\lambda }_i$ from Pois(30), and drew the total cell number for each spot from $\mathit{\text{Pois}}\left({\lambda }_i\right)$. With the total cell count and the cell type proportion, we sampled the cell count for each cell type from a multinomial distribution. Given the cell number for each cell type, we randomly sampled the cells from the single-cell reference and assigned them to the corresponding spots. We can obtain the simulated UMI count matrix by taking the sum of expression profiles of all cells in each spot.

Semi-synthetic data for robustness benchmark

After generating the semi-synthetic datasets for spot deconvolution following the aforementioned process, we used them for both the spot deconvolution benchmark and the test of robustness benchmark. In the robustness benchmark, we pass imperfect single-cell references to the reference-based methods in order to test their performance in scenarios where we don’t know what the exact cell types in the biological system are. Specifically, for each dataset with $K=\mathrm{3,4},\mathrm{5,6},\mathrm{7,8}$ cell types. We generated five single-cell references with the adjusted number of cell types $K\prime =\{K-2,K-1,K,K+1,K+3\}$. For the two single-cell references with missing cell types, we subsampled $K-2$ and $K-1$ cell types from the $K$ selected cell types, where the first $K-2$ cell types in these two single-cell references are the same. For the two single-cell references with extra cell types, despite the selected $K$ cell types, we sampled 2 unselected cell types and appended the corresponding single-cell expressions to the perfect single-cell reference. For each $K$, we passed five copies of the semi-synthetic datasets, each coupled with one of the five adjusted single-cell references, to the reference-based methods to test their performances.

Mouse cortex semi-synthetic data

We adopted Data 10 and Data 4 mouse cortex data from Li et al.²⁰. The data contains single-cell expression with the spatial coordinate of each cell. The ST slide is partitioned into grids with 750-pixel distance, resulting in squares with sides of 750 pixels. Each square simulates an ST spot. By taking the sum of all single-cell expressions in each square, we can obtain the simulated UMI count matrix. Detail refers to the method for constructing simulated datasets used in Li et al.²⁰. We also generated two copies of Data 10 with an increased resolution by changing the grid. Instead of the original 750-pixel window, we used 500-pixel and 250-pixel windows to generate the simulated ST data with 2× and 9× resolution

Semi-syntheic data with STP

To generate the semi-synthetic data for the spatial transcriptional program test, we first use the procedure described in “Semi-synthetic data for spot deconvolution” to generate the base dataset with $K=3$. In the base dataset, we have all the single-cell expressions in each spot, but did not take the sum. We also generated 6 simulated spatial transcriptional programs (STPs) as shown in Figure 4. For each cell type, we randomly assigned 2 spatial patterns and randomly selected 5 genes for each spatial pattern. In order to ensure the spatial differential expression is realistic, for each gene-celltype pair, we take the sampled single-cell expression of the selected gene in the selected cell type and permute the expression of the selected gene based on the intensity of the simulated STP, while holding the expression of all the other genes fixed. In this way, we can enforce the spatial differential expression pattern without making any artificial expression signal. Repeat the permutation process for each selected gene-celltype pair and each STP. We generated the simulated dataset with STP. We obtain the corresponding UMI count matrix by taking the sum of all single-cell expressions in each spot.

Bleed correction benchmark simulations

The benchmark bleeding simulations use a 70×70 simulated ST tissue block, roughly the size of a common real ST tissue. We ensure that the tissue region has a 10-spot margin on all sides, capturing the idea that tissue segments should be inscribed well inside of the fiducial markers. To further mimic real tissue, we randomly insert a tissue gap inside the tissue region, since most tissues are not perfectly contiguous.

We set the baseline expression of each gene to be a constant drawn from a gamma distribution with shape 2 and rate 100. We then sample reads for each in-tissue spot from a Poisson distribution with the corresponding rate. We then apply a stochastic bleeding process to corrupt the true UMIs.

We benchmark the decontaminating ability of BayesTME and SpotClean under three different bleeding patterns:

1. Gaussian bleeding. Original UMI locations are corrupted by adding noise drawn from a 2d Gaussian with covariance matrix

   $${\Sigma }_{\text{Gauss}}=\left[\begin{array}{cc}5& 1\\ 1& 5\end{array}\right].$$

   This induces thin-tailed, symmetric bleeding.

2. Student’s-t bleeding. Original UMI locations are corrupted by adding noise drawn from a 2d Student’s-t distribution with scale matrix,

   $${\Sigma }_{\text{Gosset}}=\left[\begin{array}{cc}20& 3\\ 3& 30\end{array}\right]$$

   and 10 degrees of freedom. This induces heavy-tailed, symmetric bleeding.

3. Anisotropic bleeding. Original UMI locations are corrupted by adding anisotropic noise that mimics bleeding seen in real tissues. A force 2 d vector of (105,52.5) (150% and 75% of the dimensions of the slide, respectively) is added to each UMI location. A tissue friction coefficient of 5 is used to slow down bleeding within the tissue. The bleeding likelihood is proportional to this skewed distance via a Laplace kernel with bandwidth 40. See the code function distance_weights (available on Github) in the bleeding simulation script for exact computational details. This creates heavy-tailed, asymmetric bleeding that more closely resembles bleeding seen in ST data than the other two methods.

Scalability

BayesTME scales efficiently to ultra-high resolution ST data. Next-generation ST technologies promise to deliver $2\mu m$ resolution, resulting in an increase of up to two orders of magnitude over the current number of spots. We have carefully designed BayesTME so that it scales to meet these challenges. Specifically, the main computational step in posterior inference in BayesTME is the discrete spot deconvolution which scales quadratically with the number of possible cells in a given spot. However, as the resolution increases, this number actually decreases. The result is that the quadratic increase in spots is offset by the decrease in the deconvolution burden. Figure S3 shows the relative runtime of BayesTME as we increase the spatial resolution from the current resolution (1x) with a few thousand nodes to high-resolution (121x) with hundreds of thousands of nodes. Despite increasing by two orders of magnitude, BayesTME only requires a 10x increase in computation time, which is tractable for modern compute clusters.

QUANTIFICATION AND STATISTICAL ANALYSIS

Notation and setup

We assume we are given an $N\times G$ matrix $R$ where $R_{ig}$ is the UMI counts for gene $g$ at spot $i$. The spot $i$ is associated with some known location $l\left(i\right)\in {\mathbb{R}}^2$ on the tissue. These locations define a graph $𝒢=(𝒱,ℰ)$ where each vertex is a spot. There is an edge between two vertices if they are within some $ϵ$ distance. We set $ϵ=\sqrt{2}$ such that each non-boundary spot has 4 neighbors for lattice layouts (e.g., Slide-seq) and 6 neighbors for hexagonal layouts (e.g., Visium). We assume that there are $K$ cell phenotypes (hereon simply called cell types) in the sample, each with its own expression profile. We do not assume that $K$ is known nor do we assume that there is side information about different cell types and their expression profiles (i.e., we do not assume access to paired single-cell RNA). We refer to UMI counts and read counts interchangeably, where read counts are understood to mean UMI-filtered reads and not raw, possibly-duplicated reads.

Generative model

BayesTME models several sources of spatial variation in ST data using a single hierarchical probabilistic model,

$$\begin{array}{lll}(\text{Raw},\text{ corrupted reads in spot }i\text{ for gene }g)\hfill & \hfill {\tilde{R}}_g& ~\xi \left(R_g\right)\hfill \\ (\text{Reads in spot }i\text{ for gene }g)\hfill & \hfill R_{ig}& ={\sum }_{k=1}^K{R}_{igk}\hfill \\ (\text{Reads specific to cell type }k)\hfill & \hfill R_{igk}& ~\mathit{\text{NegBinom}}\left({\beta }_k{d}_{ik}{\varphi }_{kg},\sigma \left(w_{ki}^{\left(h_{kg}\right)}{v}_{kg}+c_{kg}\right)\right)\hfill \\ (\text{Expression signature for cell type }k)\hfill & \hfill {\mathit{\varphi }}_k& ~Dir(\alpha )\hfill \\ (\text{mRNA content for cell type }k)\hfill & \hfill {\beta }_k& ~\mathit{\text{Gamma}}(a,b)\hfill \\ (\text{Total }\#\text{ cells in spot }i\text{ out of }{n}_{max}\text{ possible})\hfill & \hfill D_i& ~\mathit{\text{Binom}}\left(n_{max},1-\sigma \left({\psi }_{i0}\right)\right)\hfill \\ (\#\text{ cells of type }k\text{ in spot }i)\hfill & \hfill d_{ik}& ~\mathit{\text{Binom}}\left(n_{ik},\sigma \left({\psi }_{ik}\right)\right),\forall 1<k<K\hfill \\ (\text{Diff in cell type dist between neighbors})\hfill & \hfill {(\Delta \Psi )}_j& ~\mathit{\text{GroupHorseshoe}}(\lambda )\hfill \\ (\text{Spatial transcriptional program membership})\hfill & \hfill h_{kg}& ~Cat\left({\theta }_k\right)\hfill \\ (\text{STP membership prior odds for cell type }k)\hfill & \hfill {\theta }_k& ~Dir(10,1,1,\dots ,1)\hfill \\ (\text{Spatially-invariant dispersion factor})\hfill & \hfill c_{kg}& ~𝒩(0,1)\hfill \\ (\text{STP loading for gene }g\text{ in cell type }k)\hfill & \hfill v_{kg}& ~\mathit{\text{Horseshoe}}+\hfill \\ (\text{Null STP program})\hfill & \hfill w_k^{(0)}& =0\hfill \\ (\text{Spatial pattern of STP }h\text{ for cell type }k)\hfill & \hfill {\left({\Delta }^{(1)}{w}_k^{(h>0)}\right)}_j& ~\mathit{\text{Horseshoe}}+,\hfill \end{array}$$ (1)

where $\sigma$ is the logistic function, and $\lambda$ is the hyperparameter that controls the degree of spatial smoothing. The function $\xi (\cdot )$ is a nonparametric function defining the spot bleeding process that probabilistically maps from the true read counts ${\mathbf{R}}_g$ for each gene $g$ to the observed counts ${\tilde{\mathbf{R}}}_g$. We specify no functional form for this function and only constrain it to be decreasing in the distance from the true to the observed spot location. We estimate the cell numbers of each cell type by a graph-fused binomial tree model (GFBT) in a cascade fashion (see Figure S5 for details). For an arbitrary spot $i$, we first sample the total number of cells $D_i$, and then sample the cell numbers $d_{ik}$ one cell type at a time from a Binomial distribution with the trial number $n_{ik}$, where

$$n_{ik}=\{\begin{array}{ll}{D}_i\hfill & k=0\hfill \\ n_{ik-1}-d_{ik}\hfill & k\ge 1\hfill \end{array}$$

and the success rate is controlled by the spatially smoothed cell type probability term $\Psi$ enforcing the spatial information in ST data. The matrix $\Delta$ is the edge-oriented adjacency matrix encoding the spot graph $𝒢$, also equivalent to the root of the graph Laplacian; ${\Delta }^{\left(1\right)}={\Delta }^T\Delta$ is the first-order graph trend filtering matrix^42,45, equivalent to the graph Laplacian.

Since full Bayesian inference in the above model is computationally intractable, we develop an efficient empirical Bayes approach that splits posterior inference into stages. While it is true that any Bayesian model could conceptually be fit through MCMC (e.g. running a naïve Metropolis), doing so in practice is often intractable. Metropolis methods based on random walks are unlikely to mix in any reasonable amount of time for the thousands of parameters in BayesTME. Automated methods, such as HMC, require gradient calculations. The horseshoe priors we employ are asymptotic at zero and not differentiable and the BayesTME model includes discrete variables which cannot be estimated with HMC. Automated Gibbs sampling methods like JAGS require models to either use a conjugate prior or default to inefficient routines like Metropolis sampling. Many of the priors in our model are non-conjugate and thus we had to develop efficient latent variable data augmentation schemes for our Gibbs samplers. Similarly, the spatial priors we use require inverting large covariance matrices at each sample, requiring efficient sparse linear algebra routines. The discrete cell counts in each spot are also all dependent, leading to a challenging problem of sampling over a discrete multivariate posterior; this is one of the key computational challenges in estimating the BayesTME model and it required a filter forward-backward sampling (FFBS) algorithm that we derived. Finally, the non-parametric bleeding prior would effectively require a large constrained Gaussian process-esque likelihood; our group has developed the state-of-the-art sampler for such constrained GPs⁴⁰ and even then it would be much too compute-intensive to try to integrate such a sampler into the BayesTME posterior sampler. Our empirical Bayes approach makes what we consider to be minimal approximations necessary to achieve computational feasibility. This piecewise approach to fitting is distinguished from the ad hoc pipeline approach of existing workflows in that a single, coherent generative model is driving the estimation. The empirical Bayes approach merely plugs in point estimates for nuisance parameters while providing full Bayesian inference with uncertainty quantification for the latent variables of interest.

Gene selection

BayesTME scales linearly with the size of the gene library. To keep posterior inference computationally tractable, we select the top $G=2000$ genes ordered by spatial variation in log space. Specifically, we transform the reads as $\text{log}(1+R)$ and rank each column by the variance, keeping the top 2000. The logarithmic transform separates spatial variation from natural variation that arises due to simply having a higher overall expression rate. We then drop all ribosomal genes (i.e., those matching an ‘rp’ regular expression). After selecting and filtering the top genes, we work directly with the UMI read counts.

Anisotropic bleed correction

Technical error causes UMIs to bleed out from barcoded spots. BayesTME models this bleed as a combination of unknown global and local effects. Global effects form a baseline bleed count for any spot, corresponding to a homogeneous diffusion process. Local effects imply that the UMI count at a given spot is a function of how far it is from the original location of each of the UMIs. BayesTME employs a semi-parametric, anisotropic model for global and local effects,

$$\begin{gathered} {\tilde{R}}_g~\mathit{\text{Mult}}\left(\sum _i{\tilde{R}}_{ig},{\mathit{\rho }}_g/\sum _i{\rho }_{ig}\right) \\ {\rho }_{ig}={\rho }_{0g}+\sum _{i^{\prime }}{w}_{i{i}^{\prime }}{\mu }_{i^{\prime }g} \\ {w}_{i{i}^{\prime }}=\sum _{b=1}^B\sum _{j=0}^{s_b\left(i,i^{\prime }\right)}\text{log}\left(1+e^{{\zeta }_{bj}}\right), \end{gathered}$$ (2)

where ${\tilde{\mathbf{R}}}_g$ are the raw, observed counts and ${\rho }_{0g}$ are the global effects. The local effects in Equation (2) are modeled using a set of $B$ monotone nonparametric basis functions $\zeta$ that decay as a function of the basis-specific pseudo-distance $s_b$.² BayesTME uses the four cardinal directions (North, South, East, and West) for the basis functions. This choice is based on the observation that UMIs tend to bleed toward one corner. We also observed that bleeding appears to be less extreme in tissue regions than non-tissue regions. Thus, BayesTME distinguishes between in- and out-of-tissue distance by learning four separate basis functions for each region. The distance from an original spot $i\prime$ to its observed spot $i$ is then a summation of the in- and out-of-tissue components of a straight line between the two spots.

The bleeding model is fit by alternating minimization. At each iteration, BayesTME alternates between estimating the basis functions $\hat{\zeta }$ and global rates ${\hat{\rho }}_{0g}$, and estimating the latent true UMI rates ${\hat{\mu }}_{ig}$. After the model is fit, BayesTME replaces the raw reads with the approximate maximum likelihood estimate of read counts,

$$R_g=\text{arg max }\mathit{\text{Mult}}\left(R_g;\sum _i{\tilde{R}}_{ig},{\hat{\mathit{\rho }}}_g/\sum _i{\hat{\rho }}_{ig}\right)\approx \text{round }\left({\tilde{R}}_{ig}\times {\hat{\mathit{\rho }}}_g/\sum _i{\hat{\rho }}_{ig}\right).$$ (3)

The cleaned reads $R$ are then treated as correct in subsequent inference steps. This can be seen as an empirical Bayes approach, where the model in Equation (2) is optimized and uncertainty over $R$ is replaced with a point estimate that maximizes the marginal likelihood of possible true read configurations. See Figure S4 for examples.

Discrete deconvolution model

The spot-wise gene counts $R_{ig}$ can be decomposed into the sum of cell type-specific gene reads in any given spot, i.e. $R_{ig}={\sum }_{k=1}^K R_{igk}$. BayesTME models the cell type-specific reads with a Poisson distribution controlled by three parameters ${\beta }_k,d_{ik}$ and ${\varphi }_{kg}$. Specifically, ${\beta }_k$ denotes the expected total UMI count of individual cells of type $k;d_{ik}$ denotes the number of cells of type $k$ located in spot $i$; and ${\mathit{\varphi }}_k=\left({\varphi }_{k1},\dots ,{\varphi }_{kG}\right)$ denotes the gene expression profile of cell type $k$, where each element ${\varphi }_{kg}$ is the normalized expression of gene $g$ in cell type $k$; equivalently, ${\varphi }_{kg}$ is the proportion of UMIs that cell type $k$ allocates to gene $g$. The generative model for BayesTME follows,

$$\begin{gathered} R_{ig}=\sum _{k=1}^K{R}_{igk} \\ {R}_{igk}~\mathit{\text{Pois}}\left({\beta }_k{d}_{ik}{\varphi }_{kg}\right) \\ {\mathit{\varphi }}_k~Dir\left(\alpha \right) \\ {\beta }_k~\mathit{\text{Gamma}}\left(a,b\right) \\ {D}_i~\mathit{\text{Binom}}\left(n_{max},1-\sigma \left({\psi }_{i0}\right)\right) \\ {d}_{ik}~\mathit{\text{Binom}}\left(n_{ik},\sigma \left({\psi }_{ik}\right)\right),\forall 1<k<K \\ {(\Delta \Psi )}_j~\mathit{\text{GroupHorseshoe}}\left(\lambda \right) \end{gathered}$$ (4)

where $D_i$ is the total number of cells in spot $i$, and $\lambda$ is the hyperparameter that controls the degree of spatial smoothing. The matrix $\Delta$ is the edge-oriented adjacency matrix encoding the spot graph $𝒢$, also equivalent to the square root of the graph Laplacian. The hierarchical prior encoded by the last three lines of Equation (4) is a heavy-tailed Bayesian variant of the graph-fused group lasso prior^11,39 that uses the Horseshoe + distribution² . This prior encourages the probability distribution over cell type proportions to follow a piecewise constant spatial function, encoding the prior belief that cells form spatially contiguous communities. The model is data-adaptive, however, and able to handle deviations from this prior where warranted in the data; see for example, the smooth gradient of cell type proportions recovered in Figure 7.

Posterior inference

Posterior inference in BayesTME is performed through Gibbs sampling. The key computational innovations in BayesTME come in the form of a fast approach to update $d_{ik}$, the number of cells of type $k$ in spot $i$. Block joint sampling over all ${\mathbf{d}}_i$ and $D_i$ can be done via an efficient forward-backward algorithm (Figure S5). This algorithm effectively converts the cell count prior to a hidden Markov model prior. The Poisson likelihood in Equation (4) acts as the emissions step and the emission log-likelihood can be collapsed into a series of fast updates. This inference step enables us to sample over the entire combinatorial space of possible cell counts in $𝒪\left(N{D}_{\text{max}}^2{K}^2\right)$ time for $N$ spots, $K$ cell types, and $0\le D_i\le D_{\text{max}}$ possible total cells in each spot. BayesTME performs Gibbs sampling using these fast updates with a burn-in and Markov chain thinning; we use 2000 burn-in steps, 5 thinning steps between each sample, and gather a total of $T=100$ post-burn-in posterior samples.

Selecting the number of cell types and smoothness hyperparameters

BayesTME automatically chooses the number of cell types $K$ via $M$-fold cross-validation. For each fold, a random non-overlapping subset of the spots is held out; we use $M=5$ folds with 5% of spots held out in each fold. The spatial priors in BayesTME enable the imputation of the cell type probabilities at each held-out spot in the training data. For each fold, we fit over a discrete grid of $\lambda$ smoothness values; we use $\lambda =\left({10}^0,{10}^1,\dots ,{10}^6\right)$. For a given fold $m$, cell type count $K$, and smoothness level $\lambda$, we calculate an approximate bound on the marginal log-likelihood of the held-out spots using $T$ posterior samples,

$${ℒ}^{\text{test}}=\sum _{i\in {\text{fold}}_m}\sum _{t=1}^T\log \mathit{\text{Mult}}\left(R_i;\sum _g{R}_{ig},\frac{{\sum }_k{\beta }_k^{\left(t\right)}{\theta }_{ik}^{\left(t\right)}{\varphi }_k^{\left(t\right)}}{{\sum }_g{\sum }_k{\beta }_k^{\left(t\right)}{\theta }_{ik}^{\left(t\right)}{\varphi }_{kg}^{\left(t\right)}}\right).$$ (5)

Results are averaged over all $\lambda$ values for each fold and then averaged across each fold. The $\lambda$ averaging is an empirical Bayes estimate with a discrete prior on $\lambda$ integrated out; the cross-validation averaging is an unbiased approach to selecting $K$. After selecting $K$, we refit BayesTME on the entire data using the chosen $K$ and the $\lambda$ with average cross-validation log-likelihood closest to the overall average.

For selecting the correct number of cell types, we constructed 4 semi-synthetic ST data with the same tissue template but different number of cell types $\left(K^{*}=\mathrm{3,4},\mathrm{6,8}\right)$. For each semi-synthetic data, we ran cross-validation with the number of cell types from 2 to 15 and picked the one with maximum likelihood as our predicted cell type number. The predicted cell type number matches the ground truth $K^{*}$ in all trials. Such Observation suggests, with cross-validation, BayesTME is able to select the optimal number of cell types from the ST data without scRNA-seq reference.

Figure S3.b shows the average cross-validation log-likelihood for each of the four simulations. Each simulation used a different true number of cell types and BayesTME correctly identified the true number for each simulation. As the number of cell types increased, variance in the held-out likelihood also increased. Thus, if the number selected is beyond $\approx 10$ cell types, we recommend increasing the number of folds $m$ to compensate.

Selecting marker genes

We define a gene as a marker of a particular cell type if its expression in that cell type is significantly higher than in any other cell type. BayesTME uses posterior uncertainty to select statistically significant marker genes with control of the Bayesian false discovery rate (FDR)¹⁰. To calculate the local FDR we use the $T$ posterior samples,

$${\omega }_{kg}=1/T\sum _{t=1}^T\prod _{k^{\prime }\ne k}\mathbb{1}\left[{\varphi }_{kg}^{\left(t\right)}>{\varphi }_{k^{\prime }g}^{\left(t\right)}\right],$$ (6)

yielding the posterior probability that gene $g$ is a marker for cell type $k$. We sort the $\omega$ values in descending order and solve a step-down optimization problem,

$$\begin{array}{ll}\underset{q}{\text{maximize}}\hfill & q\hfill \\ \text{subject to}\hfill & \frac{{\sum }_{i=1}^q\left(1-{\omega }_{\left(i\right)}\right)}{q}\le \alpha .\hfill \end{array}$$ (7)

The set of $\omega$ values selected controls the Bayesian FDR at the $\alpha$ level. BayesTME can alternatively control the Bayesian Type I error rate at the $\alpha$ level by only selecting marker genes satisfying ${\omega }_{kg}\ge 1-\alpha$. We then rank the selected marker gene candidates by $\omega$ and $\xi$ jointly, where

$${\xi }_{kg}=\frac{{\overline{\varphi }}_{kg}-\text{max}{\left\{{\overline{\varphi }}_{k^{\prime }g}\right\}}_{k^{\prime }\ne k}}{\text{max}{\left\{{\overline{\varphi }}_{k^{\prime }g}\right\}}_{k^{\prime }=1}^K},$$ (8)

is the normalized expression score in [−1,1] measuring the expression level of gene $g$ in cell type $k$ compared with all the other cell types, and ${\bar{\varphi }}_{kg}$ is the posterior mean of $T$ posterior samples. By default, we set the FDR threshold to 5%; our results report an interpretable subset of the top 20 genes for each inferred cell type.

Community detection

To segment the tissue into cellular communities, BayesTME clusters the fused spatial probabilities $\mathbf{\Psi }$. First, the neighbor graph is augmented with the nearest 10 neighbors to adjust for spatially-disconnected spots due to tissue tears in sectioning. The posterior samples are flattened into a single vector for each spot. Spots are then clustered using agglomerative clustering with Ward linkage, as implemented in scikit-learn. The number of clusters $q$ is chosen over a grid of $q\in (1,\dots ,50)$ to minimize the sum of the AIC³⁴ and BIC²⁷ scores. Community distributions are calculated as the average of all posterior probabilities of all spots assigned to the community. When comparing community segmentation in benchmarks, we applied BayesTME’s clustering algorithm on DestVI and stDeconvolve, as they do not provide segmentation routines.

Spatial transcriptional program model

The deconvolution model in Equation (4) assumes gene expression is stationary within a given cell type. However, we expect that variation in a small number of important genes should be spatially dependent. BayesTME captures this spatial variation by replacing the Poisson likelihood in Equation (4) with a more complex negative binomial one,

$$\begin{gathered} R_{igk}~\mathit{\text{NegBinom}}\left({\beta }_k{d}_{ik}{\varphi }_{kg},\sigma \left(w_{ki}^{\left(h_{kg}\right)}{v}_{kg}+c_{kg}\right)\right) \\ {h}_{kg}~Cat\left({\theta }_k\right) \\ {\theta }_k~\mathit{\text{Dirichlet}}\left(10,1,1,\dots ,1\right) \\ {c}_{kg}~𝒩\left(0,1\right) \\ {v}_{kg}~\mathit{\text{Horseshoe}}+ \\ {w}_k^{\left(0\right)}=0 \\ {\left({\Delta }^{\left(1\right)}{w}_k^{(h>0)}\right)}_j~\mathit{\text{Horseshoe}}+ \end{gathered}$$ (9)

where $\sigma$ is the logistic function and ${\Delta }^{\left(1\right)}={\Delta }^T\Delta$ is the first-order graph trend filtering matrix, equivalent to the graph Laplacian. The rate in Equation (9) is equivalent to that in the simpler model in Equation (4). In both cases, the expected read count scales additively with the number of cells, a crucial property that reflects the intuition that a spot with twice as many cells should yield twice as many reads.

Gene expression within a cell type varies spatially through the success probability (the second term) in the negative binomial likelihood. The offset term $c_{kg}$ corresponds to the spatially-invariant expression term that controls the dispersion rate in the counts. Each gene $g$ in each cell type $k$ belongs to one of $H$ clusters. Each cluster defines a different spatial pattern ${\mathbf{w}}_k^{\left(h\right)}$, which we refer to as spatial transcriptional programs. The first program ${\mathbf{w}}_k^{\left(0\right)}$ is the null program corresponding to spatially-invariant expression. All subsequent programs are latent and inferred through posterior inference. BayesTME places a heavy prior on genes coming from the null, such that it takes substantial evidence to conclude that a gene is spatially varying within a cell type; this prior is necessary as otherwise the model is only weakly identifiable. Genes participating in the non-null spatial programs do so by placing a weight $v_{kg}$ on the spatial pattern. This weight shrinks, magnifies, or can even invert the pattern, allowing the clustering of negatively correlated genes into the same spatial transcriptional program. BayesTME places a sparsity-inducing prior on $v_{kg}$ in order to encourage only strongly participating genes to be assigned to non-null programs.

Spatial transcriptional program inference

Posterior inference via Gibbs sampling is possible with the STP BayesTME model. However, the fast HMM updates for the cell counts are no longer available, making the inference algorithm substantially slower. For computational efficiency, we instead take a two-stage approach. First, we fit the deconvolution model in Equation (4), collecting $T$ posterior samples of each latent variable. Then we fix $(\beta ,\mathbf{d},\Phi {)}^{\left(t\right)}$ for each sample $t=1,\dots ,T$. For each fixed sample, we run a Gibbs sampler for the non-fixed variables in Equation (9); we use 99 burn-in iterations and take the 100^th iteration as the sample for the $t^{\text{th}}$ iteration of the full model parameters. We motivate this approach mathematically by the identity that if $Y~\mathit{\text{Pois}}\left(r\right)$ and $X~\mathit{\text{NegBinom}}(r,p)$, then $\mathbb{E}\left[Y\right]=\mathbb{E}[X\mid p=0.5]$. Since we put sparsity priors on $v_{kg}$ and a standard normal prior on $c_{kg}$, all of our priors peaked at $p=0.5$. Thus, a priori, we expect the posterior mean under the full joint inference model to be nearly the same as the two-stage model; in practice, we find the two approaches produce similar results.

Selecting significant spatial transcriptional programs

Spatial transcription programs in BayesTME correspond to spatial patterns in $w_k^{\left(h\right)}$ in cell type $k$, and the members of a spatial program are the genes $g$ for which $h_{kg}$ is significantly non-null. Spatial programs are only considered active in spots $i$ where $d_{ik}>0$ with high probability. Specifically, for a given $\alpha$ significance level, we select spots and genes for the spatial program $s$ in cell type $k$ as follows,

$$\begin{gathered} {𝒮}_{sk}^{\text{genes}}\left(\alpha \right)=\left\{v_{kg}:\left(1/T\sum _{t=1}^T\mathbb{1}\left[h_{kg}^{\left(t\right)}=s\right]\right)\ge 1-\alpha \right\} \\ {𝒮}_{sk}^{\text{spots}}\left(\alpha \right)=\left\{w_{ki}^{\left(s\right)}:\left(1/T\sum _{t=1}^T\mathbb{1}\left[d_{ik}^{\left(t\right)}>0\right]\right)\ge 1-\alpha \right\}. \end{gathered}$$ (10)

If either ${𝒮}_{sk}^{\text{genes}}\left(\alpha \right)$ or ${𝒮}_{sk}^{\text{spots}}\left(\alpha \right)$ is empty, we filter out the entire program. We also filter programs where the Pearson correlation between $w_k^{\left(h\right)}$ and $d_k$ is more than 0.5 and Moran’s $I$ spatial autocorrelation less than 0.9; these programs capture technical noise and overdispersion rather than meaningful spatial signal. In practice, we find $H=10$ to be a sufficient number of potential spatial programs per cell type. BayesTME sets the spatial transcriptional program significant threshold to $\alpha =0.95$.

Figure 1: The BayesTME computational workflow and outputs.

BayesTME first corrects technical errors (spot bleeding) in the raw ST data by probabilistically mapping reads to their most likely original location in the tissue. The reference-free spot deconvolution is run to simultaneously recover the cell phenotypes and their counts at each spot. Finally, the deconvolution model is augmented with a spatially-adaptive phenotype model to infer phenotype-specific spatial variations. The final output of the complete BayesTME pipeline is the inferred cell phenotype expression signatures, the top marker genes that maximally distinguish phenotypes, the posterior distribution over the cell type probability and discrete cell counts of each type in each spot, the segmented tissue partitioned into cellular communities, and the spatial transcriptional programs discovered for each phenotype.

Highlights:

• BayesTME is a method for comprehensive analysis of spatial transcriptomics

• It achieves state-of-the-art performance in spot deconvolution without scRNA reference

• A dynamic process model reveals phenotype-specific spatial transcriptional programs