ReliST: A model-agnostic risk layer for spatial transcriptomics deconvolution

Xinyu Zhang; Li He; Yu Peng; Yijia Li; Jianyuan Kang; Lisheng Peng; Yifei Xu; Sen Lin

doi:10.1016/j.isci.2026.117206

. 2026 Aug 14;29(9):117206. doi: 10.1016/j.isci.2026.117206

ReliST: A model-agnostic risk layer for spatial transcriptomics deconvolution

Xinyu Zhang ¹, Li He ², Yu Peng ³, Yijia Li ⁴, Jianyuan Kang ¹, Lisheng Peng ⁴, Yifei Xu ^1,^∗, Sen Lin ^2,^5,^∗∗

PMCID: PMC13503129 PMID: 42643167

Summary

Spatial transcriptomics deconvolution maps cell-type abundance across tissue, but most outputs do not indicate which local predictions can be trusted. We developed ReliST, a model-agnostic risk layer that preserves base predictions while assigning spot-level risk from output ambiguity, local inconsistency, and reference-related evidence. We evaluated ReliST with five deconvolution models across human dorsolateral prefrontal cortex (DLPFC), mouse brain, human breast cancer, and an independent immunofluorescence/gene-protein dataset. In DLPFC, risk scores aligned with layer difficulty, marker discordance, and signature residuals. In mouse brain, ReliST provided reference-control and review signals without layer labels. In breast cancer, high-risk regions aligned with histology context, stromal and vascular shifts, immune-associated protein evidence, and abundance-adjusted protein residuals consistent with protein-supported under-calls. ReliST extends spatial deconvolution from prediction-only maps to risk-aware interpretation, supporting filtering, abstention, and targeted review rather than universal model ranking or spot-level truth assignment.

Keywords: ReliST, spatial transcriptomics, deconvolution, risk-aware deconvolution, reliability, selective prediction, risk-bin diagnostics, reference mismatch, artifact-aware interpretation

Graphical abstract

Highlights

•
Risk estimates flag tissue locations with less reliable cell-type predictions
•
Risk scores track prediction error using data from different donors
•
Removing high-risk locations lowers error across three prediction models
•
Combined diagnostics help prioritize tissue locations for expert review

Bioinformatics; Computational bioinformatics; Transcriptomics

Introduction

Spatial transcriptomics measures gene expression in tissue context and enables expression patterns to be related to anatomy, local neighborhoods, and disease microenvironments. In sequencing-based platforms such as 10× Visium, each spot usually captures multiple cells, so the raw expression matrix cannot be interpreted directly as a single-cell composition.¹^,²^,³^,⁴^,⁵ Cell-type deconvolution has therefore become a core step in spatial transcriptomics analysis: single-cell or single-nucleus RNA-seq references are used to estimate cell-type abundances in each spot, and the resulting maps are used to interpret cortical layers, anatomical regions, tumor microenvironments, and disease-associated niches.⁶^,⁷^,⁸

Many spatial deconvolution methods have been developed in recent years, including probabilistic mixture models, spatially informed models, marker-driven regression, deep generative models, and cross-modal mapping approaches. Cell2location, Robust Cell Type Decomposition (RCTD), Tangram, Stereoscope, and DestVI represent major families of these methods, while SPOTlight and CARD illustrate additional marker-driven and spatially informed deconvolution strategies.⁹^,¹⁰^,¹¹^,¹²^,¹³^,¹⁴^,¹⁵ Existing benchmarks mainly evaluate average prediction accuracy, abundance recovery, platform adaptation, and cross-dataset performance.⁷^,⁸ These studies address which method performs better on average, but they do not fully answer a practical question faced by users: after a deconvolution map is generated, which local regions can be interpreted with confidence, and which regions should be reviewed, down-weighted, or withheld from downstream analysis?

The average-performance view used in most benchmarks is essential for method development, but it is incomplete for practical spatial analysis. Downstream analyses often use deconvolution outputs as spatial covariates for niche discovery, spatial integration, tissue-domain refinement, interaction analysis, and disease-region comparison, among other tasks.¹⁶^,¹⁷^,¹⁸^,¹⁹^,²⁰ In these settings, a small number of locally unreliable spots can have disproportionate influence, because they may define an apparent boundary, local enrichment, or tumor-associated compartment. A method-level average score therefore cannot substitute for a spot-level readout of where the prediction is reliable enough for downstream use and where it should be treated as provisional.

This question is particularly important because many spatial transcriptomics datasets lack spot-level ground truth.⁷^,⁸ Model failures in real tissues do not necessarily appear as a global performance collapse. They may instead concentrate in tissue edges, low-quality regions, spot-swapping or local technical artifacts, reference-coverage gaps, continuous cell-state transitions, highly mixed spots, or complex stromal regions in tumors.³^,²¹^,²²^,²³^,²⁴^,²⁵^,²⁶ A base model can produce plausible predictions across most spots while generating unstable or over-interpreted outputs in a small subset of locations. If downstream analysis uses the full cell-type map without a local reliability layer, these high-risk regions may be mistaken for robust biological findings.

Native uncertainty outputs only partly address this gap. Some methods expose posterior-like quantities, some expose mapping scores or reconstruction signals, and others provide no directly comparable uncertainty field.⁹^,¹⁰^,¹¹^,¹²^,¹³^,¹⁴^,¹⁵ Even when such a signal is available, it may capture model-internal ambiguity while missing reference mismatch, spatial contradiction, histology artifacts, or local cell-state mismatch. A model-agnostic risk layer therefore has to place uncertainty-like signals inside a broader evidence structure.

We address this gap by formulating spatial deconvolution as a prediction-plus-risk problem. Let g(x) $\to$ y_hat denote an arbitrary base deconvolution model, where x is the spatial expression input and y_hat is the predicted cell-type abundance or related representation. ReliST adds a second layer, f(x, y_hat) $\to$ r, where r is a spot-level risk score. Higher scores consistently indicate lower trustworthiness. The score serves as a localized failure-risk readout for identifying predictions that require caution.

The main difficulty is that spot-level truth is usually unavailable. ReliST therefore treats the score as model violation risk rather than observed error: the risk that a model cannot provide a stable, interpretable, externally supported prediction at a given spot. This distinction keeps the claim boundary clear. Error would imply comparison to a known spot-level truth label, whereas risk describes a decision condition in which the output is numerically available but weakly supported, contradictory, or contextually confounded. We therefore evaluate risk-proxy association, selective gain, coverage-risk curves, and reliability curves, asking whether retained low-risk spots show less proxy conflict and a stronger basis for downstream interpretation.

ReliST is related to uncertainty quantification, conformal prediction, and selective prediction, but the target is different. Standard uncertainty quantification often focuses on posterior uncertainty, predictive intervals, ensemble disagreement, or calibrated coverage.²⁷^,²⁸ Conformal prediction provides distribution-free coverage guarantees under defined assumptions, and selective prediction evaluates performance after rejecting part of the sample set.²⁹^,³⁰^,³¹^,³² ReliST instead targets a spatial deconvolution-specific usage problem: in a reference-dependent, ground-truth-limited setting, a model may produce apparently definite outputs while still failing locally because of reference mismatch, spatial inconsistency, or histology artifacts.

ReliST is organized as a model-interface-agnostic, multi-axis post hoc audit framework, which separates it from generic confidence feature engineering. Reference-related, marker-sensitive, and structure-related diagnostics are treated as failure-mode axes whose utility can vary by base model, tissue context, and available validation evidence. Reference mismatch and reference-coverage failure remain important sources of deconvolution risk, but the current known-composition benchmark shows that the reference component should be interpreted as optional axis-specific evidence rather than as a mandatory contributor to the default scalar score. Model-dependent support tiers are reported under this output contract as observable support rather than intrinsic model quality.

We evaluate ReliST in three complementary settings. Human DLPFC provides strong structural anchors and is used as the primary anchored validation case. Mouse brain lacks DLPFC-like layer annotation but has an Allen reference and histology-derived auxiliary evidence, making it useful for unlabeled deployment and reference control. Human breast cancer provides a complex-tissue setting with working annotation, tumor heterogeneity, and H&E artifact confounding. The dataset contexts, validation roles, and supported claim boundaries for these three validation settings are summarized in Table 1. An independent breast cancer immunofluorescence (IF)/gene-protein dataset is used to strengthen the orthogonal consistency and protein-supported under-call evidence chain.

Table 1.

Dataset context and claim boundaries

Dataset	Validation role	Biological complexity	Histology anchor	Main claim supported
Human DLPFC	Primary anchored validation	Low	Strong structural organization	Proxy-risk alignment
Mouse brain	Generalization control	Moderate	Moderate auxiliary anchor	Reference robustness
Human breast cancer	Complex tissue stress test	High	Weak-to-moderate artifact-aware anchor	Artifact-aware validation

Open in a new tab

Summary of validation role, biological complexity, histology anchor, and supported claim boundary for DLPFC, mouse brain, and human breast cancer. The table defines how the three datasets support the reliability and risk claim.

The contribution is 3-fold. First, ReliST defines a model-agnostic, multi-axis risk-audit layer for existing deconvolution outputs rather than a competing abundance estimator, connecting heterogeneous base models through a shared output contract. Second, it evaluates risk as a decision mechanism using selective gain, coverage-risk behavior, reliability curves, controlled null comparisons, and benchmarks against common confidence-like summaries such as abundance entropy, top-1 margin, maximum abundance, and shared local inconsistency. Third, it supports a reporting standard for spatial deconvolution: a complete output should include both a cell-type map and a risk map that guides filtering, abstention, and review prioritization while preserving uncertainty about which failure-mode axes are supported in a given setting.

Results

Because ReliST is designed as a companion layer rather than a replacement deconvolution model, we evaluate its contribution as added local risk information on top of representative base-model outputs. The key comparisons therefore ask whether ReliST produces supported risk axes improves retained proxy quality after filtering, and outperforms deliberately strong common-feature null controls.

ReliST expands deconvolution outputs from prediction-only maps to prediction-plus-risk maps

The ReliST framework starts from a base deconvolution prediction, y_hat, and uses the spatial expression input together with y_hat to estimate spot-level risk, r, with higher values indicating lower trustworthiness (Figure 1). The risk signals are organized as output-ambiguity, local-inconsistency, reference-related, marker-sensitive, and structure-related diagnostics, with model-dependent support tiers rather than a fixed universal component hierarchy. Validation evidence includes risk-proxy association, selective gain, coverage-risk curves, and reliability curves. Risk-aware use includes trusted spots, review-needed spots, abstention, and artifact-aware interpretation.

ReliST model-agnostic risk-layer framework

A base deconvolution model maps spatial transcriptomics input to a prediction, ŷ, and ReliST assigns each spot a risk score, r, with higher r indicating lower trustworthiness. The risk layer can use enabled reference or marker-signature sources and optional runtime covariates, and organizes evidence into a core reference axis, an optional reference-marker auxiliary branch, a context-dependent structure axis, and model-dependent support tiers under a shared output contract. Risk-aware outputs combine prediction maps, risk maps, and decision states, with external evidence used only for validation and downstream interpretation.

The framework also marks an important boundary. External labels, H&E features, and dataset context are validation or interpretation evidence, not required runtime inputs to the risk estimator. This allows DLPFC layer annotations to be used for validation, mouse brain to be evaluated without layer labels, and breast cancer to be interpreted in an artifact-aware manner without converting all three settings into the same supervised task.

This schematic also defines how the remaining results should be read: a prediction map shows what a deconvolution model estimates, a risk map identifies where that estimate needs caution, and external evidence asks whether those warnings are supported by independent structure, marker, histology, or protein signals (Figure 1). This three-part readout keeps biological interpretation tied to the strength of the supporting evidence.

DLPFC supports risk scores as proxy-anchored diagnostic signals

Human DLPFC is the strongest anchored validation setting in this study. The primary DLPFC analysis includes 47,681 spots. Cortical layer annotations and marker-related proxies provide external signals for testing whether risk scores are enriched in regions that are harder to interpret.³³

Axis-specific evidence maps from different base models showed interpretable spatial patterns rather than random distributions (Figure 2A). Cell2location and RCTD provide stronger reference-related support under the current output contract. Tangram, Stereoscope, and DestVI are better interpreted as marker-auxiliary, structure-sensitive, or limited-support signals in these analyses. Validation proxy maps for layer prototype distance, Maynard prototype distance, marker discordance, and signature residual across all five models are shown in Figure S1.

DLPFC anchored validation

(A) Cross-model axis-specific evidence maps for DLPFC sample 151508; color indicates the within-model evidence percentile.

(B) Coverage-risk curves show the retained mean validation-proxy burden, expressed as a ratio to the corresponding full-data mean, across retained tissue coverage. Values below 1 indicate reduced proxy conflict after high-risk spots are filtered.

(C) Reliability curves show the mean validation-proxy burden within each risk quintile, expressed as a ratio to the corresponding full-data mean; the horizontal reference is 1. Lines connect model-specific summaries and no error bars are shown. More complete DLPFC validation-proxy maps and metric summaries are provided in Figures S1 and S2.

Quantitative results support this spatial pattern. Performance was quantified using the area under the receiver operating characteristic curve (AUC). In the anchored validation summary, RCTD achieved mean best AUC values of approximately 0.87 for Maynard prototype distance and 0.88 for layer prototype distance, with 95% CIs of 0.77–0.92 and 0.82–0.92, respectively, and retained strong evidence for signature residual. Stereoscope achieved mean best AUC values of approximately 0.94 for both Maynard and layer prototype proxies and approximately 0.95 for marker discordance. Tangram was strong for marker discordance, with mean best AUC around 0.84, but weaker for signature residual. DestVI showed some marker-discordance signal but did not support the same reference-related diagnostic interpretation in these experiments.

Selective use and risk-bin diagnostics moved beyond correlation (Figures 2B and 2C). The coverage-risk curve shows that as high-risk spots are filtered, proxy conflict in the retained subset decreases. The reliability curve shows that multiple validation proxies increase across risk bins. For RCTD, the difference between the highest and lowest risk quintiles is positive for layer_guess, Maynard, marker_discordance, and signature_residual proxies. Retained proxy ratios, risk-proxy AUCs, high-risk versus low-risk proxy enrichment, and bootstrap confidence intervals are summarized in Figure S2 and Table S5.

Together, these results support a clear conclusion for DLPFC: in a tissue with strong structural anchors, ReliST identifies local regions that require caution and improves the proxy quality of retained predictions through risk filtering. The DLPFC results support the risk score as a proxy-anchored diagnostic signal and a selective interpretation tool.

The DLPFC results also show why the risk layer is more than a visual overlay. If a risk map only highlighted plausible anatomical regions, it would remain a descriptive heatmap. Coverage-risk and reliability curves instead test whether the map can support a concrete operating decision: how much tissue to retain, how much proxy burden remains after filtering, and whether higher score bins are consistently less trustworthy. This is the evidence that makes risk filtering, abstention, and review-priority labels defensible as downstream workflow elements.

Common-feature controls and goal-based decision use define support tiers

Cross-model results show that base models support ReliST in different ways. Table S1 summarizes cell2location and RCTD as stronger reference-related baselines in the DLPFC output contract, Tangram as a marker-sensitive auxiliary comparator, Stereoscope as supporting structure and marker goals in the DLPFC setting, and DestVI as a limited-support comparator under this output contract. This support-tier structure explains how ReliST can interface with different model behaviors without turning the analysis into a global model ranking.

Table S3 tests whether the risk axes merely repackage output sharpness or local inconsistency. For each model-proxy pair, the strongest common abundance-derived score is selected as the best common feature, giving the null comparison a deliberately strong baseline. For marker discordance, reference-risk AUCs were approximately 0.95 for Stereoscope, 0.90 for Cell2location, 0.84 for Tangram, 0.82 for DestVI, and 0.72 for RCTD; the corresponding best common-feature AUCs were approximately 0.50–0.62. These results show that marker-sensitive and reference-sensitive signals contain information beyond entropy, top-1 margin, maximum abundance, and common local inconsistency. The same control also preserves negative boundaries: for DestVI layer and Maynard structure proxies, common φ_local outperformed the structure-risk score, placing the DestVI structure-axis in a limited-support category.

These differences were organized as model-by-axis support tiers (Figure 3A). Downstream goals were then mapped to default models and axes: RCTD is used for reference and dominant-label stability goals, whereas Stereoscope is used for structure and marker goals in the DLPFC decision bundle (Figure 3B). The empirical default policy for DLPFC samples 151508 and 151670 is summarized in Table S2. These policies are empirically supported operating choices, not universal model rankings.

Decision-support use of risk scores

(A) Model-by-axis support tiers.

(B) Goal-based model-selection policy.

(C) Representative decision-bundle risk map for DLPFC sample 151508.

(D) Trusted-only map after retaining the lowest-risk 80% of spots.

(E) Decision states assigned from risk and cross-model disagreement.

(F) Fraction of all spots assigned to trusted, use-with-caution, and review-needed states; fractions sum to 1.

These figures illustrate operating choices in the DLPFC setting rather than a universal model ranking.

Continuous risk maps can be converted into a decision-support bundle (Figures 3C–3F). In the representative 151508/RCTD/reference example, the within-sample 80th percentile risk score is used as the high-risk cutoff, approximately 0.5919. This retains 3,507 of 4,384 spots as a low-risk retained subset and filters the highest-risk 877 spots. Within the high-risk spots, cross-model disagreement is used to define review priority: high risk plus high disagreement is labeled review needed, high risk with low disagreement is labeled use with caution, and non-high-risk spots are labeled trusted. In this example, 799 high-risk spots are labeled use with caution and 78 are labeled review needed. These labels define an operational review workflow, not a truth threshold stating that the highest-risk 20% of spots are necessarily wrong.

The decision bundle also shows how the risk layer can be matched to downstream goals. A conservative atlas-style analysis may retain only low-risk or trusted spots. A hypothesis-generation analysis may keep use-with-caution spots but visually mark them and exclude them from quantitative claims. A validation-prioritization workflow may send review-needed spots to manual inspection, histology review, or orthogonal assays. ReliST therefore changes model use from a single abundance export to an abundance export with explicit decision states.

Mouse brain demonstrates reference-control value without layer labels

Mouse brain tests whether ReliST can operate without DLPFC-like layer annotation. The dataset contains 2,702 Visium spots, 7,109 Allen reference cells, 32,245 shared genes, and 34 Allen taxonomy classes.³⁴^,³⁵ Because curated layer annotations are unavailable, this setting is used for generalization and reference control rather than strong ground-truth benchmarking.

The representative RCTD run produced a spatial risk map suitable for demonstrating the reference-control interpretation in mouse brain (Figure 4A). RCTD is shown as the representative model because its output sharpness and reference sensitivity are suitable for this interpretation. Output sharpness varied strongly across models: the mean top-1 margin was approximately 0.37 for RCTD, 0.12 for Tangram, 0.05 for DestVI, and near zero for Stereoscope and Cell2location (Figure 4B). This variation in output geometry makes simple comparisons of average risk insufficient.

Mouse brain generalization and reference control

(A) Representative RCTD risk map.

(B) Bars show the mean top-1 abundance margin across all tissue spots for each base model.

(C) RCTD review-priority strata.

(D) Points show Spearman correlations between spot-level risk and H&E-derived features for each model.

(E) Points show Spearman correlations between spot-level risk and the reference-instability score. No error bars are shown. The figure supports unlabeled deployment and reference-control interpretation in a setting without DLPFC-like strong labels.

RCTD risk scores had a Spearman correlation of approximately 0.28 with phi_reference, and a high-minus-low phi_reference gap of approximately 0.01 (Figure 4E). Tangram was near zero, whereas DestVI, Stereoscope, and Cell2location were weak or negative for this metric. Thus, reference-related support in mouse brain comes primarily from this RCTD analysis, while the other models contribute weaker evidence under this output contract.

RCTD risk scores were also converted into review-priority strata (Figure 4C). The review-needed top 20% contains 541 of 2,702 spots, the high-risk top 10% contains 271 spots, and the high-risk top 5% contains 136 spots. These labels indicate review or down-weighting priority, not certain errors.

H&E features corroborated this pattern (Figure 4D). RCTD risk was negatively associated with mean H&E saturation and stain darkness, with correlations of approximately −0.257 and −0.228, respectively, suggesting that high-risk spots were enriched in lower-saturation and lighter-staining image contexts. Cell2location and Tangram risk scores were positively associated with edge-associated image features, with correlations of approximately 0.207 and 0.098, respectively. Overall, the mouse brain results show that ReliST can provide reference-control, histology-context sensitivity, and review prioritization even when strong external labels are unavailable.

The mouse brain results also expose an important reality of unlabeled deployment. Without curated layer labels, validation strength is lower, but the risk layer can still produce useful diagnostics, including output geometry, reference sensitivity, and histology context. This is the expected role of ReliST in many new spatial datasets, where it cannot certify truth but can make the uncertainty of interpretation visible and auditable.

Breast cancer high-risk signals mainly support artifact-aware review

Human breast cancer tests ReliST in complex tissue. The dataset contains 3,798 Visium spots and 9,194 reference cells.³⁶^,³⁷ The current analysis uses 13 first-pass major cell types as a working annotation. Compared with DLPFC, breast cancer has higher tumor heterogeneity, more complex stromal, immune, and epithelial mixtures, and stronger H&E artifact risk.³⁷^,³⁸^,³⁹^,⁴⁰^,⁴¹ We use this dataset for artifact-aware complex-tissue validation and review prioritization.

Representative RCTD risk scores were overlaid on the breast cancer H&E image (Figure 5A). RCTD and Tangram risk scores were strongly negatively correlated with mean H&E saturation, with Spearman correlations of approximately −0.63 and −0.58, respectively (Figure 5B). Tissue-edge regions also had higher mean risk in multiple models. These associations place high-risk regions in a histologic context and call for artifact-aware interpretation before drawing conclusions about the tumor microenvironment.

Breast cancer complex-tissue validation

(A) Representative RCTD risk scores overlaid on the breast cancer H&E image.

(B) Points show Spearman correlations between spot-level risk and H&E-derived features for each model.

(C) Consensus high-risk regions shared by RCTD and Tangram.

(D) Bars show the high-risk minus low-risk mean difference for each H&E feature, with risk groups defined by the prespecified thresholds described in STAR Methods.

(E) Stacked bars show the fraction of high-risk spots assigned to each artifact-aware evidence category; fractions within each model sum to 1. No error bars are shown. The figure supports first-pass complex-tissue validation and review prioritization.

The top 10% high-risk spots from RCTD and Tangram each contain 380 spots, with 250 consensus high-risk spots between the two models (Figure 5C). Key high-risk regions therefore recur across model overlays. High-risk top 10% and low-risk bottom 10% spots showed distinct feature profiles (Figure 5D). For RCTD, high-risk regions showed approximately 0.10 lower mean H&E saturation and approximately 0.10 higher endothelial and fibroblast abundances. For Tangram, high-risk regions showed approximately 0.10 lower mean H&E saturation, 0.15 higher endothelial abundance, and 0.09 higher fibroblast abundance.

Artifact-aware interpretation substantially limits the biological strength of these findings. High-risk spots were categorized into primary evidence classes (Figure 5E). In RCTD, empty-tissue signal accounts for approximately 32%, barcode bias for 20%, tissue-edge artifact for 14%, and genuine heterogeneity candidates for 13%. In Tangram, empty-tissue signal accounts for approximately 35%, tissue-edge artifact and local outliers for approximately 17% each, barcode bias for 15%, and genuine heterogeneity candidates for 5%. Among RCTD and Tangram consensus high-risk spots, empty-tissue signal accounts for approximately 42%, barcode bias for 17%, tissue-edge artifact for 14%, and strict genuine heterogeneity for approximately 3%.

The breast cancer results support two levels of interpretation. First, ReliST identifies high-risk regions associated with H&E features and abundance shifts, which is useful for artifact triage and review prioritization. Second, most high-risk signal is closer to histology or quality-control caution than to direct biological discovery. A smaller subset meets the stricter criteria for genuine heterogeneity candidates.

This boundary is particularly important in breast cancer because stromal, vascular, immune, epithelial, and artifact-related signals can overlap spatially. A high-risk stromal or endothelial shift may reflect a biologically meaningful microenvironment, a region of poor tissue capture, an edge artifact, or a reference-label mismatch. ReliST is useful here precisely because it forces those alternatives to be separated before biological claims are made.

IF/gene-protein validation supports protein-level under-call evidence

To strengthen the proxy-to-truth link, we performed orthogonal validation in an independent breast cancer IF/gene-protein dataset. The dataset contains spatial gene expression, antibody capture, and immunofluorescence intensity from the same Visium CytAssist FFPE section.⁴² We ran RCTD with the same first-pass breast cancer reference and compared external protein and IF evidence between the top 10% high-risk and bottom 10% low-risk spots. The analysis includes 4,169 spots, with 417 spots in each extreme group. Here, RCTD defines a representative, reference-oriented abundance expectation for residual testing, and the resulting risk map provides the spatial context for the orthogonal validation (Figure 6A).

Breast cancer IF/gene-protein residual falsification test

(A) Representative RCTD risk map in the independent IF/gene-protein dataset.

(B–D) Representative Vimentin IF, VIM antibody-capture, and ACTA2 antibody-capture maps.

(E) Bars show the high-risk minus low-risk standardized mean difference for independent IF and antibody-capture markers.

(F) Bars show Spearman correlations between spot-level risk and independent marker intensity.

(G) Bars show Spearman correlations between spot-level risk and abundance-adjusted protein residuals. Risk groups and residual calculations are defined in STAR Methods; no error bars are shown. The figure strengthens the proxy-to-truth link through external tissue-state and orthogonal-consistency evidence.

Representative protein and IF evidence maps included Vimentin IF, vimentin (VIM) antibody capture, and ACTA2/alphaSMA antibody capture (Figures 6B–6D). High-risk regions were positively associated with Vimentin IF, VIM, ACTA2, PECAM1/CD31, and CD68. Spatial block bootstrap estimated that mean Vimentin IF intensity was approximately 350 units higher in high-risk than low-risk spots (95% CI, 61–651), with a risk correlation of approximately 0.29. VIM, ACTA2, PECAM1, and CD68 had high-low mean differences of approximately 1.42, 1.36, 0.72, and 0.74, with risk correlations of approximately 0.46–0.52.

Low-risk spots, in contrast, showed higher mean DAPI intensity, mean proliferating cell nuclear antigen (PCNA) intensity, and EPCAM antibody-capture signal. Mean DAPI and PCNA intensities showed high-minus-low differences of approximately −2100 and −960, respectively, whereas EPCAM antibody-capture signal showed a high-minus-low difference of approximately −0.48. High-risk versus low-risk feature shifts and risk-marker associations across all spots are summarized in Figures 6E and 6F, respectively. These findings connect high-risk spots to stromal, vascular, and immune-associated tissue states in independent protein and IF evidence, beyond the internal proxy conflict used to define risk.

We then used cross-modal residual analysis as a residual-based falsification test. RCTD abundance first explains protein or IF evidence for each module; the analysis then focuses on the abundance-adjusted protein residual. Absolute mismatch decreased with risk, with Spearman correlation around −0.31 and 95% CI from −0.41 to −0.21 (Figure 6G). The more informative signal was positive protein residual, where protein evidence was stronger than abundance expectation. This protein-supported under-call was positively associated with risk, with high-low difference around 0.16 and Spearman correlation around 0.37. For stromal and immune under-call, the association was stronger, with high-low difference around 0.23 and Spearman correlation around 0.42.

Reference-adjusted residual control reduces the alternative explanation that the residual signal is driven only by reference mismatch. If the risk signal merely reflected marker enrichment or reference mismatch, its association with protein-supported residuals should weaken after abundance and reference adjustment. In a standardized linear model, protein residual ∼ risk score + φ_reference, the risk-score coefficient remained approximately 0.25 for protein-supported under-call and approximately 0.28 for stromal/immune under-call after controlling for φ_reference. The adjusted coefficient for φ_reference itself was near zero. High-risk regions were therefore more likely to occupy areas where protein evidence exceeded the deconvolution abundance expectation, especially for stromal, vascular, and immune-associated modules.

Together, these analyses strengthen the weakest link in a purely proxy-based validation chain (Figure 6). Marker enrichment alone could mean that high-risk spots are simply located in a different tissue state. Residual analysis asks a stricter question: after abundance expectations are accounted for, is there still external protein evidence that the model may have under-called relevant stromal, vascular, or immune-associated signal? The positive residual result supports that narrower interpretation, while the negative absolute-mismatch result prevents the broader and weaker claim that all high-risk spots are generally cross-modally inconsistent.

Controlled reference-output degradation provides a lightweight sanity check

As a lightweight control that requires no external truth labels, we performed a controlled reference-output degradation stress test on saved DLPFC RCTD abundance outputs. This analysis leaves the base model, the main risk-scoring workflow, and all training procedures unchanged. It tests a necessary condition: when abundance outputs are artificially degraded, output-derived risk should increase.

We used two degradations. Reference dropout reduces the dominant cell-type component for each spot and renormalizes the abundance vector. Label permutation mixes the original abundance vector with a label-permuted abundance vector. Table S4 shows that both degradations increased risk overall. In reference dropout, moderate and stronger degradation produced mean risk deltas of approximately 0.12 and 0.19, with 95% CIs of 0.10–0.14 and 0.16 to 0.22. The fraction of spots with increased risk was approximately 0.84–0.88, and the total-variation error proxy correlated with risk delta at approximately 0.46–0.64. Label permutation also increased risk overall, although the strongest mixture showed saturation or non-monotonicity. This stress test supports the expected directionality of output-derived risk under controlled abundance damage.

This control is placed in Table S4 because it is a necessary-condition test rather than a validation substitute. A risk score that failed this test would be hard to trust as an output-derived diagnostic. Passing it supports reproducibility and directionality, while the stronger validation evidence comes from proxy-level and orthogonal-evidence analyses.

Known-composition pseudo-spots directly link ReliST risk to true deconvolution error

Reviewer 1 asked whether high-risk spots correspond to larger deconvolution errors rather than only to indirect validation proxies. To address this point directly, we reconstructed the pseudo-spatial known-composition benchmark using a donor-disjoint DLPFC snRNA split. The donor identifier BrNum was used to separate signature-building cells from held-out simulation cells, with five donors in each pool. Of 29 labels in cellType_k, 25 pre-specified labels passed the minimum-cell and split-availability criteria. We then generated 1,200 pseudo-spots by sampling mixtures of held-out cells across smooth pseudo-spatial regions and retained the true cell-type composition for validation only. The risk layer did not receive the true composition at runtime. We evaluated the lightweight non-negative least-squares (NNLS) projection together with two representative base deconvolution models from the original manuscript, RCTD and Tangram (Figure 7A; Table S6).

Pseudo-spatial known-composition validation

(A) ReliST risk-score performance for NNLS, RCTD, and Tangram in the donor-disjoint known-composition benchmark. Points are the observed Spearman correlation with true total-variation error and AUROC for identifying the highest-error 20% of pseudo-spots; horizontal error bars are paired-bootstrap 95% confidence intervals obtained by resampling pseudo-spots within each base-model output.

(B) NNLS risk-error relationship.

(C) Retained mean total-variation error after keeping the indicated lowest-risk fraction.

(D) AUROC for identifying the highest-error 20% of pseudo-spots across ReliST diagnostic scores and comparison scores. Blue bars denote ReliST diagnostic scores, gray bars denote comparator scores, and parenthetical values indicate AUROC improvement over the selected model-specific proxy. Asterisks mark cross-model disagreement, which requires multiple complete deconvolution outputs.

In this donor-disjoint benchmark, ReliST-derived risk scores were positively associated with true composition error, with model- and score-dependent effect sizes (Figures 7B–7D). The equal-weight risk_score was retained as the original scalar formulation for audit rather than retuned after inspecting the benchmark. It reached Spearman correlations of 0.274, 0.226, and 0.174 with total-variation error for NNLS, RCTD, and Tangram, respectively; the corresponding areas under the receiver operating characteristic curve (AUROCs) for identifying the highest-error 20% of pseudo-spots were 0.625, 0.593, and 0.574. Paired bootstrap 95% CIs were 0.588–0.661, 0.548–0.631, and 0.535–0.612 for these AUROCs. As a lightweight split-and-simulation repeat check, we repeated the donor-disjoint NNLS pseudo-spot generator with two additional random seeds. These repeats re-ran both donor partitioning and pseudo-spot sampling, so they test robustness to the simulated split rather than fixed-split sampling alone. Across the primary seed and two repeats, the NNLS risk_score remained positively associated with true error, with Spearman correlations from 0.253 to 0.352 and AUROCs from 0.619 to 0.695. Because only three seeds were used, we report this as an observed range rather than as a seed-level confidence interval. The shuffled-coordinate null control reduced the NNLS risk_score AUROC from 0.625 to 0.505, indicating that the local-structure contribution depended on the spatial structure. These results establish a direct quantitative link between ReliST risk and true deconvolution error under a donor-disjoint pseudo-spot design and support the equal-weight score as a scalar audit readout with model-dependent effect size.

We next compared ReliST-derived scores with confidence and disagreement baselines. To avoid conflating deployment settings, we separated single-run, single-model heuristics from same-model perturbation ensemble disagreement and cross-model disagreement. The single-run, single-model comparison included abundance entropy, inverse top-1 margin, inverse maximum abundance, ambiguity-only score, and model-specific output confidence proxies. RCTD inverse maximum weight was treated as an output-derived confidence proxy, and Tangram ensemble standard deviation across five gene-subset runs was treated as same-model perturbation ensemble disagreement. The comparison positioned ReliST alongside abundance-derived confidence heuristics as a model-agnostic, single-output risk-audit layer with interpretable failure-mode axes. In RCTD, the local-plus-ambiguity ReliST score reached an AUROC of 0.638, close to abundance entropy (0.624) and above the RCTD inverse-maximum-weight confidence proxy (0.590). In Tangram, the reference-risk ReliST score reached an AUROC of 0.663 and exceeded the perturbation-based gene-subset ensemble SD baseline (0.484), while abundance entropy was higher in this setting (0.715). For NNLS, the local-plus-ambiguity ReliST score was comparable with abundance entropy (0.670 versus 0.682). We also included cross-model disagreement as a strong multi-model comparator. It was highest in Tangram (AUROC 0.844), and it requires multiple complete deconvolution outputs, addressing a deployment setting distinct from auditing one base-model output. Descriptive linear models of true error showed that adding the primary ReliST score to the best available baseline increased R2 by 0.049, 0.004, and 0.022 for NNLS, RCTD, and Tangram, respectively (Table S7), providing additional descriptive variance explanation. Overall, ReliST is best interpreted as a model-agnostic, single-model risk-audit layer with interpretable failure-mode axes that can be used alone or alongside abundance-derived confidence heuristics, same-model perturbation disagreement, and cross-model disagreement when those signals are available.

We further performed component ablation to clarify how the risk signals contribute to the final score. These analyses defined a clearer component hierarchy for the current implementation. Output ambiguity was the most consistently informative component in the known-composition benchmark, whereas the incremental value of local inconsistency and reference-related evidence was model- and context-dependent. In the aggregate reference-perturbation summaries, the no-reference variant matched or exceeded the equal-weight scalar score, positioning phi_reference as an optional axis-specific diagnostic for reference-related failure modes. The equal-weight risk_score remains the original scalar formulation retained for audit; component variants are supporting ablations shown in full in the supplementary source tables (Figure S8; Table S8).

Because several analyses use empirical high-risk cutoffs, we also evaluated threshold sensitivity and use-case guidance with random and oracle filtering references. Across review fractions, the error reduction in retained pseudo-spots changed smoothly across operating points (Figures 7C and S9; Table S9). In the multi-model benchmark, keeping the lowest-risk 80% using the primary risk_score reduced retained mean error from 0.504 to 0.487 for NNLS, from 0.431 to 0.419 for RCTD, and from 0.615 to 0.606 for Tangram, corresponding to reductions of 3.3%, 2.8%, and 1.4%, respectively. For NNLS, the expected retained error under random selection was unchanged, whereas oracle filtering reduced the retained mean error to 0.444 at the same coverage. We therefore describe 95%, 90%, 80%, and 70% keep fractions as illustrative review budgets selected from coverage-risk curves. These curves show review-budget sensitivity and operating-point tradeoffs across score families. Cross-tissue and cross-platform threshold calibration remains a deployment-calibration step for future applications. In practice, the threshold should be selected from the coverage-risk curve, available review budget, tissue context, base model behavior, and tolerance for false positives and false negatives.

Finally, we used controlled reference perturbation to test sensitivity to reference mismatch and reference coverage failure. We evaluated marker-gene dropout, dominant cell-type dropout, excitatory/inhibitory label coarsening, and synthetic out-of-reference contamination. These perturbations changed the error distribution as expected, with dominant cell-type dropout producing the largest mean true error. Across these aggregate summaries, the no-reference variant matched or exceeded the full equal-weight score, reinforcing the component-level interpretation that reference-related evidence is most useful as a failure-mode-specific diagnostic rather than as a uniformly weighted scalar-score term (Figures S10 and S11; Table S10).

Discussion

The revision analyses define the supported operating setting for ReliST. In pseudo-spots with known composition, risk scores rank true deconvolution error and improve retained-set error after selective filtering. In natural tissues, where spot-level composition truth is not available, ReliST functions as a risk-aware decision layer for prioritizing, filtering, reviewing, and selecting predictions for orthogonal validation. This preserves the distinction between actionable risk evidence and calibrated spot-level error probabilities.

The comparison with confidence-style baselines also refines the contribution. ReliST is designed for single-model risk auditing under a shared output contract: it converts canonical abundance output, local spatial context, and reference-sensitive diagnostics into interpretable failure-mode axes. Standard abundance-derived confidence heuristics can be strong for some models, same-model perturbation disagreement can be useful when perturbation runs are available, and cross-model disagreement is powerful when multiple complete outputs are available. ReliST’s advantage is therefore operational: it provides model-agnostic, axis-specific risk evidence when one base-model output is being audited, and it can be used alongside perturbation or multi-model disagreement when those signals are available.

This study reframes spatial transcriptomics deconvolution from a prediction-only task into a risk-aware usage problem. Standard deconvolution outputs provide cell-type abundance maps, but they rarely tell users which local regions are unreliable. ReliST adds a spot-level risk score to base-model outputs and uses that score for selective filtering, review-needed flags, abstention, and artifact-aware interpretation. The goal is to make local trust explicit, so that downstream analyses can use deconvolution maps with clearer boundaries.

The three main datasets and one orthogonal validation case provide complementary evidence. DLPFC shows that, in a tissue with strong structural anchors, risk scores align with layer-related difficulty, marker discordance, and signature residuals, and that coverage-risk and reliability curves support risk filtering as a decision tool. Mouse brain shows that the risk layer can still provide reference-sensitivity and review-priority stratification in an unlabeled setting. Breast cancer defines the complex-tissue boundary: high-risk regions can be associated with histology and abundance shifts, while strict genuine heterogeneity candidates remain a minority. IF/gene-protein validation further shows enrichment for independent stromal, vascular, and immune-associated protein evidence, with risk remaining associated with positive protein residuals after abundance and reference adjustment. This use of orthogonal protein or imaging evidence is consistent with broader spatial multi-omics and spatial proteomics validation practice.³⁸^,⁴³^,⁴⁴

The results also show that base models support the risk layer in different ways. Cell2location and RCTD are better suited as reference-related baselines in these experiments. Tangram and Stereoscope provide marker-sensitive or structure-sensitive evidence. DestVI is better described as a limited-support comparator under this output contract. ReliST therefore acts as an interpretation interface: it identifies which risk axes are supported for a model under a shared output contract and where a model should be used cautiously.

This framing distinguishes ReliST from generic uncertainty quantification and conformal prediction.²⁷^,²⁸^,²⁹^,³⁰ Output ambiguity, local inconsistency, and reference mismatch are familiar ingredients. ReliST’s contribution is to define residual failure risk for a reference-dependent, ground-truth-limited spatial deconvolution task, and to organize these signals as model-interface-agnostic audit axes whose utility is evaluated empirically. In the present revision, output ambiguity was the most consistently informative scalar component in the known-composition benchmark, while local and reference-related signals showed model- and context-dependent value. The common-feature null control in Table S3 supports the broader audit framing by testing ReliST axes against abundance entropy, top-1 margin, maximum abundance, and common local inconsistency, and negative or weak axes are reported rather than forced to succeed.

Methodologically, a central contribution is moving risk validation beyond correlation-only evidence. A simple correlation between risk and a proxy can be useful for screening, but filtering and abstention require stronger support.³¹^,³² The score should show monotonic proxy burden across risk bins, and filtering high-risk spots should improve the retained subset. Coverage-risk curves, reliability curves, selective gain, bootstrap confidence intervals, and residual validation therefore evaluate ReliST as a use mechanism rather than a visual heatmap alone.

The main limitation is the absence of spot-level ground truth. DLPFC layer annotations and marker proxies provide strong external anchors, but they cannot prove that every high-risk spot is wrong. IF/gene-protein validation strengthens part of the proxy-to-truth link, and cross-modal residual analysis shows that high risk is associated with protein-supported under-call after abundance adjustment. Reference-adjusted residual control reduces a pure reference-mismatch explanation. These signals remain orthogonal consistency evidence rather than spot-level deconvolution error labels. Controlled output degradation adds a stress test by showing that risk increases when output is artificially damaged.

A second limitation is threshold calibration. We report empirical risk-bin diagnostics rather than finite-sample conformal guarantees.²⁹^,³⁰ Because these diagnostics still rely on proxy errors rather than true spot-level errors, risk thresholds should be interpreted as operating points and review priorities. Stronger threshold calibration would require additional samples, pathologist review, matched spatial proteomics, or other direct orthogonal measurements.

A third limitation is generalization. DLPFC, mouse brain, and breast cancer represent structured tissue, unlabeled deployment, and complex tumor tissue, but they do not cover all platforms, tissues, or disease states. FFPE, Visium HD, and single-cell-resolution spatial platforms may have different noise structures and mixing patterns.⁴²^,⁴⁴^,⁴⁵^,⁴⁶^,⁴⁷^,⁴⁸^,⁴⁹ Future work should evaluate the stability of the risk layer across broader reference-matching regimes, cell-state granularities, histology qualities, and spatial technologies.

These limits point to a practical recommendation: spatial deconvolution outputs should include both cell-type maps and risk maps. This recommendation is aligned with broader reproducibility, infrastructure, and auditable-analysis principles in single-cell and spatial omics.¹⁷^,²¹^,⁵⁰ This output form helps users restrict downstream interpretation to more trustworthy regions, route high-risk regions to manual review or additional validation, and avoid over-interpreting model-instability artifacts as biological findings.

For practical use, ReliST should be reported with its failure modes. The minimum input is a spatial expression object, spot coordinates, a base-model abundance output, and a compatible reference or marker signature source when reference-related scoring is used. If reference signatures are weak, missing, or mismatched to the tissue, ReliST should be interpreted as a limited-support diagnostic rather than a probability-calibrated risk layer. Users should select operating points according to the downstream task: conservative quantification should retain fewer high-risk spots, exploratory visualization may keep use-with-caution spots with explicit labels, and validation workflows can prioritize review-needed spots for manual or orthogonal follow-up.

In practice, ReliST can be used in three steps. First, run the preferred base deconvolution model and export its canonical abundance output. Second, compute the ReliST risk layer and inspect the axis-specific evidence that is supported for that model and dataset. Third, choose an operating point based on the downstream goal, such as conservative quantification, exploratory visualization, or review-focused validation. This workflow keeps the base-model prediction intact while making the interpretation boundary explicit.

A complete deconvolution report should therefore include the cell-type abundance map, the risk map, the chosen risk threshold or retained fraction, the proportion of spots assigned to trusted, use-with-caution, and review-needed categories, and the evidence class used to interpret high-risk regions. Reporting these elements makes it easier to audit whether a downstream biological claim was made from a broadly trusted region, a filtered subset, or a region that ReliST explicitly marked as requiring caution.

We introduced ReliST, a model-agnostic risk layer for spatial transcriptomics deconvolution. ReliST leaves base deconvolution models unchanged and estimates where their outputs require caution. Across DLPFC, mouse brain, and breast cancer, risk scores aligned with external validation proxies, selective filtering behavior, coverage-risk curves, histology-context evidence, and IF/gene-protein orthogonal validation. The current evidence supports decision utility at the proxy, tissue-state, and orthogonal-consistency levels. ReliST therefore provides an operational framework for spatial deconvolution under limited ground truth: complete reports should pair abundance maps with risk maps, retained fractions, and review labels, so that downstream claims are tied to explicit trust boundaries rather than implicit model confidence.

Limitations of the study

This study has several limitations. First, the known-composition benchmark uses pseudo-spatial mixtures generated from held-out reference cells. This provides a direct quantitative test of the relationship between risk and true deconvolution error and complements the natural tissue analyses, which capture spatial, histological, and technical complexity. Donor disjointness applies to signature estimation and pseudo-spot generation; the 1,000-gene analysis feature set and optional cell-type order were inherited from the canonical DLPFC workflow to keep the revision benchmark aligned with the manuscript output contract. Natural tissue analyses therefore remain necessary for testing whether risk scores align with independent structure, histology, marker, and protein evidence.

Second, high-risk thresholds are empirical operating points rather than calibrated error probabilities. The top 10%, top 20%, or 80th percentile thresholds are useful for review prioritization, selective filtering, and downstream sensitivity analysis, but they should be adjusted according to tissue context, base model behavior, and downstream tolerance for false positives or false negatives.

Third, ReliST depends on the canonical output contract available from each base model. Abundance entropy, output ambiguity, output confidence proxies, perturbation-based disagreement, and reference sensitivity can have different meanings across methods. We therefore interpret cross-model differences as observable support under a shared output contract rather than as definitive model-quality rankings. The known-composition benchmark also supports a model-dependent view of component combinations across base models.

Fourth, reference-related evidence depends on the quality, granularity, and coverage of the reference atlas. Reference perturbation analyses show that risk scores respond to controlled reference mismatch and out-of-reference contamination, supporting careful reference construction, annotation review, and orthogonal validation in new tissues. In the present scalar benchmark, reference-related evidence is best treated as an optional diagnostic axis for reference-sensitive failure modes.

Resource availability

Lead contact

Further information and requests for resources should be directed to and will be fulfilled by the lead contact, Sen Lin (ls3994@gzucm.edu.cn).

Materials availability

This study did not generate new unique biological materials.

Data and code availability

•
This study analyzes publicly available spatial transcriptomics, single-cell reference, histology, antibody-capture, and immunofluorescence datasets. Accession numbers and source links are listed in the key resources table. The tabular source data supporting the revision analyses are provided in Tables S6, S7, S8, S9, and S10.
•
The ReliST source code, analysis scripts, selected-run metadata, and manuscript figure-generation scripts are publicly available at https://github.com/tp5353/ReliST under the MIT license. The revision-specific code and scripts are frozen in the GitHub tag and release v0.2.0-revision (https://github.com/tp5353/ReliST/releases/tag/v0.2.0-revision).
•
Any additional information required to reanalyze the data reported in this paper is available from the lead contact upon request.

Acknowledgments

This work was supported by the Shenzhen High-level Hospital Construction Fund, the Sanming Medical Project in Shenzhen (grant no. SZZYSM202211003), the General Program of the Guangdong Provincial Administration of Traditional Chinese Medicine (grant no. 20261322), and the Shenzhen Chinese Medicine Society (grant no. 2024073F). The authors thank the public dataset generators and data providers whose resources made this study possible.

Author contributions

X.Z., Y.X., and S.L. conceived the study. X.Z. developed the ReliST framework, implemented the analysis workflow, performed the computational analyses, generated the figures and tables, and drafted the manuscript. L.H., Y.P., Y.L., J.K., and L.P. contributed to study design, data interpretation, biological and clinical contextualization, and manuscript revision. Y.X. and S.L. supervised the project, contributed to funding acquisition, interpreted the results, and revised the manuscript. All authors read and approved the final manuscript.

Declaration of interests

The authors declare no competing interests.

Declaration of generative AI and AI-assisted technologies in the writing process

During manuscript revision and final-material preparation, the authors used OpenAI ChatGPT and Codex for language editing, consistency checks, document restructuring, and formatting assistance. The authors reviewed and edited all AI-assisted output and take full responsibility for the final content. No generative AI tool was used to generate research data, perform statistical analyses, interpret results, or draw scientific conclusions.

STAR★Methods

Key resources table

REAGENT or RESOURCE	SOURCE	IDENTIFIER
Deposited data

Human DLPFC Visium and snRNA-seq reference	Maynard et al.³³	https://doi.org/10.1038/s41593-020-00787-0
Mouse Brain Serial Section 1 sagittal-anterior Visium dataset	10× Genomics³⁴	https://www.10xgenomics.com/datasets/mouse-brain-serial-section-1-sagittal-anterior-1-standard-1-0-0
Allen mouse isocortex and hippocampal taxonomy snRNA-seq reference	Yao et al.³⁵	https://doi.org/10.1016/j.cell.2021.04.021
Human Breast Cancer Block A Section 1 Visium dataset	10× Genomics³⁶	https://www.10xgenomics.com/datasets/human-breast-cancer-block-a-section-1-1-standard-1-1-0
Human breast cancer single-cell RNA-seq atlas	Pal et al.³⁷	https://doi.org/10.15252/embj.2020107333
Visium CytAssist FFPE Human Breast Cancer gene/protein and IF dataset	10× Genomics⁴²	https://www.10xgenomics.com/datasets/gene-and-protein-expression-library-of-human-breast-cancer-cytassist-ffpe-2-standard
Revision source-data tables for known-composition baseline ablation threshold and reference-perturbation analyses	This paper	Tables S6, S7, S8, S9, and S10

Software and algorithms

ReliST source code	This paper	https://github.com/tp5353/ReliST
ReliST v0.2.0-revision frozen code snapshot	GitHub	https://github.com/tp5353/ReliST/releases/tag/v0.2.0-revision
Stereoscope	Andersson et al.⁹	https://doi.org/10.1038/s42003-020-01247-y
RCTD/spacexr	Cable et al.¹⁰	https://doi.org/10.1038/s41587-021-00830-w
Tangram	Biancalani et al.¹¹	https://doi.org/10.1038/s41592-021-01264-7
cell2Location	Kleshchevnikov et al.¹²	https://doi.org/10.1038/s41587-021-01139-4
DestVI	Lopez et al.¹³	https://doi.org/10.1038/s41587-022-01272-8
Python	Python Software Foundation	version 3.13.12; https://www.python.org/
NumPy	NumPy Developers	version 2.3.5; https://numpy.org/
pandas	pandas development team	version 2.3.3; https://pandas.pydata.org/
SciPy	SciPy community	version 1.17.1; https://scipy.org/
scikit-learn	scikit-learn developers	version 1.8.0; https://scikit-learn.org/
Matplotlib	Matplotlib development team	version 3.10.8; https://matplotlib.org/
seaborn	seaborn development team	version 0.13.2; https://seaborn.pydata.org/
AnnData	scverse	version 0.12.10; https://anndata.readthedocs.io/
Scanpy	scverse	version 1.12; https://scanpy.readthedocs.io/

Open in a new tab

Experimental model and study participant details

Ethics statement

This study used only publicly available, de-identified datasets from previously published studies or public data resources. No new human participants, animals, or tissue specimens were recruited, collected, or experimentally manipulated. Ethical approval and consent were obtained by the original studies and data generators.

Datasets and validation design

We use three main datasets and one independent orthogonal validation dataset. Human DLPFC contains 47,681 Visium spots and 77,604 reference cells, with 30 reference labels and strong structural anchors.³³ Mouse brain contains 2,702 Visium spots and 7,109 Allen reference cells, with 34 Allen taxonomy classes, and is used as a generalization and reference-control setting.³⁴^,³⁵ Human breast cancer contains 3,798 Visium spots and 9,194 reference cells.³⁶^,³⁷ The current breast cancer analysis uses 13 first-pass major cell types as a working annotation and is used for complex-tissue validation and artifact-aware interpretation.

The independent IF/gene-protein dataset is a public 10× Genomics Visium CytAssist FFPE human breast cancer section.⁴² It contains spatial gene expression, antibody capture, H&E and IF images, and barcode-level fluorescence intensities. Spatial RNA-protein and immunofluorescence measurements provide useful orthogonal evidence for transcriptome-derived interpretations.³⁸^,⁴³^,⁴⁴ The dataset includes 4,169 spots, 18,085 gene features, and 35 antibody capture features. Because it comes from a separate patient section, we use it as orthogonal support rather than final ground truth. It tests whether high-risk regions are supported by independent protein or IF evidence and abundance-adjusted residuals.

We avoid direct cross-tissue comparisons of risk-score means or model-output differences. DLPFC, mouse brain, and breast cancer differ in reference matching, cell-state granularity, spot size, tumor heterogeneity, and histology quality. Table 1 therefore defines dataset context and claim boundaries for each validation setting.

The validation design is layered rather than a parallel model leaderboard. DLPFC asks whether risk can be anchored against strong tissue-structure proxies. Mouse brain asks whether the same framework can still generate reference-control and review signals when strong layer labels are absent. Breast cancer asks whether high-risk regions remain interpretable in a complex tissue where artifacts, stromal variation, and tumor heterogeneity are entangled. The independent IF/gene-protein dataset then tests one part of the proxy-to-truth link through external protein and fluorescence evidence.

Method details

Problem definition and risk direction

For spot i, let x_i denote the spatial transcriptomics measurement and y_hat_i = g(x_i) denote the output of a base deconvolution model. ReliST leaves y_hat_i unchanged and calculates a risk function, r_i = f(x_i, y_hat_i). A higher r_i indicates that the prediction at spot i requires more caution. A lower r_i indicates that the prediction is better suited for downstream analysis or preliminary interpretation.

This definition separates ReliST from a conventional deconvolution benchmark. Benchmarks compare average accuracy and often rank methods globally. ReliST estimates localized failure risk and asks which locations require caution. The high-risk, trusted, and review-needed labels used here are usage recommendations and review priorities, rather than spot-level truth labels. Risk values are interpreted as ranking and operating-point scores, not as calibrated probabilities that a spot is erroneous.

Inputs and multi-axis risk signals

ReliST reads base-model outputs and reliability signals derived from those outputs, independent of internal model parameters or native posterior uncertainty from any specific method. The current implementation uses three main runtime signals. Output ambiguity captures flat abundance distributions, small top-1 margins, or the absence of a clear dominant cell type. Local inconsistency captures conflict between a spot’s predicted abundance and its spatial neighborhood or local expression context. Reference-related signal captures weak support with respect to reference composition, reference label space, or marker evidence.

The contribution lies in how these signals are organized for spatial deconvolution failure risk. ReliST treats output ambiguity, local inconsistency, reference-related evidence, marker-sensitive evidence, and structure-related evidence as audit axes whose usefulness is evaluated by model and context. The reference-related diagnostic targets reference dependence and reference-coverage failure when those signals are supported, but it is not treated as a universally required scalar component. The reference-marker-aux branch strengthens marker-sensitive proxy checks when they are informative. The structure-axis is optional and context dependent, applying only when tissue structure provides an external constraint. Different base models support these axes to different degrees, which we report as support tiers.

Base models and canonical output contract

We connect ReliST to five representative base models: cell2Location, RCTD, Tangram, Stereoscope, and DestVI. These models cover probabilistic modeling, mixture decomposition, cross-modal mapping, and deep generative modeling. To keep ReliST model agnostic, the risk layer reads a canonical output contract consisting of abundance, uncertainty-like or ambiguity-like signal when available, and metadata. Shared internal parameters or a common posterior definition are unnecessary.

This design allows both Python-native and R-native backends. Cell2Location, Tangram, and DestVI can be run through Python runners, while RCTD can be connected through an R adapter. The method boundary is the shared representation of the spatial input and the base-model prediction; the implementation language and model-specific posterior are backend details.

To test whether ReliST simply reuses confidence-style features, we construct a canonical output-contract fairness and common-feature null control. This control uses only signals shared by all base models: abundance entropy risk, inverse top-1 margin, inverse maximum abundance, and rank-normalized local inconsistency. For each model-proxy pair, Table S3 selects the common feature with the highest AUC as the best common feature and compares it with the corresponding risk-axis score.

The common-feature control is intentionally strong. It gives the null model the best available shared feature for each model-proxy pair, rather than forcing a single weak baseline across all settings. This makes the comparison conservative: if a risk-axis signal improves over the best common feature, the improvement is less likely to be explained by generic output sharpness, maximum-abundance dominance, or local inconsistency alone. Conversely, when the common feature outperforms a proposed risk axis, we treat that axis as unsupported or limited-support evidence rather than smoothing over the negative result.

Reproducible risk scoring

The current ReliST implementation is a deterministic score rather than a supervised learned model. Layer annotations, H&E features, IF/gene-protein evidence, and validation proxies are excluded from parameter fitting. These external signals enter only during validation and interpretation. Runtime parameters are specified in YAML configuration files and stored with selected runs in metadata.

This separation controls leakage. At runtime, ReliST reads the spatial expression input, spot coordinates, base-model abundance output, optional model-provided uncertainty-like output, and reference-derived signatures or marker subsets. DLPFC layer labels, pathology labels, image features, IF intensities, antibody-capture features, and validation proxy labels are introduced only after scoring, when the manuscript asks whether high-risk spots are enriched for independent evidence of unreliable interpretation.

Given a non-negative abundance matrix A = [a_ik], abundances are first normalized within each spot to proportions, p_ik = a_ik/sum_k a_ik. Spatial neighborhoods are built from spot coordinates using a k-nearest-neighbor graph, with k = 8 in the main runs. Neighbor weights use inverse-distance weighting. The local inconsistency feature, φ_local, is defined as the L1 residual between the current abundance proportion vector and the weighted neighborhood average, with a local expression heterogeneity correction to reduce the risk of treating true local heterogeneity as a spatial inconsistency.

The neighborhood average is calculated on the normalized abundance vector, so φ_local captures changes in predicted composition rather than total signal scale. The expression-heterogeneity correction reduces the weight of abundance discontinuities in regions where local expression already suggests abrupt tissue change. In practice, this makes φ_local a warning about unsupported spatial discontinuity, rather than a generic edge detector.

The uncertainty or ambiguity feature, φ_uncertainty, combines model-specific uncertainty with output-ambiguity proxies. We first calculate abundance-derived ambiguity from entropy and inverse maximum abundance. If a base model provides an uncertainty-like output, it is converted to relative uncertainty and combined with ambiguity after rank normalization. If no usable uncertainty-like output exists, φ_uncertainty reduces to abundance-derived ambiguity.

This treatment allows different models to be readthrough a common risk framework without assuming that their native uncertainty fields are equivalent. Posterior variance, mapping ambiguity, and abundance-derived entropy do not have the same probabilistic meaning. Rank normalization and explicit labeling therefore preserve their role as uncertainty-like or ambiguity-like evidence, while preventing the manuscript from claiming cross-model calibrated uncertainty.

The reference-related diagnostic uses reference_subsampling_instability: marker subsets are repeatedly sampled, discordance is recalculated, and the standard deviation across repeated draws is used as phi_reference. Marker-sensitive auxiliary runs can use the mean discordance across marker subsets, but this is interpreted as reference-marker-aux rather than as a replacement for, or proof of, a universal reference-dominated scalar score.

The single risk score is calculated with sample-aware weighted z-scores and compressed to the 0 to 1 range with a sigmoid transformation. In the main runs, the single risk score is driven primarily by local inconsistency and uncertainty or ambiguity. In the DLPFC multi-axis analysis, structure risk score emphasizes local inconsistency and output ambiguity, whereas reference risk score emphasizes uncertainty or ambiguity and φ_reference. All high-risk thresholds are empirical quantile thresholds rather than truth-calibrated cutoffs. Unless otherwise stated, descriptive analyses use the top 10% of risk scores as the high-risk group and the bottom 10% as the low-risk group. The decision-support bundle in Figure 3 uses the within-sample 80th percentile as a high-risk cutoff.

These empirical thresholds should be interpreted narrowly. A top-10% threshold creates a high-risk stratum for enrichment tests and figure summaries. An 80th-percentile threshold creates an operational review set for decision support. Neither threshold asserts that a fixed fraction of spots is wrong, nor that a score of 0.8 has the same meaning across tissues or base models. Thresholds are therefore chosen as transparent operating points and should be adjusted when the intended use changes from conservative review, to high-coverage screening, to follow-up experimental prioritization.

Implementation details and reproducibility

Runtime parameters were stored in version-controlled configuration files and selected-run metadata. Base-model execution, ReliST risk scoring, and manuscript-facing validation were performed using the archived v0.2.0-revision code snapshot. The repository provides the software environment, dependency versions, license, and command-line examples required to reproduce the manuscript figure and table bundles, while revision-specific score definitions, run identifiers, and source-data tables are provided in Tables S6, S7, S8, S9, and S10.

Pseudo-spatial known-composition benchmark

To directly evaluate whether ReliST risk scores track true deconvolution error, we generated a pseudo-spatial known-composition benchmark from the DLPFC snRNA reference. The reference cells were split by the donor identifier BrNum into donor-disjoint signature-building and held-out simulation pools. In the primary split, Br3942, Br6423, Br6432, Br6471, and Br8667 were used for signature construction, while Br2720, Br2743, Br6522, Br8325, and Br8492 were used for pseudo-spot simulation. Of 29 labels in cellType_k, 25 labels passed the pre-specified minimum-cell and split-availability criteria: at least 80 total reference cells before splitting and at least 20 cells per retained type on each side of the donor-disjoint split. The 1,000-gene analysis feature set and optional cell-type order were inherited from the canonical DLPFC run to keep the revision benchmark aligned with the manuscript output contract; therefore, donor-disjoint describes signature construction and pseudo-spot simulation, not a fully training-only gene-selection procedure. For each retained cell type, mean log-normalized reference signatures were estimated from the signature-building pool. Pseudo-spots were placed on a regular grid with six smooth pseudo-spatial regions. For each pseudo-spot, a region composition vector was sampled from a Dirichlet distribution, 4–10 held-out cells were sampled with replacement according to a multinomial draw, and raw counts from those held-out cells were summed to form pseudo-bulk expression. The primary simulation used random seed 20260620 and generated 1,200 pseudo-spots. Two additional donor-disjoint NNLS repeats used seeds 20260621 and 20260622 and re-ran both donor partitioning and pseudo-spot sampling. These three seeds were used as a lightweight split-and-simulation robustness check and are summarized by observed ranges rather than seed-level confidence intervals. Each pseudo-spot retained its known cell-type composition for validation only; the risk scoring workflow did not use this truth information.

Pseudo-spot expression was normalized as log1p counts per 10,000 for the lightweight projection analyses. The pseudo-spot generator included four scenario classes: clean pseudo-bulk spots, low-depth spots generated by binomial thinning with keep probability sampled uniformly from 0.15 to 0.45, marker-dropout spots in which markers of the dominant sampled cell type were retained with probability 0.25, and diffuse-mixture spots in which composition was smoothed toward a uniform mixture before cell sampling. No additional parametric count-noise model was added beyond held-out-cell sampling and these explicit binomial thinning operations. Predicted cell-type abundance was obtained by non-negative least-squares projection of pseudo-spot expression onto the reference signature matrix, followed by row normalization. To test model generality, the same primary pseudo-spot h5ad and donor-disjoint training-reference h5ad were also passed through RCTD and Tangram using the same canonical output contract as the main manuscript analyses. True deconvolution error was calculated by comparing the predicted abundance vector with the known composition vector. The primary error metric was total variation distance, with L1 error, RMSE, cosine distance, and dominant-label mismatch retained as auxiliary metrics. To check whether local-structure scores benefited artificially from the coordinate construction, the multi-model evaluation also included a shuffled-coordinate local score in which spatial coordinates were permuted before neighbor construction; this score was retained as a null control rather than as a primary ReliST score. The reference-perturbation stress tests re-estimated predictions from the perturbed reference/signature inputs for the lightweight projection setting and were interpreted as controlled stress tests, not as natural tissue ground truth.

Confidence and uncertainty baseline comparison

We compared ReliST risk scores with confidence-style baselines derived from the same canonical abundance output. We defined three comparison settings. The single-run, single-model setting included abundance entropy risk, inverse top-1 margin, inverse maximum abundance, ambiguity-only score, and model-specific output confidence proxies. Same-model perturbation ensemble disagreement was evaluated for Tangram gene-subset runs. Cross-model disagreement was evaluated separately as a multi-model comparator because it requires multiple complete deconvolution outputs for the same pseudo-spots. In the known-composition benchmark, each score was evaluated by its Spearman correlation with true total-variation error, its AUROC for identifying the highest-error 20% of pseudo-spots, its retained mean error after keeping the lowest-risk 80% of pseudo-spots, and AURC. Paired bootstrap was used to estimate 95% confidence intervals for AUROC, Spearman correlation, AURC, retained mean error, and score differences, with pseudo-spots resampled within each base-model output. In the DLPFC anchored proxy control, the best common feature was selected for each model-proxy pair to create a conservative null comparison.

Risk score calculation

For a score defined by feature values phi_{k,i} and weights w_k, ReliST computes r_i = sigmoid(sum_k w_k z_g(phi_{k,i})/sum_k |w_k|), where z_g is a Z score calculated within normalization group g. Missing or zero-weight components are excluded from both the numerator and denominator. In the revision known-composition benchmark, g was the pseudo-spot sample_id group and the primary scalar risk_score used equal weights for phi_local, phi_uncertainty, and phi_reference. The local_uncertainty_risk_score, reference_risk_score, leave_out_reference, and reference_only columns were retained as pre-specified diagnostic variants rather than retuned after inspecting true error. Score definitions, feature weights, and normalization groups are provided in Table S6. In natural tissue runs, the same weighted Z score form is used with the sample or dataset-specific normalization group and the selected run configuration.

Component ablation

Component ablation was performed on the known-composition benchmark using the same pseudo-spots and prediction outputs. We compared the full risk score with single-component and multi-component variants based on phi_local, phi_uncertainty, and phi_reference. For each ablation score, we computed Spearman correlation with true error, AUROC for the highest-error 20% of pseudo-spots, average precision, and retained mean error across keep fractions. Delta metrics were calculated relative to the full risk score.

Threshold sensitivity and use-case guidance

Threshold sensitivity was evaluated by varying the fraction of spots sent for review or retained for downstream analysis across NNLS, RCTD, and Tangram outputs. High-risk review fractions included 5%, 10%, 20%, and 30%. Retained low-risk fractions included 70%, 80%, 90%, and 95%. For each operating point, we reported retained mean error, error reduction relative to the full pseudo-spot set, capture of the highest-error 20% of pseudo-spots, and precision among flagged high-risk spots. Random filtering and oracle filtering were included as lower and upper reference policies within each base-model output. Thresholds were interpreted as illustrative review budgets selected from coverage-risk curves, with cross-tissue and cross-platform threshold calibration treated as a future deployment-calibration step.

Reference perturbation stress tests

Reference perturbation stress tests were performed on the known-composition pseudo-spots to evaluate sensitivity to reference mismatch. We considered five scenarios: the full-reference baseline, marker-gene dropout, dominant cell-type dropout, excitatory/inhibitory label coarsening, and out-of-reference contamination. Marker-gene dropout removed high-specificity marker genes before projection. Dominant cell-type dropout removed the three cell types with the highest average true pseudo-spot abundance from the reference signature matrix. Label coarsening merged excitatory and inhibitory subclasses into broad classes and evaluated error in the coarsened truth space. Out-of-reference contamination mixed a synthetic non-reference expression profile into pseudo-spots and evaluated against an explicit out-of-reference composition column. These analyses were interpreted as controlled stress tests and not as natural tissue ground truth.

Revision analysis reproducibility

The revision analyses were generated by the following scripts: scripts/run_revision_known_composition_benchmark.py, scripts/run_base_model.py, scripts/run_revision_known_composition_multimodel_eval.py, scripts/summarize_revision_known_composition_seed_repeats.py, scripts/run_revision_component_ablation.py, scripts/run_revision_threshold_sensitivity.py, scripts/run_revision_reference_perturbation.py, scripts/run_revision_reference_perturbation_component_ablation.py, and scripts/build_revision_figures.py. The code and revision scripts are available from the v0.2.0-revision GitHub release. The tabular source data supporting the revised figures, confidence intervals, ablation summaries, threshold analyses, and reference-perturbation stress tests are provided with the manuscript as Tables S6, S7, S8, S9, and S10.

Quantification and statistical analysis

Validation metrics, resampling, and artifact-aware interpretation

Because spot-level ground truth is unavailable, validation uses proxy conflict as anchored evidence. These proxies include layer prototype distance, Maynard prototype distance, marker discordance, and signature residual. DLPFC layer annotations provide the anchored structural reference, while marker-based and reference-based checks are standard sources of evidence for cell identity interpretation in single-cell and spatial analyses.⁶^,²³^,³³^,⁵¹ They measure disagreement between model outputs and relatively independent external evidence, making them suitable for identifying locations that require cautious interpretation.

This proxy-to-truth gap is both a limitation and the reason for the study design. If true spot-level abundance labels existed across all tissues, ReliST could be evaluated as a direct error predictor. In the realistic setting addressed here, the stronger question is whether risk consistently points to locations where multiple external validation signals advise caution. The manuscript therefore avoids converting proxies into ground truth, and instead asks whether risk improves retained proxy quality, aligns with external tissue-state evidence, and remains informative after reference-adjusted residual controls.

We evaluate risk scores with four classes of evidence. Risk-proxy AUC or correlation tests whether high-risk spots are enriched for stronger proxy conflict.⁵² Selective gain evaluates whether removing high-risk spots reduces proxy conflict in the retained set.³² Coverage-risk curves examine residual proxy burden at different retained fractions, following the broader reject-option and selective-prediction framework.³¹^,³² Reliability curves and monotonic risk-bin behavior test whether proxy conflict increases across risk bins.⁵³ We also include a controlled reference-output degradation stress test in saved DLPFC RCTD abundance outputs, applying reference dropout and label permutation to check whether output-derived risk increases with a controlled abundance-error proxy.

These four metric classes serve different functions. Risk-proxy AUC and correlation test enrichment, but they do not by themselves justify filtering. Selective gain tests whether excluding high-risk spots improves the retained subset, which is closer to the intended use case. Coverage-risk curves show how much tissue coverage must be sacrificed to reduce proxy burden. Reliability curves test whether the risk score can be read monotonically as a review priority. The controlled degradation stress test is only a necessary-condition sanity check: it asks whether risk responds in the expected direction when abundance outputs are deliberately damaged, not whether naturally high-risk spots are true errors.

For manuscript-facing metrics, we report 95% bootstrap confidence intervals.⁵⁴ DLPFC anchored validation and common-feature null control use sample-level bootstrap. Mouse brain auxiliary validation and breast cancer IF/gene-protein validation come from single sections, so they use spatial block bootstrap to reduce dependence on naive independent-spot assumptions.⁵⁵ Confidence intervals are interpreted as resampling uncertainty for proxy-level descriptive estimates, not as truth-level error probabilities or strict coverage guarantees.

In breast cancer, high-risk regions can reflect both biological heterogeneity and technical artifacts. Low H&E saturation, tissue edges, low-count spots, and barcode bias can destabilize model outputs, consistent with general single-cell and spatial-analysis quality-control concerns.³^,²¹^,²⁴^,²⁵^,²⁶^,³³ Artifact-aware interpretation is applied after scoring, categorizing high-risk spots as tissue-edge artifact, empty-tissue signal, low-saturation artifact, barcode bias, local outlier, reference mismatch, genuine heterogeneity candidate, or unresolved high risk. A high-risk region is described as a genuine heterogeneity candidate only when reference annotation, marker support, spatial-neighborhood evidence, and histologic plausibility point in the same direction.

Footnotes

Supplemental information can be found online at https://doi.org/10.1016/j.isci.2026.117206.

Contributor Information

Yifei Xu, Email: xyf2995@gzucm.edu.cn.

Sen Lin, Email: ls3994@gzucm.edu.cn.

Supplemental information

Document S1. Figures S1–S11

mmc1.pdf^{(1.5MB, pdf)}

Table S1. Base-model roles and risk-axis support tiers

Summary of evidence role, output contract, risk-axis support, and the main caveat for Cell2location, RCTD, Tangram, Stereoscope, and DestVI

mmc2.xlsx^{(4.5KB, xlsx)}

Table S2. Human DLPFC goal-based default model policy

Empirical default model, risk axis, trusted fraction, review-needed range, and primary use for reference, dominant-label, structure, and marker goals in DLPFC samples 151508 and 151670.

mmc3.xlsx^{(4.4KB, xlsx)}

Table S3. Human DLPFC canonical output-contract fairness and common-feature null control

Comparison of strict common abundance-derived scores and risk-axis scores on DLPFC validation proxies.

mmc4.xlsx^{(4.9KB, xlsx)}

Table S4. Human DLPFC RCTD controlled output-degradation stress test

Summary of reference-dropout and label-permutation tests on saved DLPFC RCTD abundance outputs.

mmc5.xlsx^{(4.5KB, xlsx)}

Table S5. Bootstrap confidence intervals for manuscript-facing evidence metrics

Bootstrap confidence intervals for DLPFC anchored validation, contract-fairness control, mouse-brain auxiliary validation, breast-cancer IF/gene-protein validation, and reference-adjusted residual control.

mmc6.xlsx^{(5.7KB, xlsx)}

Table S6. Donor-disjoint known-composition benchmark source data

(A) Cell-type inclusion criteria; (B) NNLS score-to-error summary; (C) multi-base-model selective-filtering source data; (D) multi-base-model true-error score summary; (E) bootstrap confidence intervals; (F) split-and-simulation seed-repeat check; and (G) ReliST score definitions, feature weights, and normalization groups.

mmc7.xlsx^{(68.1KB, xlsx)}

Table S7. Baseline-comparison and incremental-contribution analyses

(A) Paired-bootstrap deltas versus the best available baseline; (B) incremental contribution of ReliST after controlling the best baseline; and (C) DLPFC anchored common-feature control summary.

mmc8.xlsx^{(7KB, xlsx)}

Table S8. Component ablation summary

Full-risk, single-component, and multi-component results in the known-composition benchmark.

mmc9.xlsx^{(5.6KB, xlsx)}

Table S9. Threshold sensitivity and practical-use source data

(A) Keep-fraction sensitivity with random and oracle references; (B) practical-use guidance across operating points; and (C) high-risk capture and precision.

mmc10.xlsx^{(17.9KB, xlsx)}

Table S10. Reference-perturbation analyses

(A) Reference-perturbation stress-test summary and (B) full versus no-reference score comparison under reference perturbations.

mmc11.xlsx^{(7.5KB, xlsx)}

References

1.Ståhl P.L., Salmén F., Vickovic S., Lundmark A., Navarro J.F., Magnusson J., Giacomello S., Asp M., Westholm J.O., Huss M., et al. Visualization and analysis of gene expression in tissue sections by spatial transcriptomics. Science. 2016;353:78–82. doi: 10.1126/science.aaf2403. [DOI] [PubMed] [Google Scholar]
2.Marx V. Method of the Year: spatially resolved transcriptomics. Nat. Methods. 2021;18:9–14. doi: 10.1038/s41592-020-01033-y. [DOI] [PubMed] [Google Scholar]
3.Moses L., Pachter L. Museum of spatial transcriptomics. Nat. Methods. 2022;19:534–546. doi: 10.1038/s41592-022-01409-2. [DOI] [PubMed] [Google Scholar]
4.Yue L., Liu F., Hu J., Yang P., Wang Y., Dong J., Shu W., Huang X., Wang S. A guidebook of spatial transcriptomic technologies, data resources and analysis approaches. Comput. Struct. Biotechnol. J. 2023;21:940–955. doi: 10.1016/j.csbj.2023.01.016. [DOI] [PMC free article] [PubMed] [Google Scholar]
5.Gulati G.S., D'Silva J.P., Liu Y., Wang L., Newman A.M. Profiling cell identity and tissue architecture with single-cell and spatial transcriptomics. Nat. Rev. Mol. Cell Biol. 2025;26:11–31. doi: 10.1038/s41580-024-00768-2. [DOI] [PubMed] [Google Scholar]
6.Stuart T., Butler A., Hoffman P., Hafemeister C., Papalexi E., Mauck W.M., III, Hao Y., Stoeckius M., Smibert P., Satija R. Comprehensive Integration of Single-Cell Data. Cell. 2019;177:1888–1902.e21. doi: 10.1016/j.cell.2019.05.031. [DOI] [PMC free article] [PubMed] [Google Scholar]
7.Li B., Zhang W., Guo C., Xu H., Li L., Fang M., Hu Y., Zhang X., Yao X., Tang M., et al. Benchmarking spatial and single-cell transcriptomics integration methods for transcript distribution prediction and cell type deconvolution. Nat. Methods. 2022;19:662–670. doi: 10.1038/s41592-022-01480-9. [DOI] [PubMed] [Google Scholar]
8.Li H., Zhou J., Li Z., Chen S., Liao X., Zhang B., Zhang R., Wang Y., Sun S., Gao X. A comprehensive benchmarking with practical guidelines for cellular deconvolution of spatial transcriptomics. Nat. Commun. 2023;14:1548. doi: 10.1038/s41467-023-37168-7. [DOI] [PMC free article] [PubMed] [Google Scholar]
9.Andersson A., Bergenstråhle J., Asp M., Bergenstråhle L., Jurek A., Fernández Navarro J., Lundeberg J. Single-cell and spatial transcriptomics enables probabilistic inference of cell type topography. Commun. Biol. 2020;3:565. doi: 10.1038/s42003-020-01247-y. [DOI] [PMC free article] [PubMed] [Google Scholar]
10.Cable D.M., Murray E., Zou L.S., Goeva A., Macosko E.Z., Chen F., Irizarry R.A. Robust decomposition of cell type mixtures in spatial transcriptomics. Nat. Biotechnol. 2022;40:517–526. doi: 10.1038/s41587-021-00830-w. [DOI] [PMC free article] [PubMed] [Google Scholar]
11.Biancalani T., Scalia G., Buffoni L., Avasthi R., Lu Z., Sanger A., Tokcan N., Vanderburg C.R., Segerstolpe Å., Zhang M., et al. Deep learning and alignment of spatially resolved single-cell transcriptomes with Tangram. Nat. Methods. 2021;18:1352–1362. doi: 10.1038/s41592-021-01264-7. [DOI] [PMC free article] [PubMed] [Google Scholar]
12.Kleshchevnikov V., Shmatko A., Dann E., Aivazidis A., King H.W., Li T., Elmentaite R., Lomakin A., Kedlian V., Gayoso A., et al. Cell2location maps fine-grained cell types in spatial transcriptomics. Nat. Biotechnol. 2022;40:661–671. doi: 10.1038/s41587-021-01139-4. [DOI] [PubMed] [Google Scholar]
13.Lopez R., Li B., Keren-Shaul H., Boyeau P., Kedmi M., Pilzer D., Jelinski A., Yofe I., David E., Wagner A., et al. DestVI identifies continuums of cell types in spatial transcriptomics data. Nat. Biotechnol. 2022;40:1360–1369. doi: 10.1038/s41587-022-01272-8. [DOI] [PMC free article] [PubMed] [Google Scholar]
14.Elosua-Bayes M., Nieto P., Mereu E., Gut I., Heyn H. SPOTlight: seeded NMF regression to deconvolute spatial transcriptomics spots with single-cell transcriptomes. Nucleic Acids Res. 2021;49 doi: 10.1093/nar/gkab043. [DOI] [PMC free article] [PubMed] [Google Scholar]
15.Ma Y., Zhou X. Spatially informed cell-type deconvolution for spatial transcriptomics. Nat. Biotechnol. 2022;40:1349–1359. doi: 10.1038/s41587-022-01273-7. [DOI] [PMC free article] [PubMed] [Google Scholar]
16.Dries R., Zhu Q., Dong R., Eng C.H.L., Li H., Liu K., Fu Y., Zhao T., Sarkar A., Bao F., et al. Giotto: a toolbox for integrative analysis and visualization of spatial expression data. Genome Biol. 2021;22:78. doi: 10.1186/s13059-021-02286-2. [DOI] [PMC free article] [PubMed] [Google Scholar]
17.Palla G., Spitzer H., Klein M., Fischer D., Schaar A.C., Kuemmerle L.B., Rybakov S., Ibarra I.L., Holmberg O., Virshup I., et al. Squidpy: a scalable framework for spatial omics analysis. Nat. Methods. 2022;19:171–178. doi: 10.1038/s41592-021-01358-2. [DOI] [PMC free article] [PubMed] [Google Scholar]
18.Bergenstråhle J., Larsson L., Lundeberg J. Seamless integration of image and molecular analysis for spatial transcriptomics workflows. BMC Genom. 2020;21:482. doi: 10.1186/s12864-020-06832-3. [DOI] [PMC free article] [PubMed] [Google Scholar]
19.Zeira R., Land M., Strzalkowski A., Raphael B.J. Alignment and integration of spatial transcriptomics data. Nat. Methods. 2022;19:567–575. doi: 10.1038/s41592-022-01459-6. [DOI] [PMC free article] [PubMed] [Google Scholar]
20.Zhao E., Stone M.R., Ren X., Guenthoer J., Smythe K.S., Pulliam T., Williams S.R., Uytingco C.R., Taylor S.E.B., Nghiem P., et al. Spatial transcriptomics at subspot resolution with BayesSpace. Nat. Biotechnol. 2021;39:1375–1384. doi: 10.1038/s41587-021-00935-2. [DOI] [PMC free article] [PubMed] [Google Scholar]
21.Lähnemann D., Köster J., Szczurek E., McCarthy D.J., Hicks S.C., Robinson M.D., Vallejos C.A., Campbell K.R., Beerenwinkel N., Mahfouz A., et al. Eleven grand challenges in single-cell data science. Genome Biol. 2020;21:31. doi: 10.1186/s13059-020-1926-6. [DOI] [PMC free article] [PubMed] [Google Scholar]
22.Kiselev V.Y., Andrews T.S., Hemberg M. Challenges in unsupervised clustering of single-cell RNA-seq data. Nat. Rev. Genet. 2019;20:273–282. doi: 10.1038/s41576-018-0088-9. [DOI] [PubMed] [Google Scholar]
23.Abdelaal T., Michielsen L., Cats D., Hoogduin D., Mei H., Reinders M.J.T., Mahfouz A. A comparison of automatic cell identification methods for single-cell RNA sequencing data. Genome Biol. 2019;20:194. doi: 10.1186/s13059-019-1795-z. [DOI] [PMC free article] [PubMed] [Google Scholar]
24.Ni Z., Prasad A., Chen S., Halberg R.B., Arkin L.M., Drolet B.A., Newton M.A., Kendziorski C. SpotClean adjusts for spot swapping in spatial transcriptomics data. Nat. Commun. 2022;13:2971. doi: 10.1038/s41467-022-30587-y. [DOI] [PMC free article] [PubMed] [Google Scholar]
25.Totty M., Hicks S.C., Guo B. SpotSweeper: spatially aware quality control for spatial transcriptomics. Nat. Methods. 2025;22:1520–1530. doi: 10.1038/s41592-025-02713-3. [DOI] [PMC free article] [PubMed] [Google Scholar]
26.Mao G., Yang Y., Luo Z., Lin C., Xie P. SpatialQC: automated quality control for spatial transcriptome data. Bioinformatics. 2024;40 doi: 10.1093/bioinformatics/btae458. [DOI] [PMC free article] [PubMed] [Google Scholar]
27.Abdar M., Pourpanah F., Hussain S., Rezazadegan D., Liu L., Ghavamzadeh M., Fieguth P., Cao X., Khosravi A., Acharya U.R., et al. A review of uncertainty quantification in deep learning: Techniques, applications and challenges. Inf. Fusion. 2021;76:243–297. doi: 10.1016/j.inffus.2021.05.008. [DOI] [Google Scholar]
28.Gawlikowski J., Tassi C.R.N., Ali M., Lee J., Humt M., Feng J., Kruspe A., Triebel R., Jung P., Roscher R., et al. A survey of uncertainty in deep neural networks. Artif. Intell. Rev. 2023;56:1513–1589. doi: 10.1007/s10462-023-10562-9. [DOI] [Google Scholar]
29.Vovk V., Gammerman A., Shafer G. Springer; 2005. Algorithmic Learning in a Random World. [Google Scholar]
30.Angelopoulos A.N., Bates S. Conformal prediction: A gentle introduction. Foundations and Trends in Machine Learning. 2023;16:494–591. doi: 10.1561/2200000101. [DOI] [Google Scholar]
31.Chow C. On optimum recognition error and reject tradeoff. IEEE Trans. Inf. Theor. 1970;16:41–46. doi: 10.1109/tit.1970.1054406. [DOI] [Google Scholar]
32.Geifman Y., El-Yaniv R. Selective classification for deep neural networks. arXiv. 2017 doi: 10.48550/arXiv.1705.08500. Preprint at. [DOI] [Google Scholar]
33.Maynard K.R., Collado-Torres L., Weber L.M., Uytingco C., Barry B.K., Williams S.R., Catallini J.L., Tran M.N., Besich Z., Tippani M., et al. Transcriptome-scale spatial gene expression in the human dorsolateral prefrontal cortex. Nat. Neurosci. 2021;24:425–436. doi: 10.1038/s41593-020-00787-0. [DOI] [PMC free article] [PubMed] [Google Scholar]
34.10x Genomics. Mouse Brain Serial Section 1 (Sagittal-Anterior): Spatial Gene Expression dataset analyzed using Space Ranger 1.0.0. https://www.10xgenomics.com/datasets/mouse-brain-serial-section-1-sagittal-anterior-1-standard-1-0-0.
35.Yao Z., van Velthoven C.T.J., Nguyen T.N., Goldy J., Sedeno-Cortes A.E., Baftizadeh F., Bertagnolli D., Casper T., Chiang M., Crichton K., et al. A taxonomy of transcriptomic cell types across the isocortex and hippocampal formation. Cell. 2021;184:3222–3241.e26. doi: 10.1016/j.cell.2021.04.021. [DOI] [PMC free article] [PubMed] [Google Scholar]
36.10x Genomics. Human Breast Cancer (Block A Section 1): Visium Spatial Gene Expression Dataset. Accessed May 9, 2026, https://www.10xgenomics.com/datasets/human-breast-cancer-block-a-section-1-1-standard-1-1-0.
37.Pal B., Chen Y., Vaillant F., Capaldo B.D., Joyce R., Song X., Bryant V.L., Penington J.S., Di Stefano L., Tubau Ribera N., et al. A single-cell RNA expression atlas of normal, preneoplastic and tumorigenic states in the human breast. EMBO J. 2021;40 doi: 10.15252/embj.2020107333. [DOI] [PMC free article] [PubMed] [Google Scholar]
38.Jackson H.W., Fischer J.R., Zanotelli V.R.T., Ali H.R., Mechera R., Soysal S.D., Moch H., Muenst S., Varga Z., Weber W.P., Bodenmiller B. The single-cell pathology landscape of breast cancer. Nature. 2020;578:615–620. doi: 10.1038/s41586-019-1876-x. [DOI] [PubMed] [Google Scholar]
39.Wu S.Z., Al-Eryani G., Roden D.L., Junankar S., Harvey K., Andersson A., Thennavan A., Wang C., Torpy J.R., Bartonicek N., et al. A single-cell and spatially resolved atlas of human breast cancers. Nat. Genet. 2021;53:1334–1347. doi: 10.1038/s41588-021-00911-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
40.Keren L., Bosse M., Marquez D., Angoshtari R., Jain S., Varma S., Yang S.R., Kurian A., Van Valen D., West R., et al. A Structured Tumor-Immune Microenvironment in Triple Negative Breast Cancer Revealed by Multiplexed Ion Beam Imaging. Cell. 2018;174:1373–1387.e19. doi: 10.1016/j.cell.2018.08.039. [DOI] [PMC free article] [PubMed] [Google Scholar]
41.Risom T., Glass D.R., Averbukh I., Liu C.C., Baranski A., Kagel A., McCaffrey E.F., Greenwald N.F., Rivero-Gutiérrez B., Strand S.H., et al. Transition to invasive breast cancer is associated with progressive changes in the structure and composition of tumor stroma. Cell. 2022;185:299–310.e18. doi: 10.1016/j.cell.2021.12.023. [DOI] [PMC free article] [PubMed] [Google Scholar]
42.10x Genomics. Visium CytAssist Gene and Protein Expression Library of Human Breast Cancer, IF, 6.5 mm (FFPE). Accessed May 9, 2026, https://www.10xgenomics.com/datasets/gene-and-protein-expression-library-of-human-breast-cancer-cytassist-ffpe-2-standard.
43.Stoeckius M., Hafemeister C., Stephenson W., Houck-Loomis B., Chattopadhyay P.K., Swerdlow H., Satija R., Smibert P. Simultaneous epitope and transcriptome measurement in single cells. Nat. Methods. 2017;14:865–868. doi: 10.1038/nmeth.4380. [DOI] [PMC free article] [PubMed] [Google Scholar]
44.Merritt C.R., Ong G.T., Church S.E., Barker K., Danaher P., Geiss G., Hoang M., Jung J., Liang Y., McKay-Fleisch J., et al. Multiplex digital spatial profiling of proteins and RNA in fixed tissue. Nat. Biotechnol. 2020;38:586–599. doi: 10.1038/s41587-020-0472-9. [DOI] [PubMed] [Google Scholar]
45.10x Genomics. HD Spatial Gene Expression. Accessed May 9, 2026, https://www.10xgenomics.com/support/spatial-gene-expression-hd.
46.Chen K.H., Boettiger A.N., Moffitt J.R., Wang S., Zhuang X. RNA imaging. Spatially resolved, highly multiplexed RNA profiling in single cells. Science. 2015;348 doi: 10.1126/science.aaa6090. [DOI] [PMC free article] [PubMed] [Google Scholar]
47.Eng C.H.L., Lawson M., Zhu Q., Dries R., Koulena N., Takei Y., Yun J., Cronin C., Karp C., Yuan G.C., Cai L. Transcriptome-scale super-resolved imaging in tissues by RNA seqFISH. Nature. 2019;568:235–239. doi: 10.1038/s41586-019-1049-y. [DOI] [PMC free article] [PubMed] [Google Scholar]
48.Stickels R.R., Murray E., Kumar P., Li J., Marshall J.L., Di Bella D.J., Arlotta P., Macosko E.Z., Chen F. Highly sensitive spatial transcriptomics at near-cellular resolution with Slide-seqV2. Nat. Biotechnol. 2021;39:313–319. doi: 10.1038/s41587-020-0739-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
49.Rodriques S.G., Stickels R.R., Goeva A., Martin C.A., Murray E., Vanderburg C.R., Welch J., Chen L.M., Chen F., Macosko E.Z. Slide-seq: A scalable technology for measuring genome-wide expression at high spatial resolution. Science. 2019;363:1463–1467. doi: 10.1126/science.aaw1219. [DOI] [PMC free article] [PubMed] [Google Scholar]
50.Righelli D., Weber L.M., Crowell H.L., Pardo B., Collado-Torres L., Ghazanfar S., Lun A.T.L., Hicks S.C., Risso D. SpatialExperiment: infrastructure for spatially-resolved transcriptomics data in R using Bioconductor. Bioinformatics. 2022;38:3128–3131. doi: 10.1093/bioinformatics/btac299. [DOI] [PMC free article] [PubMed] [Google Scholar]
51.Heumos L., Schaar A.C., Lance C., Litinetskaya A., Drost F., Zappia L., Lücken M.D., Strobl D.C., Henao J., Curion F., et al. Best practices for single-cell analysis across modalities. Nat. Rev. Genet. 2023;24:550–572. doi: 10.1038/s41576-023-00586-w. [DOI] [PMC free article] [PubMed] [Google Scholar]
52.Fawcett T. An introduction to ROC analysis. Pattern Recognit. Lett. 2006;27:861–874. doi: 10.1016/j.patrec.2005.10.010. [DOI] [Google Scholar]
53.Guo C., Pleiss G., Sun Y., Weinberger K.Q. PMLR; 2017. On Calibration of Modern Neural Networks; pp. 1321–1330. [Google Scholar]
54.Efron B., Tibshirani R.J. Chapman & Hall; 1993. An Introduction to the Bootstrap. [Google Scholar]
55.Lahiri S.N. Springer Science & Business Media; 2013. Resampling Methods for Dependent Data. [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Document S1. Figures S1–S11

mmc1.pdf^{(1.5MB, pdf)}

Table S1. Base-model roles and risk-axis support tiers

Summary of evidence role, output contract, risk-axis support, and the main caveat for Cell2location, RCTD, Tangram, Stereoscope, and DestVI

mmc2.xlsx^{(4.5KB, xlsx)}

Table S2. Human DLPFC goal-based default model policy

Empirical default model, risk axis, trusted fraction, review-needed range, and primary use for reference, dominant-label, structure, and marker goals in DLPFC samples 151508 and 151670.

mmc3.xlsx^{(4.4KB, xlsx)}

Table S3. Human DLPFC canonical output-contract fairness and common-feature null control

Comparison of strict common abundance-derived scores and risk-axis scores on DLPFC validation proxies.

mmc4.xlsx^{(4.9KB, xlsx)}

Table S4. Human DLPFC RCTD controlled output-degradation stress test

Summary of reference-dropout and label-permutation tests on saved DLPFC RCTD abundance outputs.

mmc5.xlsx^{(4.5KB, xlsx)}

Table S5. Bootstrap confidence intervals for manuscript-facing evidence metrics

mmc6.xlsx^{(5.7KB, xlsx)}

Table S6. Donor-disjoint known-composition benchmark source data

mmc7.xlsx^{(68.1KB, xlsx)}

Table S7. Baseline-comparison and incremental-contribution analyses

(A) Paired-bootstrap deltas versus the best available baseline; (B) incremental contribution of ReliST after controlling the best baseline; and (C) DLPFC anchored common-feature control summary.

mmc8.xlsx^{(7KB, xlsx)}

Table S8. Component ablation summary

Full-risk, single-component, and multi-component results in the known-composition benchmark.

mmc9.xlsx^{(5.6KB, xlsx)}

Table S9. Threshold sensitivity and practical-use source data

(A) Keep-fraction sensitivity with random and oracle references; (B) practical-use guidance across operating points; and (C) high-risk capture and precision.

mmc10.xlsx^{(17.9KB, xlsx)}

Table S10. Reference-perturbation analyses

(A) Reference-perturbation stress-test summary and (B) full versus no-reference score comparison under reference perturbations.

mmc11.xlsx^{(7.5KB, xlsx)}

Data Availability Statement

•
This study analyzes publicly available spatial transcriptomics, single-cell reference, histology, antibody-capture, and immunofluorescence datasets. Accession numbers and source links are listed in the key resources table. The tabular source data supporting the revision analyses are provided in Tables S6, S7, S8, S9, and S10.
•
The ReliST source code, analysis scripts, selected-run metadata, and manuscript figure-generation scripts are publicly available at https://github.com/tp5353/ReliST under the MIT license. The revision-specific code and scripts are frozen in the GitHub tag and release v0.2.0-revision (https://github.com/tp5353/ReliST/releases/tag/v0.2.0-revision).
•
Any additional information required to reanalyze the data reported in this paper is available from the lead contact upon request.

[bib1] 1.Ståhl P.L., Salmén F., Vickovic S., Lundmark A., Navarro J.F., Magnusson J., Giacomello S., Asp M., Westholm J.O., Huss M., et al. Visualization and analysis of gene expression in tissue sections by spatial transcriptomics. Science. 2016;353:78–82. doi: 10.1126/science.aaf2403. [DOI] [PubMed] [Google Scholar]

[bib2] 2.Marx V. Method of the Year: spatially resolved transcriptomics. Nat. Methods. 2021;18:9–14. doi: 10.1038/s41592-020-01033-y. [DOI] [PubMed] [Google Scholar]

[bib3] 3.Moses L., Pachter L. Museum of spatial transcriptomics. Nat. Methods. 2022;19:534–546. doi: 10.1038/s41592-022-01409-2. [DOI] [PubMed] [Google Scholar]

[bib4] 4.Yue L., Liu F., Hu J., Yang P., Wang Y., Dong J., Shu W., Huang X., Wang S. A guidebook of spatial transcriptomic technologies, data resources and analysis approaches. Comput. Struct. Biotechnol. J. 2023;21:940–955. doi: 10.1016/j.csbj.2023.01.016. [DOI] [PMC free article] [PubMed] [Google Scholar]

[bib5] 5.Gulati G.S., D'Silva J.P., Liu Y., Wang L., Newman A.M. Profiling cell identity and tissue architecture with single-cell and spatial transcriptomics. Nat. Rev. Mol. Cell Biol. 2025;26:11–31. doi: 10.1038/s41580-024-00768-2. [DOI] [PubMed] [Google Scholar]

[bib6] 6.Stuart T., Butler A., Hoffman P., Hafemeister C., Papalexi E., Mauck W.M., III, Hao Y., Stoeckius M., Smibert P., Satija R. Comprehensive Integration of Single-Cell Data. Cell. 2019;177:1888–1902.e21. doi: 10.1016/j.cell.2019.05.031. [DOI] [PMC free article] [PubMed] [Google Scholar]

[bib7] 7.Li B., Zhang W., Guo C., Xu H., Li L., Fang M., Hu Y., Zhang X., Yao X., Tang M., et al. Benchmarking spatial and single-cell transcriptomics integration methods for transcript distribution prediction and cell type deconvolution. Nat. Methods. 2022;19:662–670. doi: 10.1038/s41592-022-01480-9. [DOI] [PubMed] [Google Scholar]

[bib8] 8.Li H., Zhou J., Li Z., Chen S., Liao X., Zhang B., Zhang R., Wang Y., Sun S., Gao X. A comprehensive benchmarking with practical guidelines for cellular deconvolution of spatial transcriptomics. Nat. Commun. 2023;14:1548. doi: 10.1038/s41467-023-37168-7. [DOI] [PMC free article] [PubMed] [Google Scholar]

[bib9] 9.Andersson A., Bergenstråhle J., Asp M., Bergenstråhle L., Jurek A., Fernández Navarro J., Lundeberg J. Single-cell and spatial transcriptomics enables probabilistic inference of cell type topography. Commun. Biol. 2020;3:565. doi: 10.1038/s42003-020-01247-y. [DOI] [PMC free article] [PubMed] [Google Scholar]

[bib10] 10.Cable D.M., Murray E., Zou L.S., Goeva A., Macosko E.Z., Chen F., Irizarry R.A. Robust decomposition of cell type mixtures in spatial transcriptomics. Nat. Biotechnol. 2022;40:517–526. doi: 10.1038/s41587-021-00830-w. [DOI] [PMC free article] [PubMed] [Google Scholar]

[bib11] 11.Biancalani T., Scalia G., Buffoni L., Avasthi R., Lu Z., Sanger A., Tokcan N., Vanderburg C.R., Segerstolpe Å., Zhang M., et al. Deep learning and alignment of spatially resolved single-cell transcriptomes with Tangram. Nat. Methods. 2021;18:1352–1362. doi: 10.1038/s41592-021-01264-7. [DOI] [PMC free article] [PubMed] [Google Scholar]

[bib12] 12.Kleshchevnikov V., Shmatko A., Dann E., Aivazidis A., King H.W., Li T., Elmentaite R., Lomakin A., Kedlian V., Gayoso A., et al. Cell2location maps fine-grained cell types in spatial transcriptomics. Nat. Biotechnol. 2022;40:661–671. doi: 10.1038/s41587-021-01139-4. [DOI] [PubMed] [Google Scholar]

[bib13] 13.Lopez R., Li B., Keren-Shaul H., Boyeau P., Kedmi M., Pilzer D., Jelinski A., Yofe I., David E., Wagner A., et al. DestVI identifies continuums of cell types in spatial transcriptomics data. Nat. Biotechnol. 2022;40:1360–1369. doi: 10.1038/s41587-022-01272-8. [DOI] [PMC free article] [PubMed] [Google Scholar]

[bib14] 14.Elosua-Bayes M., Nieto P., Mereu E., Gut I., Heyn H. SPOTlight: seeded NMF regression to deconvolute spatial transcriptomics spots with single-cell transcriptomes. Nucleic Acids Res. 2021;49 doi: 10.1093/nar/gkab043. [DOI] [PMC free article] [PubMed] [Google Scholar]

[bib15] 15.Ma Y., Zhou X. Spatially informed cell-type deconvolution for spatial transcriptomics. Nat. Biotechnol. 2022;40:1349–1359. doi: 10.1038/s41587-022-01273-7. [DOI] [PMC free article] [PubMed] [Google Scholar]

[bib16] 16.Dries R., Zhu Q., Dong R., Eng C.H.L., Li H., Liu K., Fu Y., Zhao T., Sarkar A., Bao F., et al. Giotto: a toolbox for integrative analysis and visualization of spatial expression data. Genome Biol. 2021;22:78. doi: 10.1186/s13059-021-02286-2. [DOI] [PMC free article] [PubMed] [Google Scholar]

[bib17] 17.Palla G., Spitzer H., Klein M., Fischer D., Schaar A.C., Kuemmerle L.B., Rybakov S., Ibarra I.L., Holmberg O., Virshup I., et al. Squidpy: a scalable framework for spatial omics analysis. Nat. Methods. 2022;19:171–178. doi: 10.1038/s41592-021-01358-2. [DOI] [PMC free article] [PubMed] [Google Scholar]

[bib18] 18.Bergenstråhle J., Larsson L., Lundeberg J. Seamless integration of image and molecular analysis for spatial transcriptomics workflows. BMC Genom. 2020;21:482. doi: 10.1186/s12864-020-06832-3. [DOI] [PMC free article] [PubMed] [Google Scholar]

[bib19] 19.Zeira R., Land M., Strzalkowski A., Raphael B.J. Alignment and integration of spatial transcriptomics data. Nat. Methods. 2022;19:567–575. doi: 10.1038/s41592-022-01459-6. [DOI] [PMC free article] [PubMed] [Google Scholar]

[bib20] 20.Zhao E., Stone M.R., Ren X., Guenthoer J., Smythe K.S., Pulliam T., Williams S.R., Uytingco C.R., Taylor S.E.B., Nghiem P., et al. Spatial transcriptomics at subspot resolution with BayesSpace. Nat. Biotechnol. 2021;39:1375–1384. doi: 10.1038/s41587-021-00935-2. [DOI] [PMC free article] [PubMed] [Google Scholar]

[bib21] 21.Lähnemann D., Köster J., Szczurek E., McCarthy D.J., Hicks S.C., Robinson M.D., Vallejos C.A., Campbell K.R., Beerenwinkel N., Mahfouz A., et al. Eleven grand challenges in single-cell data science. Genome Biol. 2020;21:31. doi: 10.1186/s13059-020-1926-6. [DOI] [PMC free article] [PubMed] [Google Scholar]

[bib22] 22.Kiselev V.Y., Andrews T.S., Hemberg M. Challenges in unsupervised clustering of single-cell RNA-seq data. Nat. Rev. Genet. 2019;20:273–282. doi: 10.1038/s41576-018-0088-9. [DOI] [PubMed] [Google Scholar]

[bib23] 23.Abdelaal T., Michielsen L., Cats D., Hoogduin D., Mei H., Reinders M.J.T., Mahfouz A. A comparison of automatic cell identification methods for single-cell RNA sequencing data. Genome Biol. 2019;20:194. doi: 10.1186/s13059-019-1795-z. [DOI] [PMC free article] [PubMed] [Google Scholar]

[bib24] 24.Ni Z., Prasad A., Chen S., Halberg R.B., Arkin L.M., Drolet B.A., Newton M.A., Kendziorski C. SpotClean adjusts for spot swapping in spatial transcriptomics data. Nat. Commun. 2022;13:2971. doi: 10.1038/s41467-022-30587-y. [DOI] [PMC free article] [PubMed] [Google Scholar]

[bib25] 25.Totty M., Hicks S.C., Guo B. SpotSweeper: spatially aware quality control for spatial transcriptomics. Nat. Methods. 2025;22:1520–1530. doi: 10.1038/s41592-025-02713-3. [DOI] [PMC free article] [PubMed] [Google Scholar]

[bib26] 26.Mao G., Yang Y., Luo Z., Lin C., Xie P. SpatialQC: automated quality control for spatial transcriptome data. Bioinformatics. 2024;40 doi: 10.1093/bioinformatics/btae458. [DOI] [PMC free article] [PubMed] [Google Scholar]

[bib27] 27.Abdar M., Pourpanah F., Hussain S., Rezazadegan D., Liu L., Ghavamzadeh M., Fieguth P., Cao X., Khosravi A., Acharya U.R., et al. A review of uncertainty quantification in deep learning: Techniques, applications and challenges. Inf. Fusion. 2021;76:243–297. doi: 10.1016/j.inffus.2021.05.008. [DOI] [Google Scholar]

[bib28] 28.Gawlikowski J., Tassi C.R.N., Ali M., Lee J., Humt M., Feng J., Kruspe A., Triebel R., Jung P., Roscher R., et al. A survey of uncertainty in deep neural networks. Artif. Intell. Rev. 2023;56:1513–1589. doi: 10.1007/s10462-023-10562-9. [DOI] [Google Scholar]

[bib29] 29.Vovk V., Gammerman A., Shafer G. Springer; 2005. Algorithmic Learning in a Random World. [Google Scholar]

[bib30] 30.Angelopoulos A.N., Bates S. Conformal prediction: A gentle introduction. Foundations and Trends in Machine Learning. 2023;16:494–591. doi: 10.1561/2200000101. [DOI] [Google Scholar]

[bib31] 31.Chow C. On optimum recognition error and reject tradeoff. IEEE Trans. Inf. Theor. 1970;16:41–46. doi: 10.1109/tit.1970.1054406. [DOI] [Google Scholar]

[bib32] 32.Geifman Y., El-Yaniv R. Selective classification for deep neural networks. arXiv. 2017 doi: 10.48550/arXiv.1705.08500. Preprint at. [DOI] [Google Scholar]

[bib33] 33.Maynard K.R., Collado-Torres L., Weber L.M., Uytingco C., Barry B.K., Williams S.R., Catallini J.L., Tran M.N., Besich Z., Tippani M., et al. Transcriptome-scale spatial gene expression in the human dorsolateral prefrontal cortex. Nat. Neurosci. 2021;24:425–436. doi: 10.1038/s41593-020-00787-0. [DOI] [PMC free article] [PubMed] [Google Scholar]

[bib34] 34.10x Genomics. Mouse Brain Serial Section 1 (Sagittal-Anterior): Spatial Gene Expression dataset analyzed using Space Ranger 1.0.0. https://www.10xgenomics.com/datasets/mouse-brain-serial-section-1-sagittal-anterior-1-standard-1-0-0.

[bib35] 35.Yao Z., van Velthoven C.T.J., Nguyen T.N., Goldy J., Sedeno-Cortes A.E., Baftizadeh F., Bertagnolli D., Casper T., Chiang M., Crichton K., et al. A taxonomy of transcriptomic cell types across the isocortex and hippocampal formation. Cell. 2021;184:3222–3241.e26. doi: 10.1016/j.cell.2021.04.021. [DOI] [PMC free article] [PubMed] [Google Scholar]

[bib36] 36.10x Genomics. Human Breast Cancer (Block A Section 1): Visium Spatial Gene Expression Dataset. Accessed May 9, 2026, https://www.10xgenomics.com/datasets/human-breast-cancer-block-a-section-1-1-standard-1-1-0.

[bib37] 37.Pal B., Chen Y., Vaillant F., Capaldo B.D., Joyce R., Song X., Bryant V.L., Penington J.S., Di Stefano L., Tubau Ribera N., et al. A single-cell RNA expression atlas of normal, preneoplastic and tumorigenic states in the human breast. EMBO J. 2021;40 doi: 10.15252/embj.2020107333. [DOI] [PMC free article] [PubMed] [Google Scholar]

[bib38] 38.Jackson H.W., Fischer J.R., Zanotelli V.R.T., Ali H.R., Mechera R., Soysal S.D., Moch H., Muenst S., Varga Z., Weber W.P., Bodenmiller B. The single-cell pathology landscape of breast cancer. Nature. 2020;578:615–620. doi: 10.1038/s41586-019-1876-x. [DOI] [PubMed] [Google Scholar]

[bib39] 39.Wu S.Z., Al-Eryani G., Roden D.L., Junankar S., Harvey K., Andersson A., Thennavan A., Wang C., Torpy J.R., Bartonicek N., et al. A single-cell and spatially resolved atlas of human breast cancers. Nat. Genet. 2021;53:1334–1347. doi: 10.1038/s41588-021-00911-1. [DOI] [PMC free article] [PubMed] [Google Scholar]

[bib40] 40.Keren L., Bosse M., Marquez D., Angoshtari R., Jain S., Varma S., Yang S.R., Kurian A., Van Valen D., West R., et al. A Structured Tumor-Immune Microenvironment in Triple Negative Breast Cancer Revealed by Multiplexed Ion Beam Imaging. Cell. 2018;174:1373–1387.e19. doi: 10.1016/j.cell.2018.08.039. [DOI] [PMC free article] [PubMed] [Google Scholar]

[bib41] 41.Risom T., Glass D.R., Averbukh I., Liu C.C., Baranski A., Kagel A., McCaffrey E.F., Greenwald N.F., Rivero-Gutiérrez B., Strand S.H., et al. Transition to invasive breast cancer is associated with progressive changes in the structure and composition of tumor stroma. Cell. 2022;185:299–310.e18. doi: 10.1016/j.cell.2021.12.023. [DOI] [PMC free article] [PubMed] [Google Scholar]

[bib42] 42.10x Genomics. Visium CytAssist Gene and Protein Expression Library of Human Breast Cancer, IF, 6.5 mm (FFPE). Accessed May 9, 2026, https://www.10xgenomics.com/datasets/gene-and-protein-expression-library-of-human-breast-cancer-cytassist-ffpe-2-standard.

[bib43] 43.Stoeckius M., Hafemeister C., Stephenson W., Houck-Loomis B., Chattopadhyay P.K., Swerdlow H., Satija R., Smibert P. Simultaneous epitope and transcriptome measurement in single cells. Nat. Methods. 2017;14:865–868. doi: 10.1038/nmeth.4380. [DOI] [PMC free article] [PubMed] [Google Scholar]

[bib44] 44.Merritt C.R., Ong G.T., Church S.E., Barker K., Danaher P., Geiss G., Hoang M., Jung J., Liang Y., McKay-Fleisch J., et al. Multiplex digital spatial profiling of proteins and RNA in fixed tissue. Nat. Biotechnol. 2020;38:586–599. doi: 10.1038/s41587-020-0472-9. [DOI] [PubMed] [Google Scholar]

[bib45] 45.10x Genomics. HD Spatial Gene Expression. Accessed May 9, 2026, https://www.10xgenomics.com/support/spatial-gene-expression-hd.

[bib46] 46.Chen K.H., Boettiger A.N., Moffitt J.R., Wang S., Zhuang X. RNA imaging. Spatially resolved, highly multiplexed RNA profiling in single cells. Science. 2015;348 doi: 10.1126/science.aaa6090. [DOI] [PMC free article] [PubMed] [Google Scholar]

[bib47] 47.Eng C.H.L., Lawson M., Zhu Q., Dries R., Koulena N., Takei Y., Yun J., Cronin C., Karp C., Yuan G.C., Cai L. Transcriptome-scale super-resolved imaging in tissues by RNA seqFISH. Nature. 2019;568:235–239. doi: 10.1038/s41586-019-1049-y. [DOI] [PMC free article] [PubMed] [Google Scholar]

[bib48] 48.Stickels R.R., Murray E., Kumar P., Li J., Marshall J.L., Di Bella D.J., Arlotta P., Macosko E.Z., Chen F. Highly sensitive spatial transcriptomics at near-cellular resolution with Slide-seqV2. Nat. Biotechnol. 2021;39:313–319. doi: 10.1038/s41587-020-0739-1. [DOI] [PMC free article] [PubMed] [Google Scholar]

[bib49] 49.Rodriques S.G., Stickels R.R., Goeva A., Martin C.A., Murray E., Vanderburg C.R., Welch J., Chen L.M., Chen F., Macosko E.Z. Slide-seq: A scalable technology for measuring genome-wide expression at high spatial resolution. Science. 2019;363:1463–1467. doi: 10.1126/science.aaw1219. [DOI] [PMC free article] [PubMed] [Google Scholar]

[bib50] 50.Righelli D., Weber L.M., Crowell H.L., Pardo B., Collado-Torres L., Ghazanfar S., Lun A.T.L., Hicks S.C., Risso D. SpatialExperiment: infrastructure for spatially-resolved transcriptomics data in R using Bioconductor. Bioinformatics. 2022;38:3128–3131. doi: 10.1093/bioinformatics/btac299. [DOI] [PMC free article] [PubMed] [Google Scholar]

[bib51] 51.Heumos L., Schaar A.C., Lance C., Litinetskaya A., Drost F., Zappia L., Lücken M.D., Strobl D.C., Henao J., Curion F., et al. Best practices for single-cell analysis across modalities. Nat. Rev. Genet. 2023;24:550–572. doi: 10.1038/s41576-023-00586-w. [DOI] [PMC free article] [PubMed] [Google Scholar]

[bib52] 52.Fawcett T. An introduction to ROC analysis. Pattern Recognit. Lett. 2006;27:861–874. doi: 10.1016/j.patrec.2005.10.010. [DOI] [Google Scholar]

[bib53] 53.Guo C., Pleiss G., Sun Y., Weinberger K.Q. PMLR; 2017. On Calibration of Modern Neural Networks; pp. 1321–1330. [Google Scholar]

[bib54] 54.Efron B., Tibshirani R.J. Chapman & Hall; 1993. An Introduction to the Bootstrap. [Google Scholar]

[bib55] 55.Lahiri S.N. Springer Science & Business Media; 2013. Resampling Methods for Dependent Data. [Google Scholar]

PERMALINK

ReliST: A model-agnostic risk layer for spatial transcriptomics deconvolution

Xinyu Zhang

Li He

Yu Peng

Yijia Li

Jianyuan Kang

Lisheng Peng

Yifei Xu

Sen Lin

Summary

Graphical abstract

Highlights

Introduction

Table 1.

Results

ReliST expands deconvolution outputs from prediction-only maps to prediction-plus-risk maps

Figure 1.

DLPFC supports risk scores as proxy-anchored diagnostic signals

Figure 2.

Common-feature controls and goal-based decision use define support tiers

Figure 3.

Mouse brain demonstrates reference-control value without layer labels

Figure 4.

Breast cancer high-risk signals mainly support artifact-aware review

Figure 5.

IF/gene-protein validation supports protein-level under-call evidence

Figure 6.

Controlled reference-output degradation provides a lightweight sanity check

Known-composition pseudo-spots directly link ReliST risk to true deconvolution error

Figure 7.

Discussion

Limitations of the study

Resource availability

Lead contact

Materials availability

Data and code availability

Acknowledgments

Author contributions

Declaration of interests

Declaration of generative AI and AI-assisted technologies in the writing process

STAR★Methods

Key resources table

Experimental model and study participant details

Ethics statement

Datasets and validation design

Method details

Problem definition and risk direction

Inputs and multi-axis risk signals

Base models and canonical output contract

Reproducible risk scoring

Implementation details and reproducibility

Pseudo-spatial known-composition benchmark

Confidence and uncertainty baseline comparison

Risk score calculation

Component ablation

Threshold sensitivity and use-case guidance

Reference perturbation stress tests

Revision analysis reproducibility

Quantification and statistical analysis

Validation metrics, resampling, and artifact-aware interpretation

Footnotes

Contributor Information

Supplemental information

References

Associated Data

Supplementary Materials

Data Availability Statement

ACTIONS

PERMALINK

RESOURCES

Similar articles

Cited by other articles

Links to NCBI Databases