Temporal and spatial composition of the tumor microenvironment predicts response to immune checkpoint inhibition in metastatic TNBC

Abstract

Immune checkpoint inhibition (ICI) benefits only a subset of patients with metastatic triple-negative breast cancer, and determinants of response remain unclear. We assembled a longitudinal cohort of 103 female patients from the phase II TONIC trial with samples spanning primary tumors, pre-treatment metastases, and on-treatment metastases during nivolumab therapy. We profiled 37 proteins in 270 tumors using highly multiplexed imaging and developed SpaceCat, an open-source pipeline that extracts more than 800 imaging features per sample, including cell density, diversity, spatial interactions, and functional marker expression. Metastatic, but not primary, tumors contained features predictive of outcome. Spatial metrics such as immune diversity and T-cell infiltration at tumor borders were most informative, while ratios of T cells to cancer cells and PD-L1 on myeloid cells were also associated with response. Multivariate models stratified patients with highest performance on-treatment (AUC = 0.90). Bulk RNA-seq confirmed the predictive value of on-treatment samples. These findings highlight the value of longitudinal profiling to resolve evolving TME dynamics driving ICI response.

Introduction

Immune checkpoint inhibition (ICI) has transformed cancer therapy, but response rates to monotherapy remain low in metastatic breast cancer compared to melanoma or lung cancer^1–7. Consequently, recent trials have evaluated chemo-immunotherapy, with IMpassion130 providing the first phase III evidence of benefit in metastatic TNBC (mTNBC)⁸. KEYNOTE-355 trial further confirmed efficacy, establishing combined chemo-immunotherapy as the standard of care for PD-L1-positive mTNBC⁹.

Numerous studies have attempted to identify clinical and translational predictors of ICI response in breast cancer^10–16. Identifying robust biomarkers to distinguish responders from non-responders remains challenging, largely due to the complex effects of immunotherapies. Unlike targeted therapies directed at tumor-specific proteins, immunotherapies activate a diversity of immune cell types, requiring multidimensional analysis. Conventional assays measuring limited parameters fail to capture this complexity, which is further shaped by spatial context and interactions within the tumor microenvironment¹⁷. As such, assays that lack spatial information cannot fully resolve this complexity.

Spatial profiling has provided key insights in the TNBC TME. For example, imaging mass cytometry uncovered distinct spatial organization with increased immune infiltration in TNBC compared to ER+HER2- and HER+ breast tumors, and our prior work showed separation of primary TNBC by immune-cancer cell mixing^18–19. While recent studies identified predictors of response to chemo-immunotherapy before and during treatment¹³, their relevance to metastatic disease remains unclear. Longitudinal sampling, especially in metastatic breast cancer, remains rare, limiting understanding of tissue dynamics, progression, and outcomes.

We present a spatiotemporal TNBC dataset with matched primary tumors and longitudinal metastatic biopsies collected before and during anti-PD1 therapy in a prospective trial. We generated multiplexed imaging data of pathology sections for each patient at different timepoints and combined this with previously generated genomics and transcriptomics¹⁵ data to enable multi-modal characterization of the TME. Analysis revealed imaging and transcriptomic features associated with response, including T cell infiltration at the cancer border, cellular neighborhood diversity, and PD-L1+ myeloid cells. In contrast, whole-exome sequencing from pre-treatment biopsies showed no robust correlates with response. Looking across timepoints, we found that increases in cellular diversity were consistently associated with better outcome. In contrast, other features like CD8 T cell density and PD-L1 expression levels were not universally predictive, and their association with outcome was timepoint-dependent. Multivariate models showed on-treatment samples provided the strongest predictive power, whereas primary tumors offered little. These findings underscore the importance of longitudinal,multimodal characterization for defining determinants of ICI response and guiding future clinical trials.

Results

Multimodal characterization of longitudinal metastatic TNBC

We analyzed longitudinal histological samples from 103 metastatic TNBC (mTNBC) patients treated with nivolumab (anti-PD-1) in the TONIC trial (NCT02499367; Supplementary Table 1)^15,20. Patients were enrolled in TONIC-I stage I (n=65) or stage II (n=38), with an average of 43.5 months from diagnosis to inclusion and 30.1 months from primary to relapse. To understand the evolution of the TME, tumor samples spanned four timepoints: 1) primary tumor (retrospective collection), 2) metastatic baseline biopsy at the start of TONIC trial, 3) pre-nivolumab (after induction), and on-nivolumab (after 3 cycles nivolumab) (Fig. 1a). Representative regions were selected for tissue microarrays by a pathologist (Extended Data Fig. 1a), and a 37-plex antibody panel was designed to capture TME cell types, functional states, and tissue architecture (Supplementary Table 2, Methods). Multiplexed ion beam imaging (MIBI) enabled high-dimensional, quantitative imaging ²¹ (Methods).

Figure 1. Study design, workflow and feature extraction.

a. Schematic overview of the collection of tumor tissue for MIBI analysis from patients accrued in the TONIC trial (NCT02499367). Tissue was collected from the primary tumor, as well as metastatic samples at baseline, after induction (pre-nivo), and after three cycles of nivolumab (on-nivo).

b. Venn diagram showing the number of patients (n=103 patients) who have overlapping data from the three modalities (DNA, RNA and MIBI) included in the study

c. Schematic overview of the data generation and feature extraction workflow for DNA, RNA and MIBI data.

d. Schematic representation of static and dynamic feature data. All features were calculated and compared to response on four separate, static timepoints. In addition, the dynamic change between pairs of timepoints was also calculated and compared against patient response.

e. Full FOV: MIBI field of view (FOV) color overlay of a representative tumor biopsy. Inset 1 and 2 show blowups at increased magnification from the original image.

Color overlay: Additional color overlays from the same FOV. Cell lineage: Cell mask with cells colored by cell classification. Compartment: Cell mask with cells colored by tumor compartment. Scale bars: 100um

f. Heatmap showing the 20 identified cell clusters (y-axis) with the average expression of the 36 markers in the MIBI panel (x-axis) split by marker type. Cell lineage markers were used to generate the clusters, whereas functional markers were not used for clustering.

We generated multiplexed images from 620 distinct TMA cores (Extended Data Fig. 1b–c), including 59 primary tumors, 79 baseline biopsies, 70 pre-nivo biopsies, and 62 on-nivo biopsies from 101 unique patients (Fig. 1a, Extended Data Fig. 1d, Supplementary Tables 3–4). Raw MIBI images were background subtracted, denoised, and normalized prior to analysis (Methods). Processed images were segmented with Mesmer²², identifying an average of 2,905 cells per core (Fig. 1e; Methods). Cells were then classified using Pixie²³, yielding 22 clusters that were grouped into eight major TME lineages (Fig. 1f), summarizing the major components of the TME (Extended Data Fig. 2a–d).

To complement the spatially-resolved proteomic information, we analyzed previously published bulk whole-exome sequencing (WES) from the same metastatic baseline samples and bulk whole-transcriptome RNA sequencing (RNA-seq) from the same metastatic baseline, pre-nivo and on-nivo samples (Fig. 1b), using a harmonized bioinformatics pipeline (Methods). In total, WES was available for 74 patients, RNA-seq for 191 samples from 89 patients, and MIBI for 270 samples from 101 patients, yielding multi-modal data across 103 patients (Fig. 1b–c, Extended Data Fig. 1e–g).

We quantified cell cluster abundance from the MIBI data in baseline metastatic tumors, finding cancer cells as the most prevalent population (46.3% of all cells). These were divided into three subgroups: Cancer 1 (73.8% of cancer cells) and Cancer 2 (11.7%), both ECAD⁺/CK17⁺ with elevated Ki67 and GLUT1; and Cancer 3 (14.4%), defined by dim expression of ECAD and CK17 (Fig. 1f, Extended Data Fig. 2g).

Immune cells were the next most abundant cell type (32.1% of all cells, Fig. 1f, Extended Data Fig. 2i), with CD4⁺ T (16.5% of immune cells) and CD8⁺ T cells (17.4%) as the most abundant subsets, both frequently expressing PD-1. Tregs (4.7%) were the most proliferative immune cell population, with 15% expressing Ki67²⁴, and antigen presenting cells (APCs) had the highest expression of IDO1. Four macrophage/monocyte subsets were identified, many expressing PD-L1 and TIM3. Natural killer (NK) cells, though rare (0.7%), often expressed T-BET and TIM3, indicating maturation and immunosuppressive status^25–27.

Structural cells (18.1%) included fibroblasts (77%), endothelial (15%), and smooth muscle (7%; Extended Data Fig. 2h). Fibroblasts included cancer-associated fibroblast (CAF) phenotypes similar to CAF-S1, previously linked to immune suppression and Treg interaction in TNBC²⁸, and CAF-Other population negative forFAP or SMA. Additional clusters included Immune Other (3% of total) and marker-negative cells (0.8%). Overall, the cell population abundances aligned with prior TNBC spatial analyses^19,29.

We next examined changes in cell population prevalence in metastatic lesions from baseline to on-nivo. Baseline (n=79) and on-nivo (n=62) metastatic tumors had similar cell proportions (Extended Data Fig. 2e), which persisted with finer clustering into 22 cell types (Extended Data Fig. 2f). The modest temporal differences in abundance prompted us to develop metrics to capture the spatial dynamics in the TME.

Quantification of the tumor microenvironment with SpaceCat

Highly multiplexed image data is a rich source of information for defining spatial relationships between cell types, but quantifying this information at scale is challenging. To address this gap, we developed SpaceCat, an open-source computational pipeline which generates a spatial catalog of informative features from multiplexed image data. SpaceCat can be applied to any processed multiplexed imaging dataset, generating 903 features that capture cellular and acellular abundance, location, phenotype, and organization. These features span cell density, diversity, spatial interactions, extracellular matrix composition, immune infiltration, and functional marker expression (Fig. 2a, Extended Data Fig. 3, Supplementary Table 5, Methods). Features included cell density metrics at both lineage and subset levels (e.g., total vs. CD8⁺ T cells), diversity scores reflecting balance between populations, and mixing metrics quantifying partitioning between cell types (Fig. 2a). Applied to all 620 tissue cores, SpaceCat enables scalable, standardized extraction of spatial metrics from multiplexed imaging data.

Figure 2. SpaceCat feature extraction pipeline.

a. Four categories of features (cell abundance, functional marker positivity, diversity scores, and spatial organization) calculated by SpaceCat, with an example feature belonging to each of those categories. Each column shows one example image with a high value of the feature and one example image with a low value of the feature. Scale bars: 100um.

b. Distribution of extracted features according to the category of the feature (left) and cell type associated with that feature (right). The categories consist of five feature types: cell phenotype (n=444), cell abundance (n=246), spatial structure (n=103), diversity (n=65), and cell interactions (n=45). The cell types consist of five feature classifications: immune cells (n=320), multiple cell types (n=308), cancer cells (n=118), structural cells (n=59), and ECM cells (n=93), where n=number of features.

c. Clustered pairwise correlation of 903 features across all regions of interest in the TONIC cohort. The colored squares indicate clusters of features that are characteristic of a distinct biological process, e.g., immune diversity, morphology, or hypoxia. Nuc: nuclear. Cyto: cytoplasmic. Perim: perimeter.

To assess location-specific effects, we defined four tumor compartments: cancer core (high cancer density), cancer border (outer edge of cancer), stroma border (the surrounding non-cancer edge), and stroma core (remaining image area) (Extended Data Fig. 3a, Methods). Spacecat features were computed across both whole images and individual compartments. Some features, such as Ki67⁺ cancer cells and PD-L1⁺ macrophages, were consistent across compartments, whereas the CD8/CD4 T-cell ratio was significantly higher in the cancer core, suggesting targeted CD8⁺ migration (Extended Data Fig. 4a, f–g)^30,31. In total, 28% of features varied by compartment (Extended Data Fig. 4c), mainly between tumor and stroma (Extended Data Fig. 4b), highlighting the importance of spatial context.

Of the 903 SpaceCat features, the most common were those capturing cell phenotypes (n=444, i.e. proportion of Ki67⁺ cancer cells) and cell abundance (n=236, i.e. T cell density), reflecting both how informative those features are (since highly correlated features are removed) and marker selection (Fig. 2b). Other categories of features included spatial structure (n=103), cellular diversity (n=65), and cellular interactions (n=45). Due to the immunological focus of our antibody panel, we could define immune cells with the highest degree of granularity, and hence they were the most represented cell type in SpaceCat features (Fig. 2b, 42.3%). We also identified features that involved multiple cell types, cancer cells, and the extracellular matrix (ECM).

We compared ECM features with neighboring cell phenotypes by classifying image patches into three ECM profiles: no ECM, cold (Collagen only), and inflamed (Collagen, FAP, Fibronectin) Extended Data Fig. 4h). We found a higher proportion of fibroblasts and macrophages with expression of the immunosuppressive protein TIM3 in cold regions compared to inflamed regions (Extended Data Fig. 4i–j), with macrophages in the cold regions additionally showing higher expression of PDL1 (Extended Data Fig. 4k).

A correlation matrix of the distinct features revealed clusters of biologically-related features (Fig. 2c), including broad categories such as immune diversity (e.g., CD4⁺ T neighborhoods, macrophage proportions, global diversity), along with narrower modules such as hypoxia (GLUT1⁺) and morphology (nuclear–cytoplasm ratio, membrane concavities; Fig. 2c). By defining and quantifying core TME components, SpaceCat generates interpretable features that enable separation of patient populations.

To assess pre- to post treatment-associated changes we ranked each feature by the magnitude of the shift. We found 19 features that differed between the two timepoints (Extended Data Fig. 5a), including the ratio of T cells to Cancer cells at the stroma border, which significantly increased through treatment (Extended Data Fig. 5b). This corresponds to an increase in T cell infiltration that is independent of patient outcome, and is shared across all patients.Other features, including CD45RO positivity and macrophage-associated diversity, showed numerical shifts but were not clearly separated between timepoints (Extended Data Fig. 5c–d).

We repeated the analysis in matched samples (Extended Data Fig. 5e), and again found a significant increase in the T cell to cancer cell ratio at the stroma border (Extended Data Fig. 5f). Other features, such as unclassified cell to cancer cell and NK to T cell ratios (Extended Data Fig. 5g–h), showed similar trends but lacked clear separation, suggesting limited temporal differences independent of outcome. This motivated analysis stratified by patient response.

Linking transcriptomic and spatial features of the TME

To complement the MIBI data, we integrated RNA-seq profiles³², extracting cytokine signaling signatures, outcome-associated gene sets, and predicted cell-type frequencies (CIBERSORTx) (Extended Data Fig. 6a, see Methods). We found strong cross-modality relationships (Extended Data Fig. 6b, Supplementary Table 6), including concordance between MHCI RNA scores and HLA1 MIBI-based expression (Extended Data Fig. 6c), and novel associations such as TGFβ signaling³³ with CD45RB+ CD4+ T cells at the cancer border (r=0.88, Extended Data Fig. 6d). Immune-related RNA-seq pathways including MDA5/IRF7/IRF3 (r=0.74) and RIGI/IRF7/IRF3 (r=0.72), were strongly correlated with HLA1⁺ cancer cells(Extended Data Fig. 6e), with the top RNA-seq associations all linked to interferon signaling. We identified an inverse relationship between the cellular diversity around CD8 T cells in the cancer core and a T-cell exhaustion signature (Extended Data Fig. 6f),We then assessed which MIBI feature categories correlated most strongly with RNA-seq. Significant feature pairs (Spearman’s r ≥ 0.6, adjusted p ≤ 0.05). were enriched in the cancer core and cancer border, whereas stromal features showed fewer correlations. (Extended Data Fig. 6g).

Predictors of response to immune checkpoint blockade

To link TME structure with ICI benefit, we tested each SpaceCat feature for association with nivolumab response in the TONIC trial (Fig. 3a, Methods). We quantified association strength using an importance scorecombining statistical significance and effect size (0 = least predictive, 1 = most predictive), and focused subsequent analyses on the top 100 features (Methods).

Figure 3. Extracted microenvironmental characteristics associated with patient response.

a. Volcano plot showing significance (unpaired two-sided t-test with equal variance, y axis) and effect size (difference in medians, x-axis) of features (n=903 features) to predict patient response, colored by overall ranking.

b. Comparison of cell ratios and individual cell densities to predict outcome. The score for each of the ratios (n=25 ratios) in the top 100 features is shown on the left hand side. For each ratio, which is composed of two different cell types, the cell type density with the higher score is plotted on the right. T / Cancer ratio; ratio between T cells and Cancer cells. T density; Density of T cells.

c. Enrichment within the top 100 features for features calculated within each of the tumor compartments, or those calculated across the whole image.

d. Enrichment within the top 100 features for features that do and do not require spatial information in order to be calculated.

e. An overview of the top 50 response-associated feature/compartment combinations. Each column is a feature, and each row provides context or description of the feature. The first five rows indicate which compartment(s) show(s) an association with outcome for that feature in the top 50, with the first four rows specifying individual compartments and the fifth specifying the entire image. For example, the leftmost cell ratio feature (T / Structural) is one of the top 50 highest ranked features associated with outcome when calculated in the cancer core compartment, whereas the next cell ratio feature (T / Cancer) is in the top 50 highest ranked features in the cancer border, cancer core, stroma border, and whole image. The bottom row illustrates whether the feature is positively or negatively associated with outcome. Ratios between cell types are detonated with a ‘/’, e.g. T cell to Cancer cell ratio is T / Cancer. Neighborhood diversity is a spatial metric that takes into account the immediate neighbors of a given cell type, whereas other diversity metrics are calculated using the total count of cells within a compartment.

f. Representative example of a top feature (PD-L1+CD68+ Macrophages). The boxplot on the left shows the feature stratified by outcome (N=52 patients, 10 Yes / 42 No). The overlays on the right show four specific examples (highlighted in the box plot) of patients with high and low levels of PD-L1+CD68+ Macrophages. Data is from the on-nivo timepoint. Scale bars: 100um.

g. Representative example of a top feature (diversity in the cancer border region). The boxplot on the left plots this feature stratified by responder/non-responder status (N=58 patients, 8 Yes / 50 No). The overlays on the right show two specific examples (highlighted in the box plot) of patients with high and low border diversity. The top row shows the image compartments (same coloring as in 3c and 3e), and the bottom row shows the cell types present in the cancer border compartment. Data is from the on-nivo timepoint. Scale bars: 100um.

Box plot: Lower bound is 1st quartile, center is median, upper bound is 3rd quartile, whiskers extend to 1.5*IQR beyond bound.

Among the top 100 features, cell ratios were enriched over single-population densities (Extended Data Fig. 7a). Ratios also scored higher than their component densities (unpaired t-test, p<0.0001, t-stat=−4.4, Fig. 3b)e.g., the T cell/cancer ratio in the cancer core scored 0.99 vs. 0.71 for T cell density and 0.69 for cancer density.

Spatial context further increased predictive power. Features defined within tumor compartments were enriched among the top predictors (proportions z-test, p<0.0001, z-stat=−4.3, Fig. 3c). For instance, cellular diversity at the cancer border ranked 31st, but 302nd when computed across the whole image. . Overall, spatial features were significantly overrepresented(proportions z-test, p<0.01, z-stat=-2.6, Fig. 3d).

We organized the top-ranking SpaceCat features into biological modules (Fig. 3e). Ratio-based features were consistently associated with better outcome (e.g., T cell/cancer, B cell/cancer, T cell/structural ratios; Fig. 3e). In line with prior work^19,34, another module reflected PD-L1 expression on non-cancer cells (Fig. 3e, “PD-L1 positivity”); with PD-L1⁺ CD68⁺ macrophages enriched in responders(unpaired t-test, p=0.002, t-stat=3.2, Fig. 3f). Diversity metrics also correlated with outcome (Extended Data Fig. 3): higher overall, compartment-specific, and lineage-specific diversity (e.g., around cancer or structural cells, or within monocytes) predicted benefit (Fig. 3e–g). In contrast, cancer cell diversity was mixed:total cancer and ECAD⁺/CK17⁺ Cancer 1 diversity were negatively associated with outcome, while low-epithelial Cancer 3diversity was positively associated with outcome (Fig. 3e).

We also investigated features previously reported in primary TNBC. Unlike prior work¹⁹, cancer–immune mixing was not predictive in TONIC at any timepoint (Extended Data Fig.7j). Similarly, proliferation and hypoxia¹⁸ showed no association in our cohort (Extended Data Fig. 7k–l).

To test generalizability,, we applied 134 published bulk RNA-seq signatures to the same samples and ranked them by importance scores for predicting response (Extended Data Fig. 7b). Two broad categories emerged: signatures of immune cell infiltration (e.g., effector and B cell signatures; Extended Data Fig. 7c), consistent with MIBI-derived features, and signatures of immune cell state (e.g., cytokine secretion and interferon signaling) that were not well-covered by our MIBI panel and complemented the MIBI analysis (Extended Data Fig. 7c).

Response associated features evolve through time

One of the major strengths of our cohort is the availability of longitudinal samples spanning the primary tumor, baseline, pre-nivo, and on-nivo timepoints. The preceding analyses generated a single, timepoint-agnostic ranking of TME features associated with ICI response. Building on this, we asked to what extent these associations were temporally conserved. This revealed two insights: 92 of the top 100 features were associated with outcome at only one timepoint (Extended Data Fig. 7f) and most (n=80) came from the on-nivo samples (Fig. 4a,b, Extended Data Fig. 7g).

Figure 4. Evolution of features associated with response.

a. The number of features from the top 100 that are derived from each timepoint: on_nivo (n=80), pre_nivo (n=18), baseline (n=1), primary (n=1), n=number of features.

b. Heatmap showing the overlap of the top features across different timepoints. In order to be included in the visualization, a feature needs to be within the top 100 most predictive. Using this list of features, we then plot it in any timepoint where it is ranked within the top 350 features. Features are colored by their overall ranking, and boxed in red if they are within the top 100.

c. The top row of box plots shows the ratio of T cells to Cancer cells within the cancer border broken down by response. This is plotted across all four timepoints to show the change in association with outcome (Primary N=59, 9 Yes / 50 No) (Baseline N=76, 12 Yes / 64 No) (Pre-nivo N=66, 12 Yes / 54 No) (On-nivo N=57, 7 Yes / 50 No). Underneath are representative overlays showing the compartments within each image, followed by the T cells and Cancer cells within the border compartment. Scale bars: 100um. N=number of patients

Box plot: Lower bound is 1st quartile, center is median, upper bound is 3rd quartile, whiskers extend to 1.5*IQR beyond bound.

We then examined the top features from each timepoint. The top three from the primary tumors were the B cell / NK cell ratio in the cancer core, CD68 macrophage density, and the proportion of Vimentin⁺ CD4⁺ T cells (Extended Data Fig. 8a–c). However, while these ranked highly within primary tumor samples, the separation between responders and non-responders was modest, and none were significant at other timepoints. Moreover, features often considered prognostic in primary TNBC^29,31, including T cell / cancer cell ratio, PD-L1⁺ APCs, PD-1⁺ CD8⁺ T cells and tumor-proximal T cells, were not predictive of ICI response in our cohort and ranked low (all <1000).

At baseline, the dominant feature categories were cell ratios and cell diversity (Extended Data Fig. 7i, Supplementary Table 9), including structural cell density in the stroma border, unclassified cell–to–cancer cell ratio in the stroma core, and NK-to-other cell ratio in the stroma border (Extended Data Fig. 8d–f). Additional, lower-ranked features included the cellular diversity surrounding CAFs, the linear distance from B cells to Cancer cells, and the cellular diversity surrounding APCs (Supplementary Table 9). Previously identified pre-ICI predictive features from primary TNBC¹³, were not reproducible in our cohort (Extended Data Fig. 10a–e, Supplementary Table 8), highlighting differences between primary and metastatic TNBC.

The on-nivo timepoint yielded the greatest number of outcome-associated features, with 80 of the top 100 features. Many involved T- and B-cell densities and ratios between these lymphocytes and other cell types (Extended Data Fig. 7g). In contrast, baseline features were less lymphocyte-related, instead reflecting cancer and structural cell diversity (Extended Data Fig. 7g,h). Top on-nivo correlates included previously reported predictors – such as T / B cell ratio vs. cancer cells³¹, PD-L1 expression on macrophages¹³, and CD69 expression as a marker of tissue residence³⁰ – as well as novel associations with cancer cell and fibroblast diversity. Proximity between cancer cells and fibroblasts was also positively associated with outcome, suggesting these interactions may play a role in driving good outcomes. One of the highest ranked features was the ratio between T and cancer cells in the cancer border compartment. Notably, no significant association between this feature and outcome was observed in the primary, baseline, or pre-nivo samples, consistent with an influx of T cells to the border region being an early indicator following initiation of ICI for patients who will go on to respond (Fig. 4c).

Although most features were timepoint-specific, four of the top 100 were shared across timepoints (Fig. 4b, Extended Data Fig. 7f). Three of these localized to the cancer border, including cellular diversity, PDL1⁺ CAF-S1 cells, and the ratio of other cells to cancer cells. Using a more permissive threshold (top 350; Fig. 4b), we identified 22 features shared across two timepoints and six features shared across three timepoints. All six of these features were shared across the metastatic timepoints, and none overlapped with the primary tumor. Two of these, both cancer border features (CAF-Other neighbor diversity and the T cell / cancer cell ratio)(Fig. 4c) were consistently predictive across metastatic samples. Collectively, these findings reveal strong temporal dependence, with most predictive features arising on-treatment, and only a minority conserved across timepoints.

The analyses above considered features at single, static timepoints. To capture temporal dynamics, we leveraged paired samples (Fig. 1a) to calculate changes in each feature across intervals: primary to baseline, baseline to pre-nivolumab, baseline to on-nivolumab, and pre- to on-nivolumab. We then ranked these “evolutionary” features by their association with outcome. Many overlapped with the static analysis, such as T-cell density at the cancer border, which was predictive both as an on-nivo feature and as a change from baseline to on-nivo. In total, ten features were unique to the evolution analysis (Supplementary Table 7). Consistent with the importance of the on-nivo timepoint, all reflected evolutionary changes from earlier timepoints to on-nivo, including the B / T ratio and the proportion of GLUT+ Cancer 1 cells. No changes from primary to baseline predicted outcome, underscoring the distinct biology of metastatic lesions.

Multivariate modeling to predict patient response

Having identified multiple features individually associated with ICI response, we next developed multivariate models to predict treatment response at each timepoint. We performed classification using a Lasso model to predict whether a patient was classified as a responder or non-responder. We used all 903 SpaceCat features to train a separate Lasso³⁵ model on each timepoint, using nested cross validation to estimate model accuracy (Extended Data Fig. 9a, Methods).

A model trained on MIBI data from the primary tumor performed poorly (mean AUC=0.54), only modestly above random chance (Fig. 5a). In contrast, baseline and pre-nivolumab models performed better (mean AUC = 0.79 and 0.66, respectively; p < 0.0001 for both, unpaired t-test). To contextualize these results, we compared them with the NeoTRIP trial of early stage TNBC¹³. Using multiplexed imaging, the authors reported AUCs of 0.77 for both pre-treatment and on-treatment biopsies for response prediction to ICI. Our baseline model achieved comparable accuracy (AUC = 0.79), but on-treatment samples in TONIC yielded substantially higher performance (mean AUC = 0.90), exceeding NeoTRIP’s on-treatment value (Fig. 5a).

Figure 5. Multivariate modeling to predict response.

a. AUC (y axis) of the multivariate models stratified by assay (N=10 replicates per assay) and timepoint (x axis). Each dot is a replicate from nested cross validation. On the right, data from ref. 13 was replotted on the same axis.

b. Representative features derived from the MIBI data that were strongly associated with patient outcome. The number of distinct channels required to calculate each feature is shown with a horizontal bar, along with the relevant timepoint and the analysis method (univariate or multivariate) that identified the feature.

c. Same as b), but for RNA-based features, showing the number of transcripts instead of number of channels

d. The cumulative sum of the weights of the 13 features that can be calculated (y axis) as more channels are included in the imaging panel (x axis)

Box plot: Lower bound is 1st quartile, center is median, upper bound is 3rd quartile, whiskers extend to 1.5*IQR beyond bound.

To test whether this reflected methodology or disease context, we reanalyzed the NeoTRIP¹³ data through SpaceCat and reproduced their findings, confirming cross-cohort and cross-platform robustness (Extended Data Fig. 10f). We then tested whether the NeoTRIP predictors also held in our metastatic samples. Few overlapped: proliferation and heterotypic epithelial–immune interactions, key predictors in NeoTRIP, were not associated with outcome in TONIC (Extended Data Fig. 10h–i, Supplementary Table 8). Instead, our predictors included homotypic immune interactions, particularly among CD8₊ T cells, consistent with SpaceCat results highlighting diverse immune infiltration (Fig. 3e). Recalculating enrichment using only NeoTRIP features confirmed the same temporal pattern; strong predictive power at the on-nivolumab timepoint and little at the primary (Extended Data Fig. 10j). Finally, although we identified comparable cancer cell populations to those reported by NeoTRIP, they were not outcome-associated in TONIC, and their inclusion in multivariate models did not improve performance (Extended Data Fig. 10k–l). Collectively, these findings suggest the superior predictive accuracy of on-treatment samples in TONIC reflects fundamental biological differences between primary and metastatic TNBC.

We next applied timepoint-specific modeling to bulk RNA-seq data. Similar to MIBI, the on-nivolumab RNA model performed best (AUC=0.88), outperforming pre-nivolumab (unpaired t-test, p<0.0001, t-stat=5.0, AUC=0.72) and baseline models (unpaired t-test, p<0.0001, t-stat=8.8, AUC=0.72). Both RNA and MIBI achieved excellent performance at the on-nivolumab timepoint, though the baseline MIBI model outperformed the baseline RNA model (unpaired t-test, p=0.003, t-stat=-3.4). Unsurprisingly, a model trained on baseline genomic features (somatic mutations and copy number alterations) was less accurate (AUC=0.69) (Fig. 5a). Overall, these findings underscore the importance of longitudinal sampling and highlight the predictive value of early on-treatment biopsies, consistent with prior findings neo-adjuvant anti-HER2 targeted therapy³⁶.

We next asked whether combining modalities could improve predictive accuracy. Using cross-validation with both MIBI and RNA features, we found that multimodal models did not outperform unimodal ones; in fact, combined models showed reduced performance (Extended Data Fig. 9i). This may reflect limited sample size, as only ~70% of patients had both modalities, in addition to redundancy in information across data types.

A key strength of Lasso modeling is interpretability, as coefficients directly reflect feature importance. The baseline MIBI model highlighted CD38₊ cells as a key predictor (Fig. 5b, Supplementary Table 10). CD38, a glycoprotein with receptor and enzymatic functions, has been implicated in immunosuppression during ICI therapy³⁷. In our cohort, CD38 expression was mainly endothelial, unlike prior reports of T-cell and cancer-cell expression. Notably, CD38 positivity can be measured with only three markers (two for segmentation plus CD38), making it feasible for clinical implementation. For baseline RNA, the top predictor was the cytolytic activity score, a simple two-gene metric (Perforin-1 and Granzyme-A)³⁸.

At the on-nivolumab timepoint, the MIBI model identified nine predictive features, including cancer diversity and the B cell / structural cell ratio, each requiring only 3–6 markers to compute (Fig. 5b). The RNA model identified three signatures: cytolytic activity, anti-tumor cytokines, and extracellular matrix, with the cytolytic score shared across all three timepoints (Fig. 5c, Extended Data Fig. 9e). The RNA-based signatures required an average of eight distinct genes for calculation.

Finally, we examined how panel size and feature number affected predictive accuracy. For the highly predictive on-nivo model, 28 antibody channels were needed to generate all selected features, out of 37 (Fig. 5d). However, the minor contribution of certain features suggests this number could be reduced, with implications for how our findings could be translated into a scalable assay. Imposing limits from 20 to 5 features reduced performance, but even with five features the on-nivo model achieved AUC >0.8 (Extended Data Fig. 9h). Moreover, several individual features, such as the cancer / T cell ratio in the core, the B cell / cancer cell ratio, and CD8 T cell density achieved strong predictive accuracy on their own (AUC >0.85; Supplementary Table 9). These findings indicate that streamlined, simpler assays could provide clinically actionable stratification of patients.

Discussion

Here, we examined how the TME evolves over time in patients with metastatic TNBC and how these dynamics relate to response to ICI. We assembled a unique longitudinal clinical cohort from the TONIC trial¹⁵, spanning primary diagnosis through on-treatment metastatic biopsies, and developed an interpretable, open-source computational framework (SpaceCat) to extract >800 features from multiplexed imaging data. By integrating spatial, transcriptomic, and genomic profiling, we identified both shared and timepoint-specific determinants of response, highlighting the importance of longitudinal sampling in clinical trials.

Our analysis revealed significant insights into the biology underpinning patient responses to ICI. Cellular diversity consistently correlated with improved outcome, both at the image-wide level and within compartments. In addition to overall diversity, we also found increased ratios of immune cells to cancer cells were robust predictors of patient response, both at the image-wide level as well as at the interface between stromal and cancer compartments^39–41. Beyond cell abundance, PD-L1 positivity on APCs, macrophages, and CAFs was strongly associated with benefit, underscoring the role of myeloid and structural cells in shaping immune activation. In contrast, pre-treatment lymphoid features such as T-cell density or exhaustion markers were not predictive, whereas on-treatment biopsies revealed multiple T-cell features linked to outcome. This supports the view that ICI efficacy may involve replenishment of new immune clones rather than expansion of existing ones, consistent with recent reports^39–43.

We observed a significant increase in both the number and strength of features associated with patient response at the on-treatment timepoint compared to the pre-treatment. Although it might seem intuitive that on-treatment samples would be most informative, two recent studies in early TNBC evaluating response to neoadjuvant ICI did not find this pattern. In the NeoTRIP trial¹³, multivariate models trained on pre-treatment and on-treatment samples achieved similar accuracy. Similarly, recent work identified distinct response trajectories without enrichment of predictive features in on-treatment samples⁴⁴. The greater information content we observed in on-treatment samples likely reflects both the unique biology of the metastatic setting (as the prior studies focused on untreated primary disease) and the distinct patient populations studied.

We found that the antecedent primary tumor had relatively little ability to predict subsequent ICI response for the metastatic patients enrolled in TONIC, whereas metastatic samples (particularly on-treatment biopsies) contained far richer determinants of ICI benefit. Prior studies have successfully used immune-based features to predict outcome in primary TNBC, but these were performed in unselected patient populations^45–47. In contrast, our cohort consisted entirely of patients who progressed to metastasis. Thus, our findings do not contradict previous work, but rather emphasize that features which predict outcome in the primary TNBC do not necessarily translate to the metastatic setting. This likely reflects both evolutionary changes induced by prior therapies and the biology of the metastatic process itself. An important implication is that identifying determinants of response in metastatic disease requires direct profiling of metastatic lesions, ideally as close as possible to treatment initiation. Despite the simplicity of this concept, there are numerous practical, logistical, and ethical considerations that have limited its implementation. Dedicated resources, infrastructure, and engagement with clinicians and patient advocates will be needed to prioritize longitudinal metastatic sampling. Considering that response monitoring in the clinic is becoming more adaptive and patient specific⁴⁸, it is likely that in the future, patients will be routinely monitored with serial imaging and liquid biopsies to inform their treatment. This is already being tested in several clinical trials,^16,49–51 with the potential to improve real-time treatment decisions.

The multi-modal nature of our dataset enabled direct comparison of MIBI, bulk RNA sequencing, and bulk exome sequencing from the same samples. This raises an important question: What data are most informative for understanding the TME? Our results indicate modalities capturing cellular states and interactions provide greater value than those limited to DNA alterations. In TONIC, driver mutations, copy number alterations, and genomic disruption did not reliably predict ICI benefit in metastatic TNBC patients. Although both RNA-seq and MIBI models achieved strong predictive accuracy at the on-treatment timepoint, interpretability differed between them. For example, identifying T-cell density at the tumor border offers clearer biological insight than a bulk RNA signature labeled “effector cells.”Although both RNA and MIBI-based models produced accurate predictions, we found multimodal models did not improve accuracy. This may reflect limited sample size or modeling approach. Alternatively, the necessary predictive information (e.g., CD8 infiltration, immune diversity) may be captured by either modality, even if represented differently.

Our study has several limitations to consider. First, the TONIC trial specifically profiles patients with metastatic TNBC, and it is uncertain whether these findings here extend to earlier disease stages or other cancer types. Second, because longitudinal sampling in metastatic ICI trials is rare, we were unable to validate our observations in an independent cohort with comparable clinical characteristics. Third, to make imaging analysis feasible, we used tissue microarrays constructed from pathologist-selected regions rather than whole tumor blocks, which may have excluded spatial features in the tumor.

Despite these limitations, our study underscores both the value of spatial profiling for generating accurate, interpretable and informative TME features linked to ICI response. At the same time,bulk transcriptomics emerges as a more practical, scalable alternative that can deliver similar predictive power, though at the cost of interpretability.

Methods

Ethics

The research performed here was approved by the Institutional Review Boards (IRBs) of the Netherlands Cancer Institute (protocol CFMPB716) and Stanford University (protocol IRB-46646). The trial was conducted in accordance with the protocol, Good Clinical Practice standards and the Declaration of Helsinki. The full protocol and the informed consent form were approved by the institution’s medical-ethical committee. All patients provided written informed consent before enrollment. Patients were not compensated for being enrolled in the trial. Because breast cancer predominantly affects women, only women were included in this analysis.

Study design and sample collection

The TONIC trial (NCT02499367) is an adaptive phase II, randomized, non-comparative study evaluating the feasibility and efficacy of nivolumab following a 2-week induction treatment in patients with metastatic TNBC. This single center, non-blinded trial was conducted in two stages, according to Simon’s two-stage design⁵². Initially, five cohorts were included; four receiving induction treatment (low-dose chemotherapy or irradiation) before nivolumab and one without induction treatment. Results of the first stage of the trial were previously reported¹⁵. In the second stage, the number of arms was reduced based on the first stage results, following the ‘pick-the-winner’ approach and considering both clinical and translational endpoints. For this translational study, patients from both stages I and II were included if they had at least one sample available and belonged to either the responder or non-responder group (n=103 out of 127 eligible patients, clinical data published in^15,20).

All patients biopsies from metastatic lesions were taken at baseline of the TONIC trial (baseline). Core biopsies from metastatic lesions were taken before the start of the study immunotherapy (baseline), after induction and after three cycles of nivolumab (240 mg flat dose)¹⁵. To explore the additional value of microenvironmental analyses from the primary tumor, we retrospectively pursued to collect the paraffin-embedded archival tissue blocks from each patien’s therapy-naive primary tumor (Fig. 1a). For all patients accrued up to 2019/07/23, we requested the archival formalin-fixed paraffin-embedded (FFPE) tissue blocks of the primary, therapy-naive, tumor. Archival tissue blocks were manually screened by a breast cancer pathologist in slidescore⁵³. New hematoxylin and eosin–stained (H&E) whole slides were prepared for all tissue specimens and histopathologic features were reexamined by a dedicated breast pathologist. Each H&E section was analyzed to determine which samples contained tumor cells, and were most representative of the tumor (Extended Data Fig. 1).

For primary tumor resections, up to six distinct 1.5mm cores were collected. Core selection aimed to capture either representative tumor regions or representative immune infiltrates. Tumor-focused cores reflected the growth pattern, histology, and grade of the tumor and included immune cells when infiltration was uniform. If there was significant heterogeneity in the immune infiltration, tumor-focused cores were predominantly tumor, while separate infiltrate-focused cores were taken to capture immune-rich regions. For the biopsies, up to three 1.5 mm cores were collected, often comprising most of the available tissue. Regions composed primarily of fat, fibrosis, or in situ cancer were excluded. All selected regions were annotated, cored, and assembled into 23 tissue microarrays (TMAs) of 1.5 mm cores.

We picked this selection strategy to maximize the diversity of TME features captured and to ensure cores included representative regions spanning the range of intratumoral variation. However, this approach does not guarantee proportional representation of all phenotypes.. For example, if a tumor contained 90% cancer-dense regions, and 10% immune-rich regions, our sampling would not reproduce this ratio. Although we feel this tradeoff was worthwhile, it limits our ability to make strong claims about absolute abundance of cancer versus non-cancer cell populations.

Key inclusion criteria were: ≥18 years; metastatic or incurable locally advanced TNBC with confirmed estrogen receptor negativity (< 10%) and HER2 negativity ( 0, 1+ or 2+ without amplification determined by in situ hybridization) on a biopsy of a metastatic lesion or breast recurrence. Additional criteria were detailed previously¹⁵. Neoadjuvant chemotherapy was given to 55.6% of patients for their primary tumor, with only a minority achieving a near-complete (3.4%) or complete (6.8%) pathological response at surgical resection, consistent with poor prognostic outcomes^54,55. Adjuvant chemotherapy was given to 38.4% of patients. Responders were defined by a best overall response of complete response (CR), or partial response (PR) according to RECIST1.1⁵⁶ and iRECIST⁵⁷.

Control TMA construction

We constructed a control tissue microarray to identify slide-to-slide variation in staining. Each core on this TMA was 1.5mm, for a total of 13 cores. Control tissues were carefully selected from archival FFPE tissues from the Stanford pathology department. The control TMA included two replicate cores each of tonsil, spleen, lymph node, breast (DCIS, IDC, and normal areas), colon and placenta, plus an additional tonsil for asymmetry. Serial recuts from the control TMA, along with from the cohort TMAs, were cut onto the same slide. Both TMAs on each slide went through all subsequent processing steps in parallel.

MIBI staining

Panel construction

The majority of the antibodies in this study have been previously validated for MIBI^19,58,59. New target antibodies were first validated by immunohistochemistry to confirm appropriate staining patterns in control tissue samples. All antibodies were then metal-labeled with the Ionpath conjugation kit (IonPath, Menlo Park, USA) following the manufacturer instructions. To increase reagent shelf life, labeled antibodies were then lyophilized individually with 100 mM trehalose in aliquots of 1 ug or 5 ug format. Following lyophilization, the appropriate antibody titer was determined by serial dilution with the following starting titer range (1 ug/mL, 0.5 ug/mL, 0.25 ug/mL, 0.125 ug/mL), for the new targets, or with the recommended titer, for the relabeled MIBI validated antibodies.

Cohort staining

To reduce batch effects, all TMAs were stained with the same mastermix. Fresh aliquots of each antibody were reconstituted and combined together into a single mastermix. Each step in the staining protocol was performed in pairs, with one reader and one pipettor, to reduce mistakes. For a complete description of the experimental conditions and procedures used, see our methods publication⁶⁰. For a step-by-step guide, see below.

Interactive protocols

For detailed, step-by-step guidance for reagent prepration and sample processing, please see our protocols for reagent preparation⁸⁴, IHC staining⁸⁵, MIBI staining⁸⁶, Sequenza staining⁸⁷, and antibody lyophilization⁸⁸.

MIBI data generation

MIBI run setup

All MIBI data was generated on a commercial MIBIScope instrument (IonPath, Menlo Park, USA). For each TMA core, paired H&E images guided selection of subregions to be acquired on MIBI, avoiding necrosis and empty areas (Extended Data Fig. 1b). Cores were named by row and column in the TMA grid to enable traceability to the metadata, with automated checks to identify and correct naming errors following user confirmation. Prior to beginning data acquisition, the order of the cores was randomized to mitigate potential batch effects from instrument drift.

Acquisition settings

We used the same settings across all MIBI data acquired in this study. TField of view (FOV) size depended on tissue availability: when possible, (800 μm)² (2048² pixels) was acquired; otherwise, (400 μm)² (1024² pixels) was used. We used a custom preset of 8.0 nano Amp beam current and 0.63 millisecond dwell time to balance acquisition speed and image clarity. We disabled all default background correction and noise removal settings. In addition to antibody targets, we quantified naturally abundant iron (Fe) by extracting the corresponding peak from the mass-spectrometry data.

MIBI cohort features

Multiple regions of interest (ROIs) per tissue sample were selected based on pathologist annotations of representative tumor and stroma. In total, 1614 cores from 123 patients were included on TMAs for MIBI data acquisition (Table S1). MIBI data was successfully generated for 1256 cores (success rate 78%), representing 117 patients, across 21 TMAs. The total dataset consists of 1256 FOVs, 678 were included in this study.

MIBI data processing

To remove contamination and background noise from the imaging data, we used a compensation matrix analogous to what is used in flow-cytometry for correction. We applied the same compensation matrix across all images in the cohort. Following compensation, we corrected for changes in detector sensitivity over the course of each imaging experiment using median pulse height (MPH). Finally, compensated and corrected data underwent QC to identify any potential batch effects due to TMA number or location within the TMA. Please see our online protocol⁸³ for a full description.

Cell segmentation

We used Mesmer²² to segment all images in the cohort. Mesmer is a pre-trained deep learning model that takes two channels of input data, a nuclear marker and a membrane marker. We combined the H3K27me3 and H3K9ac channels to form a single nuclear channel, and the CD14, CD38, CD45, CK17, and ECAD channels to form a single membrane channel. We then ran Mesmer using the default parameters as part of the ark-analysis pipeline⁶¹. For more details, please see our online protocol⁸³.

Cell clustering

Pipeline overview

We used Pixie²³ to cluster all of the cells in the cohort. Pixie is a cell clustering algorithm developed specifically for multiplexed imaging data. The first step is to cluster the individual pixels in each image. This produces more robust and reliable estimates of marker expression within each pixel than just using the raw expression values, as we can take advantage of marker correlation patterns. Using these labeled pixel clusters, we then perform cell clustering based on the number of pixel clusters of each type in each cell, rather than using the raw marker values. Following generation of the labeled cell clusters, we manually examined representative images to identify errors in the clustering, which was repeated as necessary. Finally, we performed post-clustering cleanup to address any remaining issues. We used the ark-analysis⁶¹ pipeline to run all of the described clustering and cleanup steps. For an in-depth description of the clustering pipeline, please see our online protocol⁸³.

SpaceCat

SpaceCat computes a wide range of distinct features to comprehensively summarize the tumor microenvironment. This includes features related to functional marker expression, cell density ratios, diversity, mixing proportions, cell neighborhoods, and more. Each image is broken up into four parts (cancer core, cancer border, stroma border, stroma core), and features are calculated at both an image-wide level, as well as separately within each compartment. These border regions are defined at the level of each individual image to identify the interface between cancer cells and non-cancer cells, and are not the same as the gross, pathological determination of a tumor margin. Importantly, enrichment in specific compartments was not due to the composition of our antibody panel or the makeup of the SpaceCat feature list. We set thresholds for the minimum number of cells needed for many of these calculations, and features which do not meet these thresholds in a given image are not computed. We filter out compartment-level features that are highly correlated with the image-level features to reduce redundancy. Following generation of each of these features, they are transformed into a standardized format, Z-scored, and combined into a single data frame for downstream analysis (Supplementary Table 5). For a full description of the SpaceCat pipeline, please see our online protocol⁸³.

Sequencing data

DNA and RNA data generation

We used the previously generated DNA and RNA data from all stage 1 patients, which was previously described^15,20. In addition to the original 60 patients described in that study, we analyzed sequencing data from an additional 33 patients.

DNA sequencing processing

To ensure the computational reproducibility, harmonizing and processing the genomic data from these large cohorts at scale, we have deployed Isabl platform⁶² locally and developed containerized and version control applications. We integrated tools for identification of Single Nucleotide Variants (SNVs), Copy Number Alterations (CNAs), HLA typing, Neoantigen prediction, IC-subtype prediction and RNA quantification that are described below. For seamless integration of these tools with the Isabel platform, Docker containers for all relevant tools and algorithms are openly available on Docker Hubs (https://hub.docker.com/u/cancersysbio, https://hub.docker.com/u/asntech).

Starting from FASTQ files, we aligned paired whole-exome sequences (n=78 tumor-normal pairs) to the human reference sequence (GRCh38, GATK version downloaded from AWS iGenome, https://ewels.github.io/AWS-iGenomes/) using BWA-MEM (v0.7.17)⁶³ as implemented in the TCGA-ICGC-PanCancer PCAP-core docker (https://github.com/cancerit/PCAP-core, v5.6.1). We assessed data quality using FastQC (http://www.bioinformatics.babraham.ac.uk/projects/fastqc/, v0.11.9) and Qualimap (v2.2.1)⁶⁴. Median tumor coverage assessed by Qualimap was 190.0 and 81.1 for tumor and normal samples, respectively.

We detected SNVs using a consensus approach leveraging two independent callers: Mutect2 (v4.1.7.0)⁶⁵ and Strelka2 (v2.9.10)⁶⁶. We ran Mutect2 on tumor/normal pairs as part of the nf-core/sarek Nextflow (v20.12.0) pipeline with the following parameters: -r 2.7.1 –step variantcalling –skip_qc all –tools Mutect2 -profile singularity. We ran Strelka2 on tumor/normal pairs with –exome and –callRegions options with the target bed file. We also leveraged indels called by MANTA (v1.6.0) using the parameters –indelCandidates. A consensus variant call set was obtained by combining Mutect2 and Strelka2 variants using ‘VariantFilter’ (https://github.com/rschenck/VariantFilter). Mutational signatures of SNVs were calculated by deconstructSig R package (v1.8.0)⁶⁷ .

We estimated tumor cell fraction and mean ploidy and found allele-specific copy number aberrations using FACETS SUITE (https://github.com/mskcc/facets-suite, v2.0.8) with an allele-specific CNA caller FACETS (v0.6.1)⁶⁸. First, we generated snp pileup files using snp-pileup-wrapper.R using dbSNP (v138) and default parameters. Next, we ran run-facets-wrapper.R using the following parameters: –purity-cval 600 –cval 300 –normal-depth 40. Loss of heterozygosity (LOH) of HLA genes was defined if the sample has a zero minor copy number in HLA regions (chr6: 29.9 - 31.4Mbps). We used iC10 (R package v1.5) with both RNA and DNA-sequencing data (iC10 DNA+RNA) to predict the integrative clustering (IC) subtypes of breast cancer⁶⁹. We also used EniClust (Houlahan, Mangiante, Sotomayor-Vivas, Adimoelja et al., in revision), which reliably infers IC subtypes from DNA-sequencing data of lesions from different stages of progression and diverse organs (primary and metastatic). To ensure sufficient representation of all subtypes when calling the IC10 subtypes, TONIC was integrated with 1102 breast cancer samples from TCGA as a reference panel. RNA from the two datasets was normalized together to remove batch effects using the variance stabilizing transformation (VST) implemented in DESeq2 (v1.26.0).

HLA typing was performed using Optitype (v1.3.5)⁷⁰. We extracted reads mapping to chromosome 6 and remapped them to HLA reference fasta provided by Optitype using BWA-MEM. We then back-extracted mapped reads using fastq function in samtools (v1.15.1)⁷¹ and used them as input to OptiTypePipeline.py using default parameters. Leveraging somatic SNVs and HLA genotypes, we perform antigen prediction on each sample using antigen.garnish R package run via docker andrewrech/antigen.garnish:2.3.1⁷². We filtered the initial list of putative neoantigens based on default thresholds of foreignness score and agretopicity score, as well as less than 1000nM for binding strength. We called clonal neoantigens using the ccf-annotate-maf function from facetsSuite R package (version 1.0). We categorized neoantigens as clonal if the corresponding SNV was annotated as clonal, and subclonal if otherwise.

RNA sequencing data

We aligned raw reads to reference GRCh38 using STAR (v2.7.9a) and read counts were estimated using RSEM (v1.3.3) and GTF file from GENCODE v39. We calculated transcripts per million (TPM) values using expected read counts and effective lengths of protein-coding genes from RSEM. The cytolytic activity (CA) score was calculated by the geometric mean of TPM values of GZMA and PRF1³⁸. T cell-inflamed gene expression profile (GEP) signatures were calculated by the mean of log10TPM values of 18 previously defined genes⁷³. We followed the TME classification procedure described in the previous study⁷⁴ to define TME subtypes. We scored the 29 signatures defining the TME subtypes, performed median normalization of the resulting scores, and classified them considering the pan-cancer TCGA samples from the original publication as a reference. PAM50 subtypes were computed by genefu R package, integrating TONIC RNA-sequencing with RNA-sequencing of 1102 breast cancer tumors from TCGA to ensure the expected representation of all breast cancer subtypes. The two datasets were normalized using the variance stabilizing transformation (VST) implemented in DESeq2 (v1.26.0). We computed additional Kegg and Biocarta signatures for immune cell polarization and cytokine secretion for a total of 132 RNA-based signatures (Supplementary Table 11).

Feature extraction

For multivariate modeling, we assembled a comprehensive set of features from genomic, transcriptomic, and spatial data.. These included: : tumor cell fraction, mean ploidy, fraction of copy-number alteration (CNA), fraction of LOH, whole genome doubling, integrative cluster (IC) subtype, arm-level CNA, copy number of individual genes from the iC10 package and previous studies^75,76. We also included counts of missense, synonymous, frameshift, clonal, and subclonal mutations; clonal and subclonal neoantigen burden; SNV signatures; HLA type; HLA-binding mutation ratio as previously implemented⁷⁷, and all 134 RNA signatures.

To identify functional modules, we used gene sets defined in MSigDB³², restricting analysis to Cytokine signaling pathways,which are both well-established and complimentary to MIBI data. signatures were calculated per sample using single-sample gene set enrichment analysis (ssGSEA;GSEApy package⁷⁸), allowing correlation of gene set features with MIBI features. We also curated outcome-associated gene sets from prior studies, including a T resident-memory gene signature³¹, a TGFb signature³³, T cell suppression/activation signatures⁷⁹, a T cell exhaustion signature^79,80, and cDC1/cDC2 signatures using a manually curated gene set.

To estimate the abundance of individual cell populations, we conducted a deconvolution analysis utilizing CIBERTSORTx. A TNBC-specific single-cell reference matrix comprising 17 cell types was generated from TNBC samples in a large-scale breast cancer single-cell dataset^78,81. Using this reference, we computed the absolute cell-type scores for each sample.

RNA-MIBI correlations

To identify RNA features linked to the spatial MIBI features, we aggregated ssGSEA scores, literature-curated gene sets, CIBERSORTx⁸² predictions, and TME signatures. We then computed Spearman correlations between all RNA-derived and MIBI features. Correlations with adjusted p ≤ 0.05 and coefficient ≥ 0.6 were defined as strong. We then compared the distribution of RNA and spatial features within this strong correlation set compared to the total feature set to identify patterns in cross-modality relationships.

Univariate outcome associations

Calculating feature associations

To assess links between features and patient outcome, we analyzed MIBI, RNA, and DNA data separately at each timepoint. Outcome was defined as responder versus non-responder status. For each feature, we calculated an independent two-sided t-test (equal variance assumed) to compare means between responders and non-responders. We recorded both the p-value and the difference in medians.

Ranking features

To rank feature importance features, we first performed FDR correction (Benjamini–Hochberg), then ordered features by corrected p-value and by median shift. We averaged these ranks to generate a composite “importance score.” Separate importance scores were calculated for MIBI, RNA, and DNA, across all timepoints, to allow relative comparison. Robustness analyses of the importance score and detailed compartment-specific enrichment are described in our online protocol⁸³.

Multivariate modeling

To predict patient response, we used a Lasso model³⁵, which is well-suited for handling high-dimensional datasets with many features and relatively few data points. The Lasso model’s ability to select the most relevant features enhances its predictive performance in such scenarios. Although we tested alternative models (e.g., XGBoost), Lasso yielded superior performance for our dataset.

Specifically, we utilized stratified cross-validation (CV) to select the level of sparsity in the Lasso model (Extended Data Fig. 9a). As compared to the regular CV procedure, stratified CV maintains a balanced representation of patient responses in each fold. This is achieved by leveraging the available patient response information as a covariate during the fold-splitting process. By stratifying the folds based on the response covariate, we ensure that each fold contains a representative sample of both responders and non-responders, mitigating the potential bias that could arise from an imbalanced distribution. We used a three-fold CV in our experiments. Moreover, in order to draw more robust conclusions, we conducted the same set of experiments 10 times with different random seeds. The Area Under the Receiver Operating Characteristic Curve (AUROC) was used as the primary metric for evaluating model performance. For a more complete description of the individual features underlying the model’s performance, as well as benchmarking of SpaceCat’s performance, please see our online protocol⁸³.

Statistics and Reproducibility

No statistical method was used to predetermine sample size. No data were excluded from the analyses. The experiments were not randomized. The Investigators were not blinded to allocation during experiments and outcome assessment. Data distribution was assumed to be normal but this was not formally tested. Model training was performed multiple times as part of the cross-validation loop (see Methods). All other experiments were performed once.

Extended Data

Extended Data Figure 1. ROI selection and sample summary.

a. Workflow for selecting cores to be included in the study. Hematoxylin and eosin (H&E) slides were digitized in slidescore.com, annotated by a dedicated breast cancer pathologist, and demarcated with areas of interest. Each identified region was punched with a 1.5mm core and placed onto a tissue microarray (TMA).

b. Illustrative examples of the H&E image of the entire core, the cropped inset of the portion of the core selected for MIBI analysis, and the corresponding MIBI image generated from the selected region.

c. Histogram showing the number of fields-of-view (x-axis) acquired on MIBI across all timepoints from all patients (y-axis)

d. Upset plot showing the overlap of distinct timepoints of MIBI data across the patients in the cohort.

e. Venn diagram showing the overlap between DNA, RNA, and MIBI data for baseline samples (N=93 patients)

f. Venn diagram showing the overlap between RNA and MIBI data for pre-nivo samples (N=77 patients)

Same as f) for on-nivo (N=66 patients)

Extended Data Figure 2. Cell clustering scheme and cell type prevalence.

a. Diagram illustrating the three levels of cell clustering (broad, intermediate, detailed), and the relationship between clusters in each level

b. Number of cells of each cell type in the broad clusters

c. Number of cells of each cell type in the intermediate clusters

d. Number of cells of each cell type in the detailed clusterings. b, c, d are all based on all primary and metastatic tumors in the sample set.

e. Mean count of cell types (broad clusters) for each timepoint

f. Mean count of cell types (intermediate clusters) for each timepoint

g. Average proportion of each intermediate cancer subpopulation in metastatic samples (N=101 patients)

h. Average proportion of each intermediate structural subpopulation in metastatic samples (N=101 patients)

i. Average proportion of each intermediate immune subpopulation in metastatic samples (N=101 patients)

j. PD1 expression in Wang et al.

Box plot: Lower bound is 1st quartile, center is median, upper bound is 3rd quartile, whiskers extend to 1.5*IQR beyond bound.

Extended Data Figure 3. Feature extraction pipeline schematics.

a. Cartoon illustrating how compartment masks are defined

b. Table illustrating how functional marker thresholds are used to binarize cells into positive or negative for each marker; these assignments are then used to generate positivity proportion statistics per image

c. Schematic showcasing how diversity scores are calculated

d. Cartoon showing how the radius around a cell is used for neighborhood diversity

e. Cartoon showing how cell distances are used to compute the linear distance feature

f. Cartoon showing how the mixing score is calculated, along with examples of high and low mixing

g. Schematic showing how the fiber segmentation pipeline is used to generate features

h. Schematic showing how the extracellular matrix pipeline is used to generate features

Extended Data Figure 4. Quantification of features across tumor compartments.

a. Heatmap showing features that are consistent across compartments. Each row represents a different feature, and each column is a distinct compartment. Features are normalized to the whole image value

b. Same as above, but for features that are enriched in specific compartment(s)

c. Histogram showing threshold used to identify varying vs. non-varying compartment features

d. Distribution of the CD8 T / CD4 T ratio, a feature that changed across compartments (n=895 features)

e. Distribution of Smooth Muscle density, a feature that changed across compartments (n=1084 features)

f. Distribution of Ki67 positivity in Cancer 1, a feature that did not change across compartments

g. Distribution of PD-L1 positivity in CD68 macrophages, a feature that did not change across compartments (n=956 features)

h. Heatmap showing expression of markers across distinct tile-based clusters (n=507 features)

i. Proportion of TIM3+ Structural cells across ECM neighborhoods (n=227 FOVs)

j. Proportion of TIM3+ CD68 Macrophages cells across ECM neighborhoods (n=200 FOVs)

k. Proportion of PDL1+ CD68 Macrophages cells across ECM neighborhoods (n=200 FOVs)

Box plot: Lower bound is 1st quartile, center is median, upper bound is 3rd quartile, whiskers extend to 1.5*IQR beyond bound.

Extended Data Figure 5. Evolution of the TME through treatment.

a. Summary of features that changed between baseline and on-nivo samples across all patients

b. Ratio of Cancer cells to T cells in stroma border at baseline and on-nivo timepoint across all patients

c. Proportion of all CD45RO+ cells at baseline and on-nivo timepoint across all patients

d. Diversity of cells surrounding Mac_Other cells at baseline and on-nivo timepoint across all patients

e. Summary of features that changed between baseline and on-nivo samples across paired samples(3.5)

f. Ratio of Cancer cells to T cells in stroma border at baseline and on-nivo timepoint across paired samples

g. Ratio of Cancer cells to unclassified cells in stroma border at baseline and on-nivo timepoint across paired samples

h. Ratio of NK cells to T cells at baseline and on-nivo timepoint across paired samples

For panels b-d and f-h, N=101 patients.

Extended Data Figure 6. Relationship between MIBI- and RNA-based features.

a. Distribution of ssGSEA normalized enrichment scores (NES) across MSigDB signatures related to cytokine signaling (N=210 samples)

b. Volcano plot showing correlation coefficient (x-axis) and p-value from an unpaired two-sided t-test (y-axis) for pairs RNA- and MIBI-based features

c. Correlation between HLA1+ cells (MIBI) and MHCI (RNA). Scatter points represent individual data values, the line indicates the linear regression fit, and the shaded area shows the 95% confidence interval of the fit.

d. Correlation between CD45RO+ CD4+ T cells at the border (MIBI) with a TGFb gene score (RNA)

e. Correlation between HLA1+ Cancer cells (MIBI) and interferon signaling (RNA)

f. Correlation between diversity surrounding CD8 T cells in the cancer core (MIBI) with a signature for T cell exhaustion (RNA)

g. Enrichment of MIBI-RNA feature pairs in top correlated features based on the compartment the MIBI feature is defined in

Box plot: Lower bound is 1st quartile, center is median, upper bound is 3rd quartile, whiskers extend to 1.5*IQR beyond bound.

Extended Data Figure 7. Outcomes associations.

a. Enumeration of the number of top MIBI features which are ratios compared to the number which are densities

b. Volcano plot of RNA-seq features associated with response

c. Top RNA features associated with response, organized by biological category

d. Comparison of number of DNA and RNA features associated with response

e. Number of RNA-seq features associated with response by timepoint

f. The number of times a given MIBI feature was repeated across distinct timepoints among the top 100 features

g. Same as Fig. 4b, but with all labels included

h. Types of features associated with response from baseline metastatic timepoint

i. Top 50 features associated with response from baseline timepoint, colored by compartment and sorted by feature type

j. Cancer/Immune mixing score stratified by response status across timepoints: primary (N=51, 7 Yes / 44 No), baseline (N=50, 10 Yes / 40 No), pre nivo (N=39, 7 Yes / 32 No), on nivo (N=37, 7 Yes / 30 No). N=number of patients.

k. Proportion of Ki67+ Cancer 1 cells stratified by response status across timepoints

l. Proportion of GLUT1+ Cancer 1 cells stratified by response status across timepoints

Box plot: Lower bound is 1st quartile, center is median, upper bound is 3rd quartile, whiskers extend to 1.5*IQR beyond bound.

For j and k: primary (N=59, 9 Yes / 50 No), baseline (N=74, 12 Yes / 62 No), pre nivo (N=67, 11 Yes / 56 No), on nivo (N=57, 8 Yes / 49 No). N=number of patients

Extended Data Figure 8. Timepoint-specific features.

a. Box plots showing the distribution of the B / NK ratio in the cancer core feature in responders and non-responders, across the four distinct timepoints in the cohort, primary (N=21, 4 Yes / 17 No), baseline (N=15, 5 Yes / 10 No), pre nivo (N=13, 2 Yes / 11 No), on nivo (N=15, 4 Yes / 11 No).

b. Same as a), with CD68 Macrophage density as the feature, primary (N=59, 9 Yes / 50 No), baseline (N=79, 12 Yes / 67 No), pre nivo (N=70, 13 Yes / 57 No), on nivo (N=62, 12 Yes / 50 No)

c. Same as a), with Vim+ in CD4 T cells as the feature, primary (N=53, 7 Yes / 46 No), baseline (N=58, 11 Yes / 47 No), pre nivo (N=52, 9 Yes / 43 No), on nivo (N=52, 12 Yes / 40 No)

d. Same as a), with Structural cell density in the stroma border as the feature, primary (N=59, 9 Yes / 50 No), baseline (N=76, 12 Yes / 64 No), pre nivo (N=66, 12 Yes / 54 No), on nivo (N=58, 8 Yes / 50 No)

e. Same as a), with Other / Cancer cell ratio in the stroma core as the feature, primary (N=59, 9 Yes / 50 No), baseline (N=78, 12 Yes / 66 No), pre nivo (N=69, 13 Yes / 56 No), on nivo (N=62, 12 Yes / 50 No)

f. Same as a), with Nk / Other cell ratio in the stroma border as the feature, primary (N=56, 7 Yes / 49 No), baseline (N=72, 12 Yes / 60 No), pre nivo (N=65, 12 Yes / 53 No), on nivo (N=53, 7 Yes / 46 No)

Box plot: Lower bound is 1st quartile, center is median, upper bound is 3rd quartile, whiskers extend to 1.5*IQR beyond bound. N=number of patients for all.

Extended Data Figure 9. Multivariate modeling approach and benchmarking.

a. Diagram illustrating the cross validation approach used for model training

b. Histogram showing the number of times the top features were selected across the 10 different iterations of model training

c. Comparison of the importance score of the top features identified by the model across distinct timepoints, stratified by MIBI vs. RNA

d. Overlap across timepoints of top features identified by the MIBI models

e. Overlap across timepoints of top features identified by the RNA models

f. AUROC evaluated for the on-nivo MIBI data using separate train, val, test split without cross validation.

g. Same as above, for AUPRC

h. Accuracy of multivariate model with decreasing number of retained features

i. Relative accuracy of combined MIBI + RNA model compared to best performing unimodal model

Box plot: Lower bound is 1st quartile, center is median, upper bound is 3rd quartile, whiskers extend to 1.5*IQR beyond bound.

Extended Data Figure 10. Comparison with previously identified predictive features.

a. Overlap in pre-treatment predictive features originally identified in Wang et al.

b. Overlap in pre-treatment predictive features after running SpaceCat on the data from Wang et al.

c. Proportion of Ki67+ Cancer cells stratified by response across timepoints in Wang et al., baseline (N=109, 55 pCR / 54 RD), on treatment (N=67, 28 pCR / 39 RD)

d. Proportion of Ki67+ CD8 T cells stratified by response across timepoints in Wang et al., baseline (N=113, 56 pCR / 57 RD), on treatment (N=97, 48 pCR / 49 RD)

e. Density of Cancer cells stratified by response across timepoints in Wang et al., baseline (N=108, 54 pCR / 54 RD), on treatment (N=93, 47 pCR / 46 RD)

f. Comparison of SpaceCat features, original features, and combined features to predict outcome in Wang et al. data

g. Comparison of SpaceCat features, Wang et al. features, and combined features to predict outcome in TONIC data

h. Overlap in on-treatment predictive features originally identified in Wang et al. (4.2)

i. Overlap in on-treatment predictive features after running SpaceCat on the data from Wang et al.

j. Distribution of top features across timepoints when using only features defined in Wang et al.

k. Change in top 100 predictive features when including cell subsets defined in Wang et al.

l. Change in multivariate model specific cell clusters that are predictive, as well as multivariate results

Box plot: Lower bound is 1st quartile, center is median, upper bound is 3rd quartile, whiskers extend to 1.5*IQR beyond bound. N=number of patients.

Data availability

All processed imaging data from this study, including antibody staining, cell segmentation masks, and SpaceCat outputs, is publicly available at: https://www.ebi.ac.uk/biostudies/bioimages/studies/S-BIAD1288. All image analysis files are available at https://zenodo.org/records/17065908

The DNA and RNA sequencing data, as well as the patient response information, is available for academic use subject to the limitations of the provided informed consent. The RNA- and DNA-sequencing data from tumor biopsies of the TNBC patients treated in the TONIC-1-trial stage 1 are deposited at the European Genome-phenome Archive (EGA) under accession number EGAS0001003535. The RNA and DNA-sequencing data from the TNBC patients treated in TONIC-1-trial stage 2 are not yet deposited in a public repository pending ongoing work. For both the already deposited and the not yet deposited sequencing data and source data supporting the findings of this study will be made available from the corresponding author (m.kok@nki.nl) for academic use, within the limitations of the provided informed consent. Data will not be made available for commercial use. A first response to the request will be sent in <4 weeks. Data requests will be reviewed by the corresponding author and Institutional Review Board of the NKI and after approval, applying researchers will have to sign a data transfer agreement with the NKI.

Code availability

The code to generate the figures in this paper is available at: https://github.com/angelolab/publications/tree/main/2024-Greenwald_Nederlof_etal_TONIC

The low-level processing code is available at: https://github.com/angelolab/toffy

The segmentation and cell assignment pipelines are available at: https://github.com/angelolab/ark

SpaceCat is available at: https://github.com/angelolab/SpaceCat