Weak supervision of H&E slides reveals systems-level biology and functional states that govern therapeutic resistance

Abstract

Precision oncology lacks scalable tools to assess, at the patient level, systems-level tumor microenvironment (TME) programs driving therapeutic resistance. To address this gap, we trained a weakly-supervised deep learning model, using routine H&E slides as input, to derive quantitative activity for therapeutically-relevant TME phenotypes, spanning immune, metabolic, and tumor cell-intrinsic programs. Using 3111 breast cancer H&E WSIs with matched bulk transcriptomics, our model accurately infers these biological states (AUROC>0.80; PCC>0.64). Validation spanned three levels: (i) tissue-matched multiplexed immunofluorescence, showing concordance between inferred functional states and immune cell fractions (p=0.006–0.106), (ii) blinded reader assessments, confirming localization of phenotype-specific morphology (p<3×10⁻⁵), and (iii) multi-institutional patient cohorts, where model-derived phenotypes stratified for clinical response (p<0.045). Despite relying on slide-level labels for training, our model’s attention mechanism identifies focal regions of tumor tissue that drive the overarching clinical phenotype or treatment response. By extracting spatially resolved TME biology from routine histology, this strategy can be applied to massive legacy biobanks to enable discovery of new morpho-molecular mediators of resistance across real-world patient populations.

INTRODUCTION

A fundamental goal of precision oncology is to identify the most effective drug for individual patients based on their unique biological vulnerabilities. Multi-omic studies of tumor tissue have established that therapeutic resistance in clinical oncology is mediated by multifaceted TME programs^1,2. However, while single-cell and spatial profiling provide these deep biological insights, these methods are too resource-intensive to be integrated into everyday clinical care. This logistical barrier creates an “implementation gap” where the biological programs that govern therapeutic resistance are known but remain inaccessible for informing clinical decision-making for vast majority of cancer patients. This unmet need results in suboptimal patient stratification for experimental therapeutics and wide variability in clinical outcomes among patients with ostensibly similar biomarker profiles.

Closing this “implementation gap” requires an assay that enables cost-effective and comprehensive assessment of the TME for each patient. In this work, we demonstrate that a deep learning (DL) model, trained using weakly supervised learning on widely available H&E whole slide images (WSIs), can accurately recover the systems-level TME biology that drives therapeutic resistance for cancer patients. By leveraging ubiquitous histology as a scalable surrogate for high-dimensional Omics profiling, our framework overcomes the logistical bottlenecks that have historically restricted TME assessments to small research cohorts. This approach offers a major upgrade over current clinical tools, which provide only a fragmented view of tumor biology and often neglect the TME.

The basis of our approach is the premise that routine H&E slides harbor critical elements of patient-specific TME biology, such as tissue architecture and the spatial patterns of varied cell populations. However, manual review of every high-power field to assess these structures within a WSI is fundamentally impractical. The immense volume of biological data within a single slide prevents human pathologists from reproducibly interpreting and aggregating a WSI’s histological structures into the quantitative metrics needed for precise treatment decisions. While cross-model foundation models that align the latent spaces of H&E slides with spatial transcriptomics (ST) illustrate promise in quantifying therapeutic insights from DL-enabled analysis of routine histology^3,4, these frameworks are limited by the scarcity of paired H&E-ST training data. To bypass this limitation, we developed a scalable alternative: a DL model trained through weak supervision on slide-level labels to quantify TME programs broadly linked to therapeutic resistance.

We used breast cancer, the tumor type with the highest incidence rate globally, as our initial application for solid tumors because of its high degree of biological heterogeneity, which is incompletely captured by standard profiling methods (e.g., estrogen and progesterone receptor immunohistochemistry [IHC], HER-2 IHC and FISH). By training on a cohort of patient-matched bulk RNA sequencing and breast cancer H&E WSIs, our approach extends prior computational pathology paradigms, which typically predict isolated mutation or gene expression snapshots^5–7, to capture the functional biology implicated in therapeutic resistance. Furthermore, by aggregating genes into coordinated TME phenotypes using gene ontology analyses, we ensure that the outputs of our model are therapeutically informative and less susceptible to the noise inherent in single-gene analyses^8,9. By prioritizing the learning of functional biology states over task-specific outcomes, our model provides a versatile platform to enhance patient stratification across virtually every treatment modality. To illustrate, using model-derived TME phenotypes, we interrogate therapeutic efficacy for triple-negative breast cancer (TNBC) patients. This aggressive molecular subtype remains a major challenge, given high recurrence rates and the lack of robust predictive biomarkers of efficacy for standard-of-care therapies.

We demonstrate the translational utility of this platform across three validation levels: (i) validating model-derived phenotypes with tissue-matched multiplexed immunofluorescence (mIF); (ii) confirming the identification of tissue regions enriched for phenotype-specific biology using a blinded pathologist study; and (iii) establishing clinical utility through patient stratification in multi-institutional cohorts for regimens with limited predictive biomarkers of efficacy (Figure 1). By deriving spatially resolved and clinically relevant TME biology directly from routine histology, our framework can be applied to study therapeutic resistance across existing large databases where high-resolution spatial profiling remains cost prohibitive. Collectively, these findings establish weakly supervised DL analysis of H&E slides as a scalable, cost-effective discovery engine for assessing TME programs pertinent to resistance across the pan-cancer landscape.

Figure 1: Study design, model training, and validation strategy.

A) H&E WSI model development. The ground truth for each H&E WSI was defined by tissue-matched bulk RNA-sequencing data. Following tissue segmentation and quality control, non-overlapping patches of 256 × 256 pixels were extracted from the tissue regions of the WSIs. Color augmentation and geometric transforms were applied to these image patches to mimic common sources of variability of H&E WSIs. Patch level features were extracted via an image encoder, and attention-weighted aggregation was employed to generate a single slide-level feature vector. This vector was passed to task-specific prediction heads using an attention-based multiple instance learning (MIL) framework. Binary classification and regression models were trained to quantify each TME phenotypic activity, with performance evaluated using area under the receiver operating characteristic (AUROC) and Pearson’s correlation coefficient (PCC) metrics, respectively.

B) Model validation. The ability of our model to accurately resolve systems-level TME phenotypes with weak supervision was validated using four distinct methods.

Internal validation: Model performance was assessed on the held-aside test set, where predictions were compared directly with ground-truth transcriptomic-derived phenotypic activity.

Validation of functional biology: The model’s ability to disentangle different immune phenotypes was assessed using tissue matched multiplexed IF from our multi-institutional cohort (n = 25). PCCs were calculated to evaluate the association between model-derived phenotypes and corresponding mIF-defined cellular fractions (CD19/20/45+ B cells; CD4/GZMB+ and CD8/GZMB+ cytotoxic T cells; CD4/FoxP3+ regulatory T cells)

Validation of attention maps: A blinded reader study was conducted to assess the biological fidelity of attention maps. For fifty randomly selected H&E WSIs within our test set, two pathologists blinded to the model output evaluated three groups of patches (e.g., those with the highest attention scores, the lowest attention scores, and randomly sampled patches) to determine if observed histologic features were consistent with known phenotypic activity.

Validation on multi-institutional patient cohorts: Model generalizability and clinical utility were evaluated on multi-institutional cohorts (n = 199) treated with either neoadjuvant pembrolizumab and chemotherapy (Keynote-522) or adjuvant dose-dense chemotherapy. Clinical outcomes were correlated with model-derived immune and cell cycling phenotypes, in accordance with established Omics correlates of therapeutic efficacy.

RESULTS

Deep learning accurately infers systems-level TME biology from H&E WSIs.

To establish a scalable platform for TME assessment at the patient level, we developed a weakly supervised framework trained on a cohort of patient-matched breast cancer H&E WSIs (n = 3111 WSIs; 1098 patients). Using tissue-matched bulk RNA-sequencing data, we defined ground truth functional states for ten TME phenotypes spanning tumor-intrinsic, immune, and metabolic programs using single-sample gene set enrichment analysis (ssGSEA). On the held-out test set, our DL models achieved strong performance in quantifying systems-level biology (Figure 2). Performance was similar across biological systems, including immune (area under receiver operating characteristic curve [AUROC]: 0.861 {95% confidence interval: 0.805–0.907}; Pearson’s correlation coefficient [PCC]: 0.789 {0.731–0.840}, cancer cell-intrinsic (AUROC: 0.861 [0.811–0.911]; PCC: 0.744 [0.671–0.802]), and metabolic phenotypes (AUROC: 0.843 [0.791–0.890]; PCC: 0.741 [0.664–0.812]). Subgroup analysis across our test set revealed receptor profile-specific differences in performance, though this conclusion is tempered by the small number of patients in the HR+/HER2- cohort (Supplemental Table 1). Notably, our model’s performance exceeded that of prior histology-based efforts focused on predicting individual gene or protein expression^5,7, highlighting the potential advantages of phenotype-level supervision compared to isolated molecular snapshots.

Figure 2: Deep learning infers therapeutically relevant systems level tumor microenvironment biology from routine H&E WSIs.

Model performance is shown for ten pre-defined TME phenotypes, spanning immune, tumor, and metabolic phenotypes, relevant for therapeutic resistance across the pan-cancer landscape. For each phenotype, model performance is reported for binary classification (AUROC; purple) and regression (PCC, teal). Box-and-whisker plots display the median and 95% confidence intervals estimated via bootstrap resampling across the held-aside test set. Across all tasks, models achieved AUROC > 0.80 for binary classification and PCC > 0.64 for regression, suggesting that tissue architecture and spatial distribution of cell populations within H&E slides encode system levels TME biology.

Models using tabular clinico-genomic data commonly used for patient stratification showed inferior performance to that of WSI-only models (Supplemental Table 2). Notably, we observed relatively stronger performance (AUROC 0.70–0.80) for cell cycling and glycolysis phenotypes for the tabular models, which likely reflects higher expression of these programs in certain molecular subtypes. For example, TNBCs frequently exhibit elevated aerobic glycolysis, which supports rapid proliferation and immunosuppression, compared to hormone receptor-positive tumors^10,11. Consistent with these known trends, 75.62% and 77.50% of TNBC cases in our dataset exhibited up-regulation of cell cycling and glycolytic activity, respectively. Collectively, these findings illustrate that biomarkers used in patient selection criteria in clinical trials possess limited ability to quantify the full spectrum of an individual patient’s tumor biology. To address this unmet need, weakly supervised DL-based analysis of H&E WSIs provides a scalable high-dimensional lens for systems-level biology across large patient populations.

We evaluated several strategies to optimize model performance. First, we compared open-source image encoders to identify the optimal feature representation for TME phenotyping. Since the CONCH vision-language foundation model outperformed both PLIP and ResNET-50 backbones (Supplemental Table 3), all subsequent experiments in this work were performed using CONCH-derived features. Next, to further improve model generalization, we implemented label-preserving data augmentation strategies, including patch-level geometric transformations and color augmentations. These augmentations improved performance for most phenotypes compared to models trained without them (Supplemental Figure 1). Finally, as previously described in prior work¹², we compared performance of attention-based MIL (AM-SB) with a vision transformer-based architecture (TransMIL). We observed no meaningful performance differences between these architectures, suggesting that at the current dataset size, explicitly modeling long-range spatial dependencies does not confer additional benefit for assessing systems-level resistance biology.

Multimodal integration of H&E WSIs and clinico-genomic data does not confer a performance gain over H&E WSI-based models

We next evaluated whether the integration of routinely acquired clinico-genomic metadata (e.g., age, tumor grade, stage, receptor profile) with H&E WSIs improves inference of TME phenotypes. We evaluated two multi-modal fusion strategies: (i) intermediate cross-attention fusion, adapted from the SurvPath architecture¹³ to facilitate learning of inter-modality interactions, and (ii) late fusion, where modality-specific predictions were combined with a multi-layer perceptron. Performance of these multimodal models was benchmarked against the H&E WSI model on the held-out test set.

Across nearly all phenotypes, multimodal models failed to yield measurable improvements over the H&E WSI baseline. In several instances, particularly with the multimodal cross-attention models, the inclusion of clinical features reduced model performance (Supplemental Figure 2). Feature ablation analyses further revealed that predictive signal in the clinico-genomic models was driven primarily by molecular data (e.g., ER, PR, HER-2 status), with negligible contributions from demographics or tumor stage (Supplemental Table 4). Moreover, the addition of any clinico-genomic feature category to the multimodal models provided no measurable improvement (Supplemental Table 5). Together, these results suggest that routine histology is an information-rich modality that already encapsulates the biological information inherent within a patient’s clinico-genomic status. Consequently, multimodal integration of redundant or uninformative data may introduce spurious associations that detract from the biological signal present in the H&E WSI.

Model-derived phenotypic assessments are linked with functional protein expression

To provide orthogonal validation that model-derived phenotypes reflect underlying functional biology, we performed tissue-matched mIF on a subset of our ddACT cohort (n = 25). This mIF panel defined B cells, cytotoxic T cells, and regulatory T cells (Tregs) by employing antibodies against CD4, CD8, CD19, CD20, CD45, granzyme B (GZMB), and FoxP3. While these markers quantify canonical cell populations related to our model’s immune phenotypes, the transcriptomic signatures used to train our model encompass a broader range of functional genes. Despite this difference in scale and the potential for discordant mRNA-protein expression due to post-transcriptional regulation, we reasoned that accurately captured model-derived immune phenotypes should demonstrate meaningful alignment with corresponding protein-level states. Therefore, we calculated PCCs to evaluate the association between model-derived phenotypes and corresponding mIF-defined cellular fractions.

First, model-derived T-cell cytotoxicity expression was significantly linked with the fraction of cytotoxic T cells (PCC: 0.534; p = 0.006; Table 1), suggesting that our model effectively recovers state-specific immune function from weak supervision of H&E WSIs (Supplemental Figure 3). Similarly, FoxP3-mediated immunosuppression was significantly associated with the fraction of Tregs (PCC: 0.437; p = 0.029; Table 1, Supplemental Figure 4). Qualitative review of discrepant cases identified FoxP3 expression in other cell populations (e.g., CD11b+ myeloid cells and PanCK+ tumor cells; Supplemental Figure 5). These findings suggest that our model captures a holistic immunosuppressive program associated with FoxP3 expression across multiple TME compartments, rather than simply predict the abundance of Tregs. The biological rationale for these associations across related but distinct measurement scales is expanded upon in the Discussion.

Table 1: Model-derived phenotypic activity is linked with multiplexed immunofluorescence-defined fractions of canonical cell populations.

Cellular fractions were derived from 25 tissue-matched tumor samples within the ddACT cohort. mIF gating strategies defined B cells as CD19/CD20/CD45-positive; cytotoxic T cells as CD4/GZMB-positive and CD8/GZMB-positive; and regulatory T cells as CD4/FoxP3-positive. Fractions were calculated relative to the total cell count per sample. Statistically significant positive correlations were observed between model-derived T-cell cytotoxicity and mIF-defined cytotoxic T cell fractions (r = 0.534; p = 0.006), as well as model-derived FoxP3-mediated immunosuppression and fraction of Tregs (r = 0.437; p = 0.029). B-cell proliferation demonstrated a positive trend with the mIF-defined B-cell fraction (r = 0.331; p = 0.106), perhaps reflecting the functional divergence between cellular density of B-cells and transcriptomic-based proliferative activity.

Sample ID	B-cell proliferation		T-cell cytotoxicity		FoxP3-mediated immunosuppression
	Fraction of B cells (%)	Model-derived quantitative phenotypic activity	Fraction of cytotoxic T cells (%)	Model-derived quantitative phenotypic activity	Fraction of regulatory T cells (%)	Model-derived quantitative phenotypic activity
TN04	1.0143	0.9109	12.0239	2.1544	6.4866	0.9188
TN12	0.0883	0.1827	2.0002	0.7356	0.7282	0.0488
TN19	0.4905	0.0379	3.9243	0.3316	2.2547	−0.1887
TN21	0.0508	−0.3256	3.4961	−0.2867	0.6418	0.0493
TN27	0.0311	−0.1267	2.3280	−0.0431	3.4717	−0.1694
TN30	0.0024	−0.1066	0.9036	0.3811	0.8560	0.1063
TN52	0.3755	0.5489	5.7815	1.5972	1.4326	0.4705
TN53	8.9977	1.2606	12.7830	2.1239	16.5946	1.5677
TN54	0.7880	−0.0644	7.2795	1.9053	1.8355	0.4142
TN57	0.0652	0.3018	4.5408	2.3541	1.3257	1.2354
TN63	1.7561	1.1876	5.8682	2.3274	3.8394	1.0402
TN64	0.0496	1.1784	2.4034	2.1693	1.6169	1.1857
TN66	0.2012	1.4543	0.6626	1.5997	6.7022	0.2734
TN68	0.1140	0.4629	1.7861	1.6275	0.3706	0.6477
TN69	0.0236	1.0559	0.9601	1.7683	1.1466	1.0364
TN74	0.0028	0.8243	0.7056	0.9206	4.0023	0.9460
TN75	1.5887	0.7639	11.9110	1.9903	4.4656	0.7061
TN80	1.6593	0.5888	7.1477	1.8002	2.4287	0.7171
TN81	25.9727	1.0710	5.8876	1.8973	1.5193	1.2011
TN93	0.1006	0.0911	3.9186	1.6058	0.5904	0.4003
TN95	0.0000	−0.3395	1.0876	1.5165	0.4124	0.1007
TN97	0.1181	0.9587	8.8162	2.2929	4.0073	0.9501
TN101	3.4829	1.2681	11.2163	2.1634	10.2394	1.1150
TN107	0.0343	0.2184	0.6575	0.2063	4.6852	0.0785
TN109	1.0784	0.5645	2.1096	1.8215	1.1409	0.7320

	Pearson’s correlation coefficient (r)	Statistical Significance
B-cell proliferation	0.331	p = 0.106
T-cell cytotoxicity	0.534	p = 0.006
FoxP3-mediated immunosuppression	0.437	p = 0.029

Finally, B-cell proliferation demonstrated a positive trend with mIF-derived B cell fractions (PCC: 0.331; p = 0.106; Table 1, Supplemental Figure 6). This association might be explained by inherent differences between total cellular density, as quantified by CD19, CD20, and CD45 markers, and the integrated proliferative activity captured by the model’s transcriptomic-based supervision. While mIF provides a static census of B cells, the transcriptomic labels used for supervision encompass a 98-gene signature that represent functional cell-cycling programs not explicitly captured by B-cell protein expression alone. Nonetheless, the observed alignment across these disparate measurement scales (e.g., systems-level biology vs. fractions of cell populations) add to the body of evidence that weak supervision of routine H&E WSIs can successfully disentangle clinically relevant immune phenotypes.

Attention mechanisms reliably localize tissue regions enriched with phenotype-specific biology.

To determine whether our model’s predictions were driven by biologically relevant features rather than spurious correlations^14,15, we leveraged the model’s attention mechanism, which assigns a learnable weight to each image patch reflecting its contribution to the final WSI-level score. By projecting these weights back onto the original WSIs, we generated attention maps for H&E WSIs in our test set to evaluate whether high-scoring regions corresponded to recognizable histologic hallmarks. These maps consistently highlighted spatially localized regions corresponding to recognizable histologic hallmarks, such as dense lymphocytic aggregates for T-cell cytotoxicity and hypercellular tumor regions for cell cycling (Figure 3). Notably, the same image patch often received distinct attention scores across different phenotypes, demonstrating the model’s capacity to spatially localize multiple biological programs within a single WSI. In contrast, low-attention regions exhibited tissue morphologies irrelevant to the phenotype of interest.

Figure 3: Weakly supervised attention mechanisms localize phenotype-specific TME biology within H&E WSIs.

(A-C) Three representative cases illustrate the spatial localization of four TME functional programs derived from weak supervision on slide-level labels. For each case, an H&E WSI thumbnail (left) is presented alongside phenotype-specific attention heatmaps, and the top three high- and low-attention patches for each phenotype. The slide-level phenotype expression score, derived from tissue-matched bulk transcriptomics and used as ground truth, is labeled directly on each map in Red (Low expression) or Blue (high expression) text. The heatmap color scale (bottom) denotes patch-level attention weights from low (blue) to high (red).

Across phenotypes, high-attention regions consistently correspond to visually explicit histologic structures consistent with the TME program of interest, such as hypercellular tumor regions for tumor-intrinsic phenotypes and lymphocyte-rich aggregates for immune phenotypes. High-attention regions additionally demonstrate distinct morphology, such as epithelioid cells for low expression of EMT and spindled, mesenchymal-like cells for high EMT expression. In contrast, low-attention regions were correctly assigned to morphologically uninformative areas, including adipose tissue, acellular eosinophilic material, or stroma. Together, these visualizations illustrate that attention scores prioritize tissue regions most informative for phenotype inference. Notably, the same image patch often received different, phenotype-specific attention scores, underscoring the model’s ability to spatially localize and assess different TME programs within the same WSI.

To quantitatively evaluate the ability of attention to accurately identify phenotype-specific morphology, we conducted a blinded reader study where two board-certified pathologists (C.M.P; J.A.H.) evaluated groups of high-attention, low-attention, and randomly sampled patches. Three phenotypes with recognizable morphologic correlates were evaluated: B-cell proliferation (lymphocytic infiltrates), epithelial-mesenchymal transition (invasive tumor cells and stromal fibroblasts), and angiogenesis (blood vessels with tumor cells). Pathologist assessments using high-attention patches consistently outperformed assessments using randomly sampled and low attention patches across all evaluated tasks (Table 2A). For instance, high-attention patches reached a classification accuracy of up to 82% for B-cell proliferation, whereas low-attention patches were frequently determined to be “not assessable” due to irrelevant tissue morphology. Pairwise chi-squared testing confirmed significant improvements in reader performance with high-attention patches compared to both comparator groups (Table 2B; all p-values < 2.408 × 10⁻⁵).

Table 2: Blinded reader evaluations from two independent raters demonstrate that attention mechanisms identify patches within an H&E WSI enriched for task-specific biology.

A) Reader assessments with high-attention patches consistently outperform assessments with randomly sampled or low-attention patches for tasks with visually explicit histologic patterns. B) Pairwise chi-squared comparisons across patch types show significant improvements in reader performance with high-attention patches, and significant reduction in performance with low-attention patches. Together, these results illustrate that attention mechanisms prioritize regions enriched for task-specific histology and de-emphasize less relevant tissue.

Task	Image patches used for Reader evaluations	Correct assessment (Rater 1 | Rater 2)	Incorrect assessment (Rater 1 | Rater 2)	Unable to discern (Rater 1 | Rater 2)	Percentage correct (Across both Raters)
B-cell proliferation	High-attention patches	41 | 39	9 | 11	0 | 0	80%
	Randomly sampled patches	29 | 23	19 | 20	2 | 7	52%
	Low attention patches	10 | 5	12 | 9	28 | 36	15%
Cell cycling	High-attention patches	35 | 33	15 | 16	0 | 1	68%
	Randomly sampled patches	23 | 24	23 | 17	4 | 9	47%
	Low attention patches	13 | 6	6 | 7	31 | 37	19%
Angiogenesis	High-attention patches	31 | 32	18 | 14	1 | 4	63%
	Randomly sampled patches	21 | 13	22 | 22	7 | 15	34%
	Low attention patches	15 | 4	6 | 8	29 | 38	19%

Task	Pairwise comparison	Pairwise chi-squared p-value
B-cell proliferation	High attention vs. randomly sampled patches	p = 2.675 * 10−5
	High attention vs. low attention patches	p = 2.754 * 10−24
	Low attention vs. randomly sampled patches	p = 2.467 * 10−15
Cell cycling	High attention vs. randomly sampled patches	p = 4.853 * 10−4
	High attention vs. low attention patches	p = 1.910 * 10−22
	Low attention vs. randomly sampled patches	p = 2.110 * 10−14
Angiogenesis	High attention vs. randomly sampled patches	p = 2.408 * 10−5
	High-attention vs. low attention patches	p = 5.630 * 10−19
	Low attention vs. randomly sampled patches	p = 5.860 * 10−10

Model-derived phenotypic signatures stratify for efficacy across multi-institutional cohorts

We evaluated the clinical utility and generalizability of our H&E WSI model using data from 199 patients treated at Massachusetts General Hospital (MGH), Brigham & Women’s Hospital (BWH), and Ohio State University. Notably, while many DL models predict clinical outcomes directly, such approaches are often limited by data scarcity and tend to produce narrow drug- or task-specific models with limited generalizability. In contrast, our framework was trained to recover fundamental TME phenotypes. As these phenotypes are linked to therapeutic resistance across the pan-cancer landscape, their association with clinical efficacy would emerge naturally without the need for treatment-specific supervision. Here, our clinical validation focused on two treatment scenarios where predictive biomarkers of efficacy remain limited: (i) the neoadjuvant KEYNOTE-522 regimen (pembrolizumab with chemotherapy, followed by definitive surgery) in stage II/III TNBC, and (ii) adjuvant dose-dense anthracycline and taxane-based chemotherapy (ddACT) in high-risk, early-stage TNBC.

For the KEYNOTE-522 cohort, we curated pre-treatment H&E WSIs from core biopsies of 109 TNBC patients who received neoadjuvant pembrolizumab with chemotherapy. Outcomes were dichotomized as pathologic complete response (pCR) or residual disease at time of definitive surgery. Given established associations between immunogenic tumor profiles and checkpoint inhibitor (ICI) efficacy^16,17, analyses were restricted a priori to model-derived immune phenotypes. Consistent with existing data, tumors achieving a pCR demonstrated higher model-derived activity for B-cell proliferation (p = 0.017), T-cell cytotoxicity (p = 0.020), and antigen processing and presentation (p = 0.045). Conversely, tumors from patients with residual disease exhibited higher model-derived FoxP3-mediated immunosuppression activity (Table 3A; p = 0.004). These associations demonstrate that biological programs inferred through weakly supervised models can serve as predictors of ICI response, even in limited tissue contexts as seen in core biopsies.

Table 3: Model-derived quantitative TME phenotypes stratify for therapeutic outcomes across multi-institutional cohorts for therapeutic regimens with limited predictive biomarkers of efficacy.

Values represent median model-derived TME phenotype activity scores (95% confidence interval [CI]) estimated from pre-treatment H&E WSIs within each clinical outcome group. P-values were calculated using two-sided Wilcoxon rank-sum tests.

A) KEYNOTE-522 (neoadjuvant pembrolizumab + chemotherapy): Pre-treatment core biopsies from patients achieving a pathologic complete response (pCR) demonstrated significantly higher model-derived immunogenic activity compared with biopsies from patients with residual disease at surgery.

B) Adjuvant dose-dense chemotherapy: Pre-treatment H&E WSIs from patients achieving long-term remission after dose-dense chemotherapy exhibited a trend towards higher immunogenic model-derived phenotypes compared with those from patients who subsequently relapsed.

A)
Tasks	Model-derived quantitative expression for pathologic complete response cohort (n = 57 patients): Median (95% CI)	Model-derived quantitative expression for residual disease cohort (n = 52 patients): Median (95% CI)	Statistical Significance
B-cell proliferation	0.831 (0.534, 1.127)	0.251 (-0.021, 0.523)	p = 0.017
T-cell mediated cytotoxicity	0.365 (0.040, 0.689)	−0.160 (-0.461, 0.141)	p = 0.020
FoxP3-mediated immunosuppression	−0.135 (-0.421, 0.150)	0.454 (0.107, 0.802)	p = 0.004
Antigen processing and presentation	0.543 (0.224, 0.862)	0.061 (-0.255, 0.376)	p = 0.045
B)
Tasks	Model-derived quantitative expression for Remission cohort (n = 50): Median (95% CI)	Model-derived quantitative expression for Recurrence cohort (n = 29): Median (95% CI)	Statistical Significance
B-cell proliferation	0.559 (0.414, 0.703)	0.431 (0.098, 0.642)	p = 0.186
T-cell mediated cytotoxicity	1.763 (1.545, 1.980)	1.597 (1.170, 2.024)	p = 0.355
FoxP3-mediated immunosuppression	0.585 (0.456, 0.713)	0.611 (0.179, 0.692)	p = 0.366
Antigen processing and presentation	1.390 (1.155, 1.627)	1.089 (0.460, 1.276)	p = 0.360
Cell cycling	−0.800 (−0.667, −0.932)	−0.880 (−0.761, −0.964)	p = 0.106

For the adjuvant ddACT cohort, we curated pre-treatment H&E WSIs from 79 high-risk, early-stage TNBC patients with a minimum of seven years of follow-up. Outcomes were dichotomized as remission or relapse, defined by local recurrence or metastatic progression. Based on suggested associations between chemotherapy efficacy with immune activation and tumor proliferative capacity^18–20, analyses for this cohort were restricted a priori to immune and cell cycling phenotypes. While some associations in this cohort trended towards expected biological signatures, these results did not reach the same level of statistical power observed in the KEYNOTE-522 cohort (Table 3B). This discrepancy likely reflects the inherent differences between the two clinical endpoints. While pCR represents a standardized assessment of response at a pre-defined time point, disease recurrence is a longitudinal event subject to temporal heterogeneity across different patients. In addition, the extended time interval associated with relapse likely introduces additional biological variables that may attenuate the predictive signal of a baseline, pre-treatment biopsy. Nonetheless, these findings suggest that DL-enabled analysis of H&E WSIs can bridge the gap between widely available H&E slides and resource-intensive Omics profiling. By prioritizing the learning of clinically relevant biological states over task-specific outcomes, our framework provides a versatile and scalable platform to enhance patient stratification across diverse treatment modalities.

DISCUSSION

While precision oncology relies on established companion diagnostics (e.g., activating point mutations, fusion, PD-L1 expression), these markers do not capture the broader, systems-level programs that drive therapeutic response. Assessing these biological programs typically requires high-resolution profiling assays that remain restricted to small research cohorts and inaccessible in routine practice. To close this “implementation gap”, we demonstrated that a weakly-supervised DL framework can accurately infer patient-specific functional biology directly from ubiquitous H&E slides. While histopathology has historically served as a qualitative diagnostic tool, our results illustrate that the architectural and morphologic patterns within H&E WSIs encode systems-level biology that govern therapeutic resistance. More importantly, we demonstrate that models can learn these functional states using only slide-level labels for training, thus providing a scalable avenue to obtain clinically relevant biology for cancer patients. In contrast, models trained on tabular clinico-genomic data, the current gold standard for determining trial eligibility, showed limited ability to quantify these complex systems-level programs. Therefore, integrating our H&E WSI model into clinical trials would be a scalable strategy to move beyond coarse eligibility criteria by enriching for patients whose TME biology aligns with the experimental therapeutic’s mechanism of action.

A key advantage of our approach is the use of systems-level biology, rather than isolated molecular snapshots, as ground truth labels. Models trained with single-gene expression^5–7 possess limited biological significance because of functional redundancy (e.g., if one gene is downregulated, others in that pathway may compensate) and the susceptibility of single-gene measurements to technical variance and stochastic biological fluctuations across sequencing platforms^8,9. By using systems-level phenotypic activity as supervision, our model can better capture the true perturbations in functional biology underlying therapeutic efficacy. Consistent with this rationale, our pathway-level models outperform historical single-gene benchmarks^5,7, achieving higher AUROC and PCC across nearly all TME phenotypes. Furthermore, prioritizing the learning of therapeutically relevant biological states over task-specific outcomes ensures that our model’s learned features can be applied across multiple therapeutic contexts.

To bridge the gap between high-dimensional profiling and clinical care, we evaluated whether H&E-derived TME phenotypes could effectively stratify clinical outcomes for therapeutic contexts where current biomarkers are insufficient. Because comprehensive transcriptomic profiling is not scalable for routine care, our framework addresses an unmet need by deriving patient-specific TME biology that can inform clinical management. This need is especially pronounced in TNBC, which lacks targetable oncogenic drivers and therefore would benefit from TME-specific biomarkers to guide selection of therapy. Across multi-institutional cohorts of TNBC patients, model-derived phenotypic insights were linked to therapeutic efficacy in a manner consistent with established biological determinants of response^16–20. Specifically, immunogenic signatures were significantly associated with a pCR in the KEYNOTE-522 cohort. The more modest associations between immune phenotypes and ddACT response align with the multi-faceted and tumor-intrinsic mechanisms underlying chemotherapy response^21,22 (e.g., DNA repair deficiency, apoptotic activity). The model’s associations using core biopsy H&E WSIs confirms that systems-level biology can be quantified in tissue-scarce settings typical of clinical oncology. More broadly, these data support a paradigm shift towards developing models that quantify clinically relevant biology, rather than those trained solely to predict narrow, isolated outcomes.

Next, using only slide-level labels, our approach successfully localized tissue regions enriched for the specific biology driving systems-level TME phenotypes. The computational challenge of weakly-supervised learning, extracting task-relevant signals from thousands of heterogenous image patches, mirrors the biological complexity of deconvoluting therapeutic resistance within a multifaceted TME. Just as resistance mediators often occupy localized niches within tumor tissue, typically only a minority of image patches contribute to specific functional or therapeutic state. While bulk profiling assays average molecular signals across heterogeneous tissue and can obscure localized niches that drive immune activity, our model’s attention mechanism specifically prioritizes these high-yield regions. Despite relying on slide-level labels, our model resolves these localized signals to accurately estimate biological activity and isolate task-specific patches that drive the overarching clinical phenotype.

Our blinded reader study provided empirical validation for this capability: pathologist assessments were most accurate on high-attention patches (patches deemed by the model as most relevant). Conversely, uninformative regions of tumor tissue were correctly assigned low-attention scores. These findings confirm that our framework reliably prioritizes tissue morphology consistent with underlying functional states, effectively localizing relevant biology even when the corresponding histological features may be under-recognized by human experts. Collectively, these findings establish attention mechanisms as a principled framework for prioritizing tissue regions linked to biological programs underlying therapeutic resistance. Moving forward, these attention scores can be repurposed into a high-yield roadmap for therapeutic discovery. By directing spatial profiling to high-attention regions, our framework may accelerate mechanistic investigation of therapeutic phenomena that lack established a priori biological drivers.

The orthogonal validation of model-derived phenotypes with mIF offers additional confirmation that weak supervision of routine histopathology effectively recovers functional immune biology. While the correlations between inferred TME phenotypes and corresponding mIF-derived cell counts may appear moderate in isolation, perfect concordance is not expected given that these assays quantify similar, but not identical, biological measures. Importantly, the presence of measurable, phenotype-consistent agreement supports the biological validity of the inferred states, as functional immune programs should exhibit some alignment with marker-defined cell composition. For example, the association between B-cell proliferation and mIF-derived B-cell fractions (PCC: 0.331) is consistent with the model capturing B-cell proliferative state rather than B cell density alone. Similarly, discrepant cases with FoxP3 signal outside of Tregs (e.g., myeloid or tumor cells) suggest that our model is learning a broader FoxP3-associated immunosuppressive program that extends beyond abundance of Tregs. Taken together, the observed agreement across these disparate measurement scales (e.g., dynamic transcriptomic activity vs. static protein-level snapshots) strengthens our conclusion that weak supervision of H&E WSIs can aggregate clinically relevant histologic patterns into accurate, quantitative readouts of systems-level TME biology.

This capacity to infer functional states from histologic patterns aligns with emerging cross-modal foundation models that use contrastive learning to align the latent spaces of histopathology with ST or multiplexed imaging^3,4,23,24. While such models (e.g., GigaTIME²⁴, STORM²³) can derive high-resolution spatial readouts from routine H&E WSIs, they depend on resource-intensive paired training datasets, which inherently limit scalability. By contrast, our weakly supervised model demonstrates that spatially resolved TME biology can be inferred from H&E WSIs without paired spatial profiling data. Here, our model’s attention mechanism deconvolutes coarse, slide-level labels to identify the focal tissue regions that drive a patient’s tumor phenotype. Therefore, this strategy can be applied to massive real-world cohorts where direct spatial TME assessment is logistically prohibitive. Rather than replacing high-resolution multi-Omics, our approach serves as a complementary discovery engine by identifying “high attention” tissue regions for deeper mechanistic interrogation. As foundation models and ST datasets continue to mature^3,4,25,26, integrating our scalable H&E-based phenotyping strategy with spatially resolved assays may reveal new morpho-molecular signatures of efficacy, ultimately enabling precise immune profiling (e.g., distinguishing regulatory from cytotoxic lymphocyte populations) directly from routine histology.

Despite the potential of this approach, several limitations warrant consideration and motivate future work. First, our models were trained using AM-SB, an architecture that treats individual WSI patches as independent instances, which cannot capture long-range spatial dependencies across adjacent tissue regions²⁷. While transformer-based models were evaluated, they did not outperform our AM-SB framework, perhaps because vision transformers require substantially larger training datasets to learn these complex interactions^28,29. Next, we observed no performance gains from multimodal models integrating H&E WSIs with clinico-genomic data. This finding suggests that the included tabular features (e.g., demographics) were either irrelevant for TME phenotyping or already implicitly captured within histology (e.g., molecular data). Multimodal integration may instead be more impactful for predicting therapeutic outcomes, where resistance results from interactions between tumor biology, patient-specific factors, and the drug’s mechanism of action. Finally, as our analyses were restricted to breast cancer, future work is needed to evaluate our model’s ability to generalize to different tumor types. While some histological features pertinent to TME phenotypes (e.g., immune infiltration) may be conserved across tumor types, other manifestations may be histology dependent. Extending our analysis to pan-cancer cohorts will define both histology-specific and shared morpho-molecular signatures of therapeutic resistance.

In summary, our findings illustrate that DL-based analysis of routine histopathology can serve as a scalable tool to quantify the biological programs that drive therapeutic resistance across large populations. By demonstrating that systems-level TME phenotypes can be inferred from H&E slides via weakly supervised DL, we provide a pragmatic solution to the logistical bottlenecks that have historically restricted deep profiling to small, specialized cohorts. Furthermore, our framework can serve as a complement to refine high-resolution multi-Omics. By screening massive legacy biobanks and pinpointing high-attention regions where resource-intensive spatial profiling will yield the greatest mechanistic insight, this strategy functions as a discovery engine to identify new morpho-molecular mediators of resistance. Transforming histology into a quantitative, spatially resolved readout of TME biology positions the field to move beyond coarse diagnostic categories toward an improved systems-level understanding of therapeutic efficacy across the pan-cancer landscape.

METHODS

Datasets

The training dataset was comprised of H&E WSIs, patient-matched bulk RNA-sequencing data, and associated clinical data obtained from The Cancer Genome Atlas – Breast Cancer and cBioPortal for Cancer Genomics^30,31. In total, the dataset included 3111 WSIs from 1098 breast cancer patients. All WSIs were scanned at 40x magnification using Aperio scanners. All data were generated as part of routine clinical care and not within the context of a prospective clinical trial.

The ground truth was derived from the bulk RNA-sequencing data for each WSI in our training set. Normalized gene expression values for the TCGA cohort were obtained from cBioPortal. TME phenotypes were defined using pre-specified gene sets derived from gene ontology-based pathway analyses³². We selected ten TME phenotypes, spanning tumor-intrinsic, immune, and metabolic programs, each of which has been implicated in therapeutic resistance across diverse tumor types and treatment modalities^1,2,33. For each patient-TME phenotype pair, we computed gene set enrichment scores using single-sample gene set enrichment analysis (ssGSEA). ssGSEA converts an individual sample’s gene expression profile into a gene set-level enrichment profile, yielding a quantitative score that reflects the coordinated up-regulation (positive values) or down-regulation (negative values) of genes within a given biological pathway^34,35.

To evaluate model generalizability and clinical relevance, we curated two further cohorts of H&E slides from breast cancer patients treated at three institutions: Massachusetts General Hospital (MGH), Brigham & Women’s Hospital (BWH), and the Ohio State University (OSU). All slides were scanned at 40x magnification using routine clinical workflows on either Lecia Aperio GT450 DX (MGH, BWH) or Philips Intelisite Ultrafast scanners (OSU). The first cohort (n = 121 patients from MGH, BWH, OSU) consisted of H&E WSIs of pre-treatment core biopsies from triple-negative breast cancer patients who went on to receive the KEYNOTE-522 regimen³⁶. This regimen consists of neoadjuvant pembrolizumab, paclitaxel, and carboplatin, followed by definitive surgery. Clinical outcomes were binarized based on surgical pathology following neoadjuvant therapy. Patients were classified as achieving a pathologic complete response (pCR) if no residual tumor was identified in the breast or sampled lymph nodes, and as having residual disease if any residual invasive tumor was present at definitive surgery. A second cohort (n = 78 patients from MGH and BWH) consisted of pre-treatment H&E WSIs from patients with triple-negative breast cancer who subsequently received adjuvant anthracycline- and taxane-based chemotherapy³⁷. Clinical outcomes were binarized based on longitudinal follow-up: patients were classified as remission if they remained disease-free for at least seven years of follow-up, and as recurrence if they developed loco-regional or distant disease recurrence. For both cohorts, there was no associated tabular clinical or demographic data. All WSIs were scanned at 40x magnification using routine clinical workflows. Automated quality control checks for artifact burden and tissue sufficiency were applied, followed by pathology review to confirm tumor adequacy. These WSIs were processed with the same pre-processing pipelines used for model development.

WSI Processing

Segmentation, quality control, and patching

WSI pre-processing followed a protocol consistent with prior work in WSI DL-based classification^4,7,38–41. Each WSI underwent quality assessment using the HistoQC framework, an open-source tool for automated WSI quality control that evaluates image quality metrics, artifact detection, and tissue segmentation⁴². HistoQC was applied using the default parameters, yielding curated segmentation masks distinguishing tissue from background. Using this mask, each WSI was partitioned into non-overlapping 256 × 256-pixel patches at 40x magnification.

Label-preserving data augmentation strategies

To augment the training data, we implemented label-preserving strategies designed to improve model generalization by simulating common sources of variability in histopathology.

1. HED Color Augmentation: To address the color and intensity variability arising from differences in tissue preparation and staining within batches of WSIs from different sites, we applied stain-aware color perturbations at the tile level. Image patches were first converted from RGB to the Hematoxylin-Eosin-DAB (HED) color space, which separates the contributions of individual stains and enables targeted manipulation. The intensities of the hematoxylin-and-eosin channels were independently scaled by a factor sampled uniformly from 0.8 to 1.2, a range consistent with established approaches for simulating realistic staining variability without introducing visually implausible artifacts that could negatively impact model training⁴³. This process was used to pre-compute four augmented versions of each slide.

2. Tile Shifting Augmentation: To enhance spatial invariance and robustness to minor variations introduced during slide digitization, we implemented a tile-shifting strategy that alters the origin of the tiling grid applied to each WSI. Specifically, we generated three additional sets of tiles by shifting the grid by half a patch's length (128 pixels) horizontally, vertically, and along both diagonal directions. This strategy produces alternate patch configurations with overlapping but distinct spatial contexts, encouraging the model to learn more generalizable morphological features.

3. Geometric transforms. We applied standard geometric transformations at the tile level, including horizontal and vertical flips, to further increase morphological diversity.

During the training loop, we employed a two-stage probabilistic sampling scheme to introduce feature-level variability on-the-fly, balancing data diversity with the computational constraints of multiple-instance learning workflows that rely on pre-computed features. At the slide level, each WSI had a 20% probability of being selected for augmentation. For selected slides, a second sampling step was performed at the tile level, in which each tile had a 50% probability of having its feature vector replaced. For tiles selected for replacement, one of the corresponding pre-computed augmented feature vectors was chosen uniformly at random. This stochastic augmentation strategy ensured continuous exposure to diverse and iterative augmented features during training without requiring repeated feature extraction.

Feature extraction

We extracted feature vectors from each patch using three different feature extractors. First, we used CONtrastive learning from Captions for Histopathology (CONCH)⁴¹, a vision-language foundation model for histopathology pretrained on paired histopathology image and captions. Second, we evaluated Pathology Language–Image Pretraining (PLIP), a multimodal foundation model jointly trained on image-text pairs from the OpenPath dataset⁴⁴. Third, we used ResNet50⁴⁵, a convolutional neural network (CNN) pre-trained on ImageNet.

To identify the best feature representation for downstream analysis, we trained a binary classification model using each set of feature vectors and compared performance across the three feature extractors. Based on these comparisons, the best-performing feature extractor was used for all subsequent experiments.

H&E WSI model training

Our model was trained to quantify TME phenotypes using two approaches: binary classification and regression. This dual modeling strategy was motivated by clinical considerations: while binary classification enables coarse stratification of phenotypic activity (e.g., up- or down-regulation), continuous estimates of TME phenotype expression may better support precision-based decision making. We did this because in clinical practice, binary classification is too coarse a classification by which to make precision-based decisions; hence it is helpful to know the exact amount of over- or under-expression. For binary classification, ssGSEA scores were binarized such that negative values (pathway down-regulation) were assigned label 0 and positive values (pathway up-regulation) were assigned the label 1. Therefore, the output for binary classification models was either label 0 or 1. For regression, our model directly predicted normalized ssGSEA scores as continuous outcomes. A random data split, at the patient level, allocated 70% to training, 15% to validation, and 15% to test sets, ensuring that all data from a given patient were confined to the same split. Data within each partition was balanced for ER, PR, and HER2 receptor profiles.

For the models using only WSIs as the sole input modality, we employed the attention-based multiple-instance learning architecture (AM-SB) described in our prior work for phenotyping¹². AM-SB is a modification of the single-branch CLAM framework (CLAM-SB)⁴⁰. Unlike the original CLAM-SB architecture, AM-SB removes instance-level clustering under a mutual exclusivity assumption, which is inappropriate for TME phenotypes that may co-occur spatially within the same slide. In addition to AM-SB, we also evaluated a vision transformer (ViT)-based MIL architecture (TransMIL) to assess whether modeling long-range spatial dependencies between WSI patches improved performance for TME phenotyping⁴⁶.

Patch-level feature vectors were extracted using the CONCH encoder⁴¹, the best-performing feature extractor, were provided as input to the model. These features were first projected from their original dimensionality into a 512-dimensional latent space using a fully connected layer with a ReLU activation function. These transformed features were then processed by a gated attention mechanism that assigns an attention weight to each patch, reflecting its contribution to the slide-level representation. Attention-weighted aggregation across the slide yielded a single 512-dimensional slide-level feature vector, which was passed to a task-specific prediction head. For TransMIL models, patch embeddings were processed through transformer layers to capture contextual relationships across spatially separate instances, followed by slide level aggregation and a task-specific prediction head.

For binary classification, the prediction head consisted of a fully connected linear layer that performs a linear mapping from the 512-dimensional slide-level feature vector to a 2-dimensional logit space, followed by a softmax activation. Models were trained using a slide-level cross-entropy loss function. To mitigate class imbalance at the slide level, we employed a class-weighted sampling strategy.

For regression tasks, the classification head was replaced with a single linear neuron mapping the slide-level feature vector to a scalar output, with no final activation function applied. Because regression targets are continuous and defined at the slide level, instance-level clustering supervision was not applied. Regression models were trained using slide-level mean squared error loss.

All models were trained for 300 epochs using the adaptive moment (Adam) optimizer with a learning rate of 2 × 10⁻⁴ and a weight decay of 1 × 10⁻⁵. To prevent overfitting, we applied a dropout rate of 0.25, applied to the units in the network layers, and used early stopping with a patience of 20 epochs, monitored on the validation loss. Early stopping was not initiated before a minimum of 25 training epochs had been completed. Model selection was performed based on validation performance, and the best-performing checkpoint was used for evaluation on the heldout test set.

Training of classification models using clinico-genomic data

We trained binary classification models using clinico-genomic metadata routinely used in clinical trial stratification as model inputs. The clinical variables evaluated in this study included: age, sex, race, American Joint Committee on Cancer (AJCC) staging, histological subtype, PAM50 molecular subtype, estrogen receptor (ER) status, progesterone receptor (PR) status, and human epidermal growth factor 2 (HER2) status. Age was discretized into categorical bins: 18–30; 30–40; 40–50; 50–60; 60+. All variables (both originally categorical and discretized) were encoded using one-hot representations prior to model input.

To establish a baseline for the ability of standard-of-care clinical data to measure therapeutically relevant biology, we trained binary classification models, using only tabular clinico-genomic data. These experiments were conducted prior to multimodal model development to quantify the independent predictive contribution of these tabular clinical variables. We evaluated three supervised learning approaches for clinico-genomic-only modeling: random forest, support vector machine (using a radial basis function non-linear kernel), and tabPFN, a transformer-based foundation model for tabular data pretrained on synthetic classification tasks⁴⁷.

Multimodal model training strategies

As the optimal strategy for combining H&E WSI with clinico-genomic data remains an active area of investigation, we evaluated two fusion approaches: cross-attention based intermediate fusion and late fusion of modality-specific predictions.

Our first approach was an intermediate fusion strategy adapted from the SurvPath architecture¹³. In this framework, we modified the original transcriptomics-based biological pathway tokenization to accommodate tabular clinico-genomic features, while retaining the cross-attention mechanism to integrate clinical and whole-slide image (WSI) representations. The cross-attention layer enables the model to learn inter-modality interactions that may be relevant for the downstream task. To address the dimensionality mismatch between CONCH-derived WSI features (512 dimensions) and the lower-dimensional clinical feature vectors (typically 1–10 dimensions after encoding), we applied a learnable multilayer perceptron (MLP) to project clinical features into a higher-dimensional embedding space (expanded by a factor of 100), ensuring compatibility with the WSI feature representations prior to fusion. Models were trained for 300 epochs using mean-squared error loss and the Adam optimizer, with a learning rate of 2 × 10⁻⁴ and weight decay of 1 × 10⁻⁵. The model checkpoint achieving the lowest validation loss was selected for final evaluation.

Our second approach was a late fusion strategy³⁹, in which modality-specific models were trained independently and their outputs combined at the prediction level. Predictions from the WSI-only and clinico-genomic-only models were aggregated using an MLP consisting of an input layer with two neurons; two hidden layers with four and two neurons, respectively; and a single-neuron output layer. This fusion network was trained for 300 epochs using mean-squared error loss and the Adam optimizer, with identical learning rate and weight decay settings as above. Model selection was again based on validation loss.

Evaluation of model performance

For binary classification on H&E WSI-only and multimodal models, performance was evaluated using the area under the receiver operating characteristic curve (AUROC) metric. For regression models predicting continuous ssGSEA-derived phenotypes, we report Pearson correlation coefficient (PCC) between predicted and ground-truth scores. The clinico-genomic-only models were evaluated using AUROC. To quantify uncertainty, we performed bootstrap resampling with 1000 iterations. For each bootstrap run, performance metrics were recomputed on the resampled test sets, and a median and 95% confidence interval was reported across bootstrap distributions.

Ablation studies

To quantify the impact of different categories of clinico-genomic information within the clinico-genomic and multimodal model settings, we conducted ablation studies by training models using predefined subsets of clinical features. Three clinical feature sets were considered: (i) All (age, gender, race, AJCC stage, histologic subtype, PAM50 subtype, ER status, PR status, and HER2 status); Demographics (age, gender, race); and Molecular (PAM50 subtype, ER status, PR status, and HER2 status).

Computational environment

Model training was conducted on a SLURM-managed high-performance computing (HPC) cluster, using single NVIDIA RTX 8000 GPUs with 4 CPU cores and 200 GB of RAM. The data loading pipeline utilized 4 parallel workers to ensure efficient data throughput. For neural network training and optimization, we used the PyTorch framework and for WSI manipulation, we used OpenSlide. The code related to the implementation of this paper is publicly available in a GitHub repository.

Visualization of high- and low-attention regions on H&E WSIs

Attention weights were produced by the attention-based MIL framework during slide-level inference⁴⁰. For each WSI and TME phenotype, attention scores were assigned to individual image patches and normalized within each WSI. These scores reflect the relative contribution of each patch to the model’s final prediction.

Next, to interpret model predictions and assess spatial correlates of TME phenotypes within WSIs, attention maps were generated. These maps were visualized by projecting patch-level attention scores back onto their spatial coordinates within the original WSIs. Heatmaps were generated using a continuous color scale, with regions in red corresponding to high attention weights and regions in blue corresponding to low attention weights. As distinct TME phenotypes correspond to different morphologic patterns or cell populations, attention maps were generated separately for each phenotype to enable phenotype-specific interpretation. High-attention regions were qualitatively reviewed by a board-certified pathologist to identify recurring spatial or morphologic patterns associated with specific phenotypes.

Model validation on therapeutic contexts with multi-institutional external data

To evaluate the generalizability and clinical relevance of our model, we curated multi-institutional cohorts of pre-treatment H&E WSIs from breast cancer patients treated at Massachusetts General Hospital (MGH), Brigham & Women’s Hospital (BWH), and the Ohio State University. None of these WSIs overlapped with the training or internal validation datasets. For our KEYNOTE-522 cohort, inference was performed on pre-treatment H&E core biopsy WSIs, using the H&E WSI-only regression model. These model-derived phenotypes were compared between pCR and residual disease (thus implying ICI resistance) outcome groups. For the ddACT cohort, inference was performed using the H&E WSI-only regression model. These model-derived phenotypes were compared between remission and relapse outcome groups. For both clinical use cases, differences in model-derived phenotype scores across the outcome groups were assessed using two-sided Mann Whitney U tests. Clinical outcome information was not used during model training or inference.

Reader study to assess biological fidelity of attention maps

To evaluate whether attention-based learning reliably captured task-specific biology, we conducted a blinded reader study by two board-certified pathologists. Fifty H&E WSIs were randomly selected from the held-aside test set. Using our H&E WSI classification model, inference was performed for three TME phenotypes selected a priori based on their known association with histological patterns identifiable on routine H&E review. These phenotypes included: B-cell proliferation, epithelial-mesenchymal transition, and angiogenesis. For each TME phenotype, attention scores were extracted for all image patches within each WSI. From each slide and task, three patch groups were algorithmically selected: the five patches with the highest attention scores, the five patches with the lowest attention scores, and five randomly sampled patches. Patch selection was fully automated and performed without human input.

These patch sets were presented to both blinded pathologists in randomized order, blinded to attention scores and model predictions. For each set of five patches, the pathologist was asked to determine whether the observed histologic features were most consistent with high expression, low expression, or not assessable for the queried phenotype (e.g., insufficient tissue, adipose tissue, background artifact). Because high-attention patches are expected to most strongly reflect task-specific biology, we hypothesized that pathologist assessment of TME phenotypes would be most accurate using the high-attention patches, compared to randomly sampled or low attention patches. Conversely, as low attention patches are expected to have minimal biological relevance to the classification task, we anticipated a higher frequency of “not assessable” determinations within this group.

Multiplexed immunofluorescence analysis

Staining and Imaging:

Highly multiplexed spatial proteomics was performed on formalin-fixed, paraffin-embedded (FFPE) Triple-Negative Breast Cancer (TNBC) tissue microarrays using the CODEX/PhenoCycler platform⁴⁸. Tissues were stained with a custom 19-marker panel of oligonucleotide-barcoded antibodies from Akoya Biosciences. These anti-human antibodies targeted CD3e, CD4, CD8, CD11b, CD11c, CD14, CD20, CD21, CD38, CD45, CD56, CD68, CD79a, CD206, FoxP3, Granzyme B (GZMB), HLA-DR, iNOS, and Pan-Cytokeratin (PanCK). Automated cyclic imaging was performed to iteratively hybridize, image, and strip fluorescently labeled reporter oligonucleotides corresponding to the antibody barcodes.

Image Processing and Single-Cell Segmentation:

Computational processing of the raw multi-channel QPTIFF images was executed using the SPACEc (Structured Spatial Analysis for Codex Exploration) Python framework⁴⁹. Within the SPACEc pipeline, deep learning-based whole-cell segmentation was performed using the integrated Mesmer algorithm, utilizing the DAPI nuclear counterstain and membrane markers to define cell boundaries. Single-cell mean fluorescence intensities for all 19 markers were quantified and extracted into a single-cell expression matrix.

Data Preprocessing and Batch Correction:

Downstream preprocessing and clustering were also managed within SPACEc and the Scanpy ecosystem. Single-cell marker intensities were Z-score normalized per slide. To correct for batch effects across the four distinct tissue microarray slides, Harmony integration was applied to the principal components⁵⁰.

Unsupervised Clustering and Cell-Type Annotation:

To prevent the artificial fragmentation of functional cell states, clustering was strictly limited to a 19-marker structural lineage sub-panel. Cells were clustered using the Leiden algorithm via a k-nearest neighbor (k-NN) graph built on the Harmony-corrected embeddings. Clusters were manually annotated into specific immune and tumor cell lineages based on canonical marker expression profiles. Finally, absolute cell counts and relative cellular fractions for these specific populations were extracted per tissue region to serve as the ground-truth cellular metrics for downstream correlation analysis.

STATEMENT OF SIGNIFICANCE.

Multi-Omics is too resource-intensive for everyday clinical use. Using slide-level labels, weakly supervised deep learning infers quantitative and spatially resolved TME phenotypes directly from H&E slides. By highlighting high-attention regions of tumor tissue that drive therapeutic efficacy, this strategy can serve as a discovery engine to identify morpho-molecular mediators of resistance.

Data Sharing Statement:

Our model was trained using publicly available data from The Cancer Genome Atlas and the cBioPortal for Cancer Genomics. The raw clinical, H&E WSI, and multiplexed immunofluorescence data in our external validation set are protected due to patient privacy laws. Any requests for raw and analyzed data should be sent in writing to Albert Kim (akim46@mgh.harvard.edu) and will be reviewed by the DF/HCC Institutional Review Board (IRB). Pending IRB approval, deidentified data then will be transferred to the inquiring investigator in an expeditious fashion over secure file transfer.

Code Availability:

Our repository for data analysis is located at: https://github.com/QTIM-Lab/tme-wsi.