Deep learning informed multimodal fusion of radiology and pathology to predict outcomes in HPV-associated oropharyngeal squamous cell carcinoma

Summary

Background

We aim to predict outcomes of human papillomavirus (HPV)-associated oropharyngeal squamous cell carcinoma (OPSCC), a subtype of head and neck cancer characterized with improved clinical outcome and better response to therapy. Pathology and radiology focused AI-based prognostic models have been independently developed for OPSCC, but their integration incorporating both primary tumour (PT) and metastatic cervical lymph node (LN) remains unexamined.

Methods

We investigate the prognostic value of an AI approach termed the swintransformer-based multimodal and multi-region data fusion framework (SMuRF). SMuRF integrates features from CT corresponding to the PT and LN, as well as whole slide pathology images from the PT as a predictor of survival and tumour grade in HPV-associated OPSCC. SMuRF employs cross-modality and cross-region window based multi-head self-attention mechanisms to capture interactions between features across tumour habitats and image scales.

Findings

Developed and tested on a cohort of 277 patients with OPSCC with matched radiology and pathology images, SMuRF demonstrated strong performance (C-index = 0.81 for DFS prediction and AUC = 0.75 for tumour grade classification) and emerged as an independent prognostic biomarker for DFS (hazard ratio [HR] = 17, 95% confidence interval [CI], 4.9–58, p < 0.0001) and tumour grade (odds ratio [OR] = 3.7, 95% CI, 1.4–10.5, p = 0.01) controlling for other clinical variables (i.e., T-, N-stage, age, smoking, sex and treatment modalities). Importantly, SMuRF outperformed unimodal models derived from radiology or pathology alone.

Interpretation

Our findings underscore the potential of multimodal deep learning in accurately stratifying OPSCC risk, informing tailored treatment strategies and potentially refining existing treatment algorithms.

Funding

The National Institutes of Health, the U.S. Department of Veterans Affairs and National Institute of Biomedical Imaging and Bioengineering.

Research in context.

Evidence before this study

We searched PubMed for publications without date or language restrictions, with the terms (“deep learning” OR “machine learning” OR “artificial intelligence” OR “AI”) AND (“head and neck cancer” OR “oropharyngeal cancer” OR “OPSCC” OR “HPV”) AND (“multimodal learning” OR “data integration” OR “data fusion”) AND (“medical images” OR “radiology”) AND (“pathology” OR “whole slide images” OR “biopsy”). There has been no studies to combine medical images and pathology whole slide images to develop multimodal biomarkers for oropharyngeal cancer. Although radiology CT and pathology whole slide images are routinely employed for disease diagnosis and prognosis, there has been no dedicated effort to investigate the integration of multiscale imaging data for biomarker discovery in the context of oropharyngeal cancer.

Added value of this study

In this study, we introduce a Swin Transformer-based multimodal framework (SMuRF) designed to integrate radiological CT and pathological whole slide images (WSI) for predicting survival and tumour grade in HPV-related OPSCC. SMuRF captures and preserves the spatial context across both imaging modalities (CT and whole slide images) and between distinct tumour habitat regions (primary tumour and metastatic cervical lymph node). By leveraging state-of-the-art transformer-based deep learning architectures, the model learns multiscale and multi-region features simultaneously, which are then fused using cross-modality and cross-region attention mechanisms to enhance prognostic accuracy.

Implications of all the available evidence

By integrating features of primary tumour and metastatic cervical lymph node from radiology CT and pathology WSI, the multimodal deep learning model (SMuRF) significantly improved survival and grade prediction performances compared to conventional unimodal models (i.e., using CT or WSI alone). SMuRF was also independently associated with disease-free survival and tumour grade after controlling for clinical variables. Additionally, the innovative multiscale and multi-region imaging fusion approach introduced in this study offers a versatile framework that can be extended to tackle broader challenges in multimodal learning and data integration for cancer research.

Introduction

The incidence of human papillomavirus (HPV)-associated oropharyngeal squamous cell carcinomas (OPSCC) has risen significantly over the past several decades, especially in Western countries.¹ The most recent (8th) edition of the AJCC staging system separates HPV-positive (HPV+) OPSCC from its HPV-negative (HPV−) counterpart to account for its improved prognosis.² Despite their improved prognosis, many patients undergoing treatment for HPV-associated OPSCC are younger on average and face long-term survival challenges alongside a diminished quality of life.³ These challenges primarily stem from the adverse treatment effects (surgery, radiotherapy, and concurrent chemotherapy) and recurrent disease. Thus, it is critical to identify approaches that will enable risk-stratification of patients into distinct groups for precision treatment strategies that can help improve their clinical outcome. For example, identifying those patients with HPV-positive OPSCC⁴ who have significantly lower disease risk and hence may benefit from treatment de-escalation. Conversely, the ability to identify HPV-positive patients¹ with more aggressive disease could make them candidates for more aggressive interventions up front.

Several prognostic molecular biomarkers have already been identified in the context of HPV-associated OPSCC. For example, mutations in PIK3CA have been correlated with less favourable disease control in trials aimed at reducing OPSCC treatment intensity.⁵ Conversely, somatic mutation in TRAF3 (tumour necrosis factor receptor-associated factor 3) and CYLD (cylindrotomids lysine 63) have been linked to a better prognosis.⁶ However, relying solely on molecular biomarkers may be limiting in that the approach might result in ignoring of other prognostic factors, such as morphologic heterogeneity associated with the tumour microenvironment.^7,8

Leveraging information from both pathology and in vivo radiology offers an opportunity to acquire tumour-specific insights across microscopic to macroscopic scales in addition to the molecular scale.9, 10, 11 The advent of digital pathology has paved the way for integrating artificial intelligence (AI) with histological imaging features12, 13, 14, 15, 16, 17, 18 for characterizing and prognosticating disease outcome. Microscopic level cellular features, which can be captured through pathology imaging, provide detailed insights into the tumour microenvironment, including cellular morphology and the interaction between different cell types.¹⁹ While whole slide images (WSI) allow for in-depth quantification of tissue-specific attributes at a cellular level, they only inspect a sliver of the entire tumour. In contrast, application of AI and radiomic approaches to radiology images20, 21, 22, 23, 24, 25, 26, 27 enables the identification of macro-level features of the tumour and its spatial relationship with respect to lymph nodes. Thus, radiology and pathology imaging provide complementary information, helping to unearth attributes across different length scales, in turn potentially enabling enhanced tumour characterization and prognosis prediction.

In the context of oropharyngeal carcinoma, the nodal status is a crucial determinant of patient outcomes.^28,29 Despite this, prior research has predominantly focused on the prognostic significance of the primary tumour within a single imaging modality.^15,20, 21, 22 In contrast, the combined prognostic implications of both the primary tumour and lymph node involvement across multiscale imaging have received comparatively less attention. In this work we present a swin transformer-based multimodal framework (SMuRF) for integrating radiological CT and pathological WSI to predict survival and tumour grade outcomes in patients with HPV-associated OPSCC. SMuRF can explicitly retain and exploit spatial context both between image modalities (i.e., CT and WSI) as well as between different regions that comprise the tumour habitat (i.e., primary tumour and metastatic cervical lymph node). Multiscale and multi-region features are learnt using state-of-the-art transformer-based deep learning architectures and subsequently fused using cross-modality and cross-region attention.³⁰ On a multi-institutional cohort of N = 277 patients, we conducted intensive experiments to validate the incremental prognostic value of SMuRF compared with the other unimodal or multimodal approaches. In addition, we also validated the independent prognostic value of SMuRF, controlling for existing clinical factors.

Methods

Ethics

After obtaining approval of Institutional Review Board (IRB) from corresponding institutions, matched CT scans, pathology WSI images, clinical and survival outcome information for 277 HPV-associated OPSCC in the Cleveland Clinic Foundation (CCF, N1 = 230) and Winship Cancer Institute (Winship, N2 = 47) cohorts were retrospectively obtained by chart review (CCF IRB ID: CCF 14–551; Winship IRB ID: STUDY00005261).

Dataset description

The ICD-10 code used to retrieve the cases included C01.9, C02.4, C05.1, C05.2, C09.0, C09.1, C09.8, C09.9, C10.0, C10.2, C10.3, C10.8, C10.9, and C11.1. Patients were identified retrospectively within the period 2002–2015 for the study. Sex information was self-reported by participants during enrolment and recorded as part of the demographic information. The contrast-enhanced CT scans were acquired as part of baseline imaging prior to the initiation of treatment. The CT imaging parameters for the CCF and Winship cohorts are provided in Supplementary Table S5. Patients were randomly allocated to development (120), internal validation (52) and internal test sets (105). HPV status for both cohorts were determined using p16 immunohistochemistry (IHC). A patient selection and allocation diagram is provided in Supplementary Fig. S5. Inclusion criteria were: (i) non-metastatic (M0) HPV-associated (strong and diffuse, block-like nuclear and cytoplasmic p16 staining present in ≥70% of the tumour cells. OPSCC, (ii) treatment with curative intent, (iii) patient did not undergo surgery or induction chemotherapy, (iv) matched clinical information (e.g., age, stage and survival). Patients with any of the following were excluded: (i) without either available baseline pretreatment CT images covering the head and neck region or digitized H&E-stained WSIs, (ii) lack of identifiable tumours from CT scans or WSIs. Disease-free survival (DFS) is the primary endpoint for this study, which is defined as time from radiotherapy end date to the date of following events: local, regional, distant failure or death whichever occurred earlier and was censored at the date of last follow-up for those without events. tumour grade was determined by pathologists using microscopic biopsy evaluation of the tumour tissue. Overall survival (OS) was chosen as the secondary endpoint, which is defined as time from date of diagnosis to death and was censored at the date of last follow-up for those alive. WSIs were scanned at × 40 magnification for both cohorts and were reviewed by pathologists for selecting the most representative tumour slide. Primary tumour regions on all the fragments from the WSI were annotated by collaborating pathologists for each patient using the QuPath software.³¹ The tumour annotations were then exported to binary masks in PNG format. Similarly, the boundaries of primary tumour and the largest metastatic cervical lymph nodes on CT images were manually delineated by three board-certified head and neck radiologists across all two-dimensional CT axial slices using the 3D Slicer software.³²

Pathology WSI preprocessing and hierarchical embedding

We first excluded the background and segmented the fragment contour using the CLAM³³ (clustering-constrained attention-based multiple-instance learning) toolbox. CLAM performs tissue fragment segmentation at a low resolution via binary thresholding. In addition, it performs median blurring, morphological closing and filtering of contours below a minimum area to smooth tissue contours.

Next, we reduced the memory footprint of WSI by aggregating each 2048 × 2048 patch into a hierarchical representation. We applied the HIPT¹⁶ (Hierarchical Image Pyramid Transformer) method, which was trained across 33 cancer types, to build self-supervised region-level embeddings of size 192. HIPT leverages ViTs to hierarchically aggregate representations aware of both micro- and macro-scale interactions between cells. From each tissue fragment of varying size, we first divided it into image regions of size 2048 × 2048 corresponding to 1 square mm, which largely corresponded to length scale of radiological images, making it amenable for radiology-pathology fusion. For each such region, we further divided it into 8 × 8 = 64 patches, each of size 256 × 256. Thus, each patch represents around 1/(8 × 8) square mm. Eventually, we divided each patch into 16 × 16 = 256 small images, each of size 16 × 16. Thus, each small image represents around 1/(8 × 8 × 16 × 16) square mm ∼ 61 square μm, comparable to cell-level images. From these cell-level images, we applied the first stage of HIPT, leading to a patch-level representation for each 256 × 256 patch, which encoded visual concepts regarding single/few cells. From these patch-level representations, we then applied the second stage of HIPT, leading to region-level representation for each 2048 × 2048 region. Eventually, we applied the last stage of HIPT, leading to 192-dimensional fragment level for each fragment in the WSI, which encoded visual concepts regarding interactions between macro-scale regions. The 192-dimensional embeddings capture multiscale hierarchical information from the micro cell level to the macro tissue level. This process is illustrated in Supplementary Fig. S6. Because the size of each fragment varies within a WSI and across patients, we fixed the size of all images in a batch to match its biggest fragment with dynamical padding for memory optimization. Finally, those 192-dimensional region-level feature vectors are utilized as input for the pathology SwinT blocks,³⁴ where multi-scale WSI embeddings are learnt.

Radiology CT preprocessing

All CT volumes were resampled to 1 mm isotropic resolution and intensity normalized. We extended both the 3D tumour and cervical lymph node volumes by 15 mm in all directions to encompass the peritumoral area. Similar to previous work,³⁵ we randomly cropped 4 sub-volumes of size 48 × 48 × 3 from four even quadrants of both tumour and cervical lymph node along the z-axis. The position was fixed to the middle of each quadrant during inference. More details on tumour and lymph node sub-volume extraction are provided in Supplementary Method SI. For data augmentation, we applied random vertical and horizontal flips. The 4 cropped CT sub-volumes containing primary tumour and another 4 sub-volumes containing cervical lymph nodes served as inputs to two separate radiology SwinT blocks.

Multi-region and multiscale data fusion with swin-transformer and attention

We used the swin-transformer blocks³⁴ (SwinT) to model the multi-region and multi-scale image embeddings. This hierarchical structure offers adaptability in modelling vision tasks across different scales and maintains linear computational complexity relative to image dimensions.^36,37 We will discuss in detail the model architecture from two different aspects: extracting hierarchical embedding with Swin Transformer³⁴ and fusing multi-region and multi-scale image embeddings through attention-based methods.

Hierarchical embedding with swin transformer

The SwinT blocks for obtaining hierarchical embeddings for radiology and pathology are slightly different: the SwinT blocks are 2-dimensional for pathology while they are 3-dimensional for the two radiology input branches (corresponding to the primary tumour and lymph node respectively). The reason for this architecture is that the inputs to the pathology SwinT are the outputs from the HIPT model, which consists of a sequence of feature vectors representing the 2-dimensional WSI fragments. On the other hand, the inputs to the radiology SwinT are raw image sub-volumes with 3-dimensional spatial context information. Independent of its dimension, each SwinT includes three successive modules: 1) window based multi-head self-attention³⁴ (W-MSA), 2) shifted-window (SW) strategy³⁴ and patch merging procedure. W-MSA computes the importance of each image patch with respect to every other patch within a defined window region simultaneously. This approach enables capture of relevant spatial information within a local region. The SW mechanism solves the problem of quantifying global interactions across windows by displacing the window focus by M/2 pixels, where M indicates number of patches along the X or Y directions. This allows the model to aggregate information from a large and flexible receptive field, enabling the model to capture long-range spatial dependencies with global image-based context. The patch merging procedure progressively reduces the spatial dimensions of the feature maps while increasing the number of channels, which is a crucial step to reduce the computational complexity of the model. The local and global image contexts learning mentioned above is applied to both radiology and pathology imaging data, which in turn contribute to multi-region imaging data retrieval within individual ROIs (i.e., CT sub-volumes and WSI fragments). More mathematical details are provided in Supplementary Method SII.

Attention-based multi-region and multi-scale data fusion

During each 3-dimensional radiological SwinT block, a patch-merging layer was incorporated to group, neighbouring image patches along the depth dimension. After repeating this procedure 2 times, the SwinT block gradually decreases the resolution of the input which in turn enables modelling of the intra-modality multi-scale image embeddings. To capture the pathological micro-scale cellular interactions and radiological macro-scale anatomic information across modalities, we applied the self-attention pooling.³⁰ Finally, we used a multitask learning network supervised by the Cox partial negative log-likelihood loss and binary cross entropy loss to output a continuous risk score for survival and tumour grade prediction. We also compared our multi-region and multi-scale fusion scheme with three other fusion approaches, which included Kronecker product,³⁸ vector concatenation and Kronecker product with self-attention pooling, using exactly the same multi-region and multiscale data. More details are provided in Supplementary Method SIII.

To aid in model interpretability, we computed the Integrated Gradients (IG)³⁹ of model predictions and mapped them back to both original CT scans and WSI to highlight the respective prognostically relevant areas contributing to the final predictions. Attention heatmaps are overlaid on pathology WSI patches to highlight morphological attributes (i.e., tumour cells, collagen fibre) at varying scales. The calculation of attention heatmaps for radiology CT scans closely resembles the IG calculation and is therefore omitted. The overall workflow covering all methodological aspects of SMuRF is illustrated in Fig. 1.

Fig. 1.

Flowchart of this study: a) multimodal data curation and annotation; b) preprocessing on pathology WSI, fragment contours (green) are generated using the CLAM toolbox,³³ tumour annotations (red) are provided by pathologists. On radiology CT, primary tumour (yellow) and metastatic cervical lymph node annotations (blue) are provided by radiologists. c) multi-region and multiscale fusion with SwinT; d) model inference: survival and grade predictions. Red regions on CT and WSI indicate the prognostic relevant regions that the model is focusing on. W-MSA: window-based multi-head self-attention; SW-MSA: shifted window multi-head self-attention; SwinT: swin-transformer; HIPT: Hierarchical Image Pyramid Transformer.

Reproducibility/implementation details

Our experiments were built on Pytorch1.13⁴⁰ with an Nvidia GeForce RTX 3080Ti GPU. Unimodal models were first trained for 100 epochs with a constant learning rate = 0.002, a batch size of 16, the AdaGrad optimizer,⁴¹ and a dropout probability p = 0.3. We implemented early stopping so that the model checkpoint was saved on the epoch during which validation loss reached the minimum. The multimodal SMuRF was then trained using the same hyperparameters as the unimodal models. The SwinT blocks were never frozen during training. All hyperparameters were optimized based on the validation set and the external test set was kept blinded until the final model was locked down. The code implementations are available on GitHub: https://github.com/BolinSong1995/SMuRF_MultiModal_OPSCC/tree/main.

Statistical analysis on survival prediction

The difference of the continuous variables (i.e., age, smoking pack-year [PY] and SMuRF score) among 3 datasets (training, validation and test sets) were compared using the one-way analysis of variance test (ANOVA)⁴² and the association of the categorical variables were estimated using the Fisher’s exact test.⁴³ SMuRF was trained solely on the training set in an end-to-end fashion: after the multiscale and multi-region fusion by the SwinT, the fused embeddings were fed into multitask learning layers to output continuous risk scores for survival and tumour grade predictions. The cutoff value for dividing patients into low- and high-SMuRF groups for survival predictions was chosen using the median SMuRF value from the training set. The potential association between the SMuRF risk score and survival was first evaluated in the training set and then validated in the validation and the test sets based on Kaplan–Meier (KM) survival analysis⁴⁴ and Harrell’s concordance index (C-index).⁴⁵ Survival differences between SMuRF groups are evaluated using log-rank test.⁴⁶ To test the independent prognostic value of SMuRF, we performed multivariable Cox (survival prediction) and logistic regression (tumour grade classification) analysis controlling for clinical factors (T-, N-Stage, smoking pack/year, age, sex and treatment received). To visualize individual variable importance in the multivariable analysis, we computed the Shapley Additive exPlanations⁴⁷ (SHAP) values across all patients for all the variables considered in the multivariable analysis and visualized them with the bee swarm plot. To prove the additive prognostic value of the multimodal fusion, we also performed a comprehensive comparison of SMuRF against six other models trained using either only radiology or pathology images or in a multimodal setting with single region. More details on these models and the comparative evaluation are provided in Supplementary Method SIV. Statistical analyses were performed using R version 3.4.0.⁴⁸ The R packages used in this study included glmnet,⁴⁹ survival,⁵⁰ survcomp⁵¹ and SvyNom.⁵² A threshold of 0.05 was used to define statistical significance. p values between C-indices from different models were calculated using the survcomp package in R.

Role of the funders

The funder of the study provided the financial support, but had no role in study design, data collection, data analysis, data interpretation, or writing of the report.

Results

Clinical characteristics

Table 1 lists the detailed demographic and outcome characteristics of the patients in training (n = 120), validation (n = 52) and test sets (n = 105). Of the total 277 patients included in the study, 249 (89.9%) were men, and the median (interquartile range) age was 59 (53.6–64.4) years. 35 patients (12.6%) had AJCC 8th edition stage I OPSCC, 169 patients (61%) had AJCC 8th edition stage II OPSCC. Twenty-two (7.9%) patients went through definitive radiation and the remaining had concurrent chemoradiation. The median values for DFS on three sets were 47.64, 43.63 and 51.78 months. There were 50, 16 and 42 high-grade patients in the three sets, respectively.

Table 1.

Demographic and outcome characteristics of the patients.

Parameter	Training set (n = 120)	Validation set (n = 52)	Test set (n = 105)	p value
Age	59.88 ± 9.76	60.23 ± 8.73	60.05 ± 9.77	0.91
Sex
Male	108 (90%)	47 (90.4%)	94 (89.5%)	0.97
Female	12 (10%)	5 (9.6%)	11 (10.5%)
Smoking PY	19.02 ± 21.94	15.25 ± 21.58	12.69 ± 17.74	0.43
T-stage
T1	33 (27.5%)	14 (26.9%)	24 (22.9%)	0.9
T2	44 (36.7%)	20 (38.5%)	44 (41.9%)
T3	19 (15.8%)	8 (15.4%)	19 (18.1%)
T4	24 (20.0%)	10 (19.2%)	18 (17.1%)
N-stage
N0	3 (2.5%)	1 (1.9%)	12 (11.4%)	0.001
N1	26 (21.7%)	7 (13.5%)	8 (7.6%)
N2	76 (63.3%)	41 (78.8%)	80 (76.2%)
N3	15 (12.5%)	3 (5.8%)	5 (4.8%)
AJCC 8th stage
I	17 (14.2%)	6 (11.5%)	12 (11.4%)	0.37
II	65 (54.2%)	33 (63.5%)	71 (67.6%)
III	38 (31.6%)	13 (25%)	22 (21%)
Treatment
Radiation	8 (6.7%)	2 (3.8%)	12 (11.4%)	0.28
Chemoradiation	112 (93.3%)	50 (96.2%)	93 (88.6%)
DFS
Event	26 (21.7%)	15 (28.8%)	24 (22.9%)	0.42
Censored	94 (78.3%)	37 (71.2%)	81 (77.1%)
OS
Event	23 (19.2%)	4 (7.7%)	16 (15.2%)	0.26
Censored	97 (80.8%)	48 (92.3%)	89 (84.8%)
Median follow-up (months) [IQR]	56.6 [36.2]	53.7 [41]	55.9 [35.4]	0.36
Tumour grade
Low	70 (58.3%)	36 (69.2%)	63 (60%)	0.07
High	50 (41.7%)	16 (30.8%)	42 (40%)

DFS (disease-free survival), OS (overall survival), PY (pack-year), AJCC (American Joint Committee on Cancer), IQR (interquartile range).

SMuRF for DFS and tumour grade predictions

The median SMuRF value from training set was 0.5 and was used for dividing patients into high- and low-SMuRF groups for survival risk stratifications. Clinical characteristics within the high- and low-SMuRF groups are summarized in Supplementary Table S1. In training set, there were significant differences in DFS (Fig. 2a) and OS (Supplementary Fig. S2A) comparing patients with high-vs. low-SMuRF. The prognostic performance of SMuRF was confirmed on the validation set (Fig. 2b) and the test set (Fig. 2c, Supplementary Table S2) for DFS and for OS (Validation set, Supplementary Fig. S2B; test set, Supplementary Fig. S2C). On the validation set, the 3-year DFS rate was 53% for the high-SMuRF patients, and 100% for the low-SMuRF patients. On the test set, the 3-year DFS rate was 54% for the high-SMuRF patients, and 92% for the low-SMuRF patients. SMuRF also demonstrated the ability to distinguish tumour grade (low vs. high) across training (AUC = 0.99, Fig. 2d), validation (AUC = 0.84, Fig. 2e) and test sets (AUC = 0.74, Fig. 2f).

Fig. 2.

The Kaplan–Meier survival analysis of SMuRF for OPSCC DFS stratification from training set (a), validation set (b) and test set (c) and for grade classification from training set (d), validation set (e) and test set (f). Model comparisons of 7 comparable models on the test set (g) accounting for input modalities and cancer habitat regions are considered. Model comparisons using 4 different fusion schemes is performed on the test set (h).

Fig. 2g summarizes the quantitative metrics of C-index (for DFS prediction) and AUC (for tumour grade classification) on the test set from 7 ablative models accounting for input modalities (i.e., CT and WSI), regions (i.e., primary tumour and metastatic cervical lymph node). SMuRF achieved the highest C-index for DFS prediction (0.79) and the highest AUC for tumour grade classification (0.74). The confidence intervals for C-indices generated by 1000 bootstrap resampling are provided in Supplementary Table S2. Performance comparisons on the test set using different fusion scheme (vector concatenation, Kronecker product, self-attention pooling) with the same multimodal and multi-region data are provided in Fig. 2h. SMuRF embedded with self-attention pooling outperformed the other three fusion approaches in terms of C-index and AUC across the two endpoints. Adding Kronecker product with self-attention pooling did not improve the performances.

Univariate Cox and logistic regression analysis are provided in Supplementary Tables S3 and S4. In multivariable analysis on the test set accounting for clinicopathologic factors (T-, N-Stage, smoking pack/year, age, sex and treatment received), the continuous risk score SMuRF was still statistically significant in predicting both DFS (Fig. 3a, HR = 17, 95% CI, 4.9–58, p < 0.0001) and tumour grade (Fig. 3a, odds ratio [OR] = 3.7, 95% CI, 1.4–10.5, p = 0.01). Variables relative importance in the multivariable Cox regression using DFS as endpoint (Fig. 3b) and multivariable logistic regression using tumour grade as endpoint (Fig. 3d) are visualized in the bee swarm plot with the SHapley Additive exPlanations⁴⁷ (SHAP) values. All variables are shown in the order of global feature importance, the first one being the most important and the last being the least important one. SMuRF score was identified to be the most important predictor across the two endpoints in multivariable Cox/logistic regression analysis. We also calculated mean SHAP values across all patients for all the variables to reflect their relative contributions in the multivariable analysis. SMuRF contributed 37.3% (Fig. 3c) in the DFS prediction task and 58.1% (Fig. 3e) in the tumour grade classification, ranking as the most important variable across the two endpoints. To compare the performance of SMuRF with clinical factors in predicting DFS and tumour grade, we constructed separate multivariable Cox and logistic regression models using only clinical factors. The clinical factors included in both models were selected based on their statistical significance in univariate analysis performed on the training set (Supplementary Table S4). The clinical-based models were then compared to SMuRF and combined with it on the test set. SMuRF outperformed the clinical-based models for both endpoints on the test set (Supplementary Fig. S3, C-index for DFS: 0.79 vs. 0.57; AUC for grade: 0.74 vs. 0.60). While combining the clinical-based models with SMuRF did not improve DFS predictions, it enhanced tumour grade classification.

Fig. 3.

Multivariable Cox regression analysis using DFS as endpoint and multivariable logistic regression using grade as endpoint on the test set (a). Beeswarm plot of SHAP variable importance in the multivariable Cox regression analysis (b) and the multivariable logistic regression analysis (d). Mean SHAP values were converted to proportion for each variable, quantifying their contributions to the DFS predictions (c) and the grade classification (e).

To test the model’s resilience against batch effects, we performed qualitative t-distributed stochastic neighbour embedding (t-SNE) visualization⁵³ of the 16-dimensional features from the SMuRF penultimate layer. For each patient coming from any of the 2 institutions, SMuRF penultimate features were reduced to two-dimensional space and were plotted using sites as the corresponding labels. Likewise, we also performed the same t-SNE mapping except for the data points are colour-coded using the labels corresponding to 3-year DFS. There was no obvious cluster of data points observable based on the site labels, while there were two distinct clusters discernible as a result of the survival outcomes (Supplementary Fig. S4).

Interpretabilities of SMuRF

The multitask loss and C-index/AUC with respect to epoch on training and validation sets are provided in Supplementary Fig. S5. The Integrated Gradients (IG) on radiology CT images for four representative patients are illustrated in Fig. 4, two being predicted as having poor prognosis (high-SMuRF) and two being predicted as having good prognosis (low-SMuRF). The IG heatmaps for primary tumours indicate that both intratumoral and peritumoral regions carry prognostic value (Fig. 4d). In contrast, the IG heatmaps for lymph nodes are more localized within the lymph node itself (Fig. 4f). These findings potentially highlight the differing roles of surrounding tissues outside of the cancerous area in primary tumours vs. lymph nodes, which could inform the development of region-specific prognostic models.

Fig. 4.

SMuRF histogram distributions on validation and test sets (a), and four representative patient examples of clinical information (b), cropped CT scans with primary tumour (c) and metastatic cervical lymph node annotations (e). Corresponding integrated gradient (IG) attention maps (d, f) highlighted the important regions for predictions. The IG results shown that the deep learning model focused regions within the primary tumour and metastatic cervical lymph nodes.

To visualize morphologic attributes at varying scales identified by SMuRF on pathology WSI, we extracted the pathology embeddings from HIPT model and overlaid the corresponding attention heatmap back on to WSI. Within a 2048 × 2048 image patch, there exhibits a hierarchical structure of visual tokens: at macro-scale, the 256 × 256 patches capture tumour-collagen fibre interface (Fig. 5c) and tumour cell clusters (Fig. 5d). At the micro-scale, the 16 × 16 patches encompass a bounding box of fine-grained features such as individual collagen fibres (Fig. 5e) and tumour cells (Fig. 5f), as highlighted by the attention heatmaps.

Fig. 5.

One example high-SMuRF (a) and one example low-SMuRF (b) pathology WSIs with primary tumour annotations (red boundaries) and IG overlaid to highlight prognostic relevant regions (bottom left) detected by SMuRF. For each pathology WSI, two example 2048 × 2048 regions with corresponding attention heatmaps (c, d) within the highlighted prognostically relevant regions are provided. The attention heatmaps of HIPT model at a patch size of 256 (macro-scale) expressed differently for the high- and low-SMuRF patients: there are more condensed high attention areas related to the tumour-collagen fibre interface (c) for the high-SMuRF WSI than the low-SMuRF WSI, which contains primarily the tumour cell clusters (d). The attention heatmaps at a patch size of 16 (micro-scale) highlighted individual collagen fibre (e) for high-SMuRF while emphasized mainly the tumour cells (f) for low-SMuRF WSI. Presence of individual morphologic hallmarks (i.e., tumour-collagen fibre interface, tumour cell clusters) were evaluated and confirmed by a pathologist (T.P.).

Discussion

Numerous biomarkers have been developed for cancer prognostication, leveraging histology slides,54, 55, 56, 57 radiology images,58, 59, 60 and gene expression signatures.^61,62 Most existing prognostic models primarily depend on unimodal data or combine information solely from clinical variables (such as N-stage and smoking status⁶³), neglecting the valuable insights that can be gained from multiscale image modalities.^22,64 Integrating radiology images and histology slides present a pivotal opportunity for automated multimodal biomarker discovery.65, 66, 67 Existing studies showed promising results in combining the tumour region from these two modalities for risk stratification for ovarian cancer,⁶⁸ prostate cancer,⁶⁹ lung cancer,⁷⁰ brain tumour,^35,71 nasopharyngeal carcinoma,⁷² rectal cancer⁷³ and breast cancer.⁷⁴ However, comparatively less attention has been directed towards understanding the joint prognostic implications of primary tumour and lymph node involvement across both radiology and pathology. We believe that the discovery of prognostic biomarkers for oropharyngeal cancer could be enhanced through a deep learning-based multiregional and multimodal approach.

In this work, we present SMuRF, a Swin Transformer-based multimodal and multi-region imaging fusion assay to predict OPSCC survival and tumour grade outcomes. SMuRF is an end-to-end deep learning model incorporating multimodal imaging data, which explicitly retains and exploits spatial context from primary tumour and metastatic cervical lymph node across CT scans and WSI. Developed and validated on a cohort of 277 patients with OPSCC with matched radiology CT scans and pathology WSI, SMuRF significantly risk-stratified HPV-associated OPSCC into high- and low-risk groups with distinct DFS and was also found to be associated with tumour grade across two test datasets (Fig. 2). In addition, SMuRF was found to be an independent prognostic factor for HPV-associated OPSCC, after controlling for clinical variables (Fig. 3). When applying the SHAP variable analysis on both DFS and tumour grade, SMuRF demonstrated consistent prognostic performance, emerging as the most significant variable in the multivariable Cox and logistic regression analysis. In the comparative experiments, SMuRF as a multimodal approach significantly outperformed the unimodal radiology model (tumour + node) and even several other multimodal platforms with single region input, suggesting that the multiscale and multi-region image representation fusion induced by the dual modality is adding additional prognostic granularities. While tumour grade in OPSCC is not typically considered a clinically relevant endpoint, its evaluation through imaging could offer valuable insights into the efficacy of the methodology. In addition, survival outcomes are complex endpoints and are influenced by clinical confounders, so incorporating a secondary endpoint like tumour grade, which is also linked to prognosis, can help to capture additional prognostic information.⁷⁵

HPV-associated OPSCC is typically characterized by a favourable clinical prognosis and treatment response.¹ Consequently, there is increasing interest in tailoring treatment strategies aimed at reducing therapy-related morbidity in “low-risk” HPV-positive OPSCC and enhancing clinical outcome for the “high-risk” group.^28,29 However, de-intensification based on clinical staging has not yielded encouraging results based on recent reports⁷⁶ as patients omitting chemotherapy based on conventional TNM staging criteria did not achieve the desired 2-year disease-free survival rate exceeding 85%. This underscores the urgent need for identifying prognostic biomarkers that can enhance the identification of true “low-risk” patients who stand to benefit from treatment deintensification and true “high-risk” patients suitable for treatment intensification. SMuRF could separate the HPV-associated OPSCC into high-vs. low-risk groups based on disease-free survival, representing a potential useful tool for guiding treatment deintensification. In addition, SMuRF could also identify a subset of high-risk patients with OPSCC with significantly worse survival outcome, potentially suggesting that these patients are suitable candidates for first line immunotherapy.

One of the main technical innovations of SMuRF is its ability to reconcile multiregion and multiscale image representations. Within the radiology CT images, our model highlights the significance of both intratumoral and peritumoral regions in primary tumours, aligning with prior evidence linking the peritumoral microenvironment to cancer progression and metastasis.^77,78 In contrast, for lymph node-based predictions, only the internal features of the lymph node—such as size, shape, and texture—appear to provide meaningful prognostic information. This emphasizes the importance of focusing on the lymph node itself, rather than the surrounding tissues, when assessing its prognostic value. Within the pathology WSI data, we utilized the Hierarchical Image Pyramid Transformer (HIPT) to extract multi-scale embeddings.^16,37,79 WSI data exhibits a hierarchical structure of visual tokens at varying image scales depending on the patch size: the 16 × 16 image patches encompass the bounding box of cells and other fine-grained features⁸⁰ (stroma, tumour cells, lymphocytes), 256 × 256 image patches capture local clusters of cell-to-cell interactions⁸¹ (tumour cellularity) and 4096 × 4096 image patches further characterize macro-scale tissue interactions.⁸² Through applying self-attention³⁰ on pretrained embeddings¹⁶ on varying sizes of image patches, the output from HIPT reflect scaled representations aware of both micro- and macro-scale interactions of morphologic attributes on the pathology WSI: for example, collagen-fibre interface and tumour cell clusters were identified as prognostic relevant regions at a macroscale and individual collagen fibre and tumour cells were clearly captured at a microscale by HIPT (Fig. 5). In addition to the multiscale attributes identified by the HIPT model, there appears to exist inherent scale difference between the radiology CT and pathology WSI representations. Such multiscale features are learnt and encoded through four consecutive SwinT blocks, which progressively lower the input resolution, making them conducive for the subsequent survival prediction tasks. Empirical results from comprehensive ablation studies demonstrated that SMuRF outperformed ten comparable models, achieving a superior C-index for DFS prediction and a higher AUC for tumour grade classification. These results suggest that the complementary prognostic information from the microscopic WSI and macroscopic CT images are effectively captured and utilized by SMuRF.

Another unique aspect of SMuRF is that the feature interactions across multi-region and multiscale are both considered and learnt, enabling more seamless data integration compared with the conventional concatenation-based approaches^72,74 and kronecker product fusion.³⁸ Specifically, instead of applying late-stage aggregation, SMuRF involves multi-head self-attention³⁴ to capture region-wise feature interactions (i.e., tumour and node within CT and different fragments within WSI) after each downscale by the SwinT blocks. To capitalize multiscale interactions across CT and WSI, we applied self-attention pooling³⁰ as the multiscale fusion inspired by transformer-related architectures. The premise behind this fusion scheme is that the intricate feature interplays among various regions and modalities may have been overlooked during more naïve fusion strategies like concatenation.72, 73, 74 To test this hypothesis, we assessed our fusion method against three alternative fusion approaches using identical multimodal data inputs. SMuRF embedded with self-attention pooling outperformed the models using vector concatenation and Kronecker product. These results empirically suggest that the multiscale and multi-region feature interactions are internally captured by SMuRF.

AI-based prognostic approaches on radiology data have been explored for OPSCC risk stratification.^28,83, 84, 85, 86 Cheng et al.²² developed a model termed as “DeepPET-OPSCC”, to predict OS in OPSCC using fluorodeoxyglucose (FDG)-PET imaging. On a cohort of 353 patients on the test set, their model achieved hazard ratio of 2.39 and C-index of 0.689. While these results are promising, HPV-associated and HVP-independent OPSCC were combined in the same analysis, neglecting the striking difference in biology and outcome between these two diseases.⁸⁷ In another study conducted by Song et al.²¹ using hand-crafted radiomic approach, a C-index of 0.64 was obtained when using OS as the endpoint. In the context of utilizing AI-based computational pathology for OPSCC survival prediction, Koyuncu et al. developed a pathology-based imaging biomarker quantifying tumour multinucleation index (MuNi) to predict p16+ OPSCC survival.¹³ Another study by Corredor et al. investigated a signature characterizing the spatial interplay between tumour-infiltrating lymphocytes (TILs) and surrounding cells.¹⁵ C-indices were not reported in these two studies despite significant survival separations being observed between predicted risk groups across multiple data cohorts. While many of the aforementioned studies had superior prognostic performance compared to the unimodal radiology or pathology models implemented in this study, they were inferior to SMuRF which utilized both radiology and pathology. However, SMuRF was only internally validated for non-metastatic HPV-associated OPSCC undergoing curative radiotherapy or chemoradiation, and we recognize that the applicability of the model may be limited to this patient population.

Our study was not without its limitations. First, SMuRF itself as a multimodal learning framework included only imaging data without incorporating genomic sequence data. Second, SMuRF was only validated in terms of prognostic value; identifying its potential predictive value to treatment benefit and response (e.g., immunotherapy) remains part of future work. Third, although primary tumour and metastatic lymph node were jointly analysed on radiology CT images, only primary tumour was included from pathology WSI due to lack of tissue specimens for lymph nodes given the prevailing pattern of clinical practice. Additionally, biopsy samples from the primary tumour may not capture the full extent of the TME’s heterogeneity as effectively as surgical resections might. Furthermore, multiple representative slides may be taken from the tumour sites, but here we selected the best available slide based on tumour size and tissue representation for computational analysis. Finally, the limited study power due to the absence of an external validation set and the small event rates for both oropharyngeal recurrence and death should be considered as potential limitations. The findings are based on a single cohort, and external validation is warranted to confirm the reproducibility of these results.

Despite the aforementioned limitations, this study introduces a deep learning fusion framework that integrates information from various cancer habitats across multiscale imaging data. It combines radiology and pathology imaging as predictors for survival and tumour grade outcomes in the context of oropharyngeal cancer. With subsequent validation, SMuRF has the potential to enhance our current risk stratification tools for HPV-related OPSCC. The multiscale and multi-region imaging data fusion scheme presented in this work can also be adapted to address other multimodal learning and data integration questions in cancer research.

Contributors

Bolin Song and Anant Madabhushi designed the study; Bolin Song, Kailin Yang, Deborah J. Chute collected the data; Bolin Song, Tanmoy Dam, Himanshu Maurya and Xiangxue Wang analysed the data; Bolin Song and Amaury Leroy proposed the model; Bolin Song wrote the original draft; Amaury Leroy, Jonathan Lee, Sarah Stock, Xiao T Li, Pingfu Fu, Cheng Lu, Paula Toro, Deborah J. Chute, Shlomo Koyfman, Nabil F Saba, Mihir R Patel and Anant Madabhushi revised the manuscript. Bolin Song and Amaury Leroy contributed equally to this work. Bolin Song, Tilak Pathak, Jonathan Lee, Sarah Stock, Paula Toro, Deborah J. Chute directly accessed and verified the underlying data reported in the manuscript. All authors read and approved the final version of the manuscript.

Data sharing statement

Detailed intermediate data on all results are available immediately following publication upon request to the first author at bsong47@emory.edu. The data associated with the study has not been deposited into a publicly available repository. The authors do not have permission to share data. Requests to access the datasets from Cleveland Clinic and Winship Cancer Institute (used with permission for this study) should be made directly to the institutions via their data access request forms. The code implementations are available on GitHub: https://github.com/BolinSong1995/SMuRF_MultiModal_OPSCC/tree/main.

Declaration of interests

Dr. Madabhushi is an equity holder in Picture Health, Elucid Bioimaging, and Inspirata Inc. Currently he serves on the advisory board of Picture Health, and SimBioSys. He currently consults for Takeda Inc. He also has sponsored research agreements with AstraZeneca and Bristol Myers-Squibb. His technology has been licenced to Picture Health and Elucid Bioimaging. He is also involved in 2 different R01 grants with Inspirata Inc. He also serves as a member for the Frederick National Laboratory Advisory Committee. Dr. Kailin Yang was supported by RSNA Research Fellow Grant and ASTRO-LUNGevity Foundation Radiation Oncology Seed Grant.