Multimodal data fusion for cancer biomarker discovery with deep learning

Abstract

Technological advances now make it possible to study a patient from multiple angles with high-dimensional, high-throughput multi-scale biomedical data. In oncology, massive amounts of data are being generated ranging from molecular, histopathology, radiology to clinical records. The introduction of deep learning has significantly advanced the analysis of biomedical data. However, most approaches focus on single data modalities leading to slow progress in methods to integrate complementary data types. Development of effective multimodal fusion approaches is becoming increasingly important as a single modality might not be consistent and sufficient to capture the heterogeneity of complex diseases to tailor medical care and improve personalised medicine. Many initiatives now focus on integrating these disparate modalities to unravel the biological processes involved in multifactorial diseases such as cancer. However, many obstacles remain, including lack of usable data as well as methods for clinical validation and interpretation. Here, we cover these current challenges and reflect on opportunities through deep learning to tackle data sparsity and scarcity, multimodal interpretability, and standardisation of datasets.

Introduction

Over the past decades, technological innovations have transformed the healthcare domain with ever-growing availability of clinical data supporting diagnosis and care. Medicine is moving towards gathering multimodal patient data, especially in the context of age-related chronic diseases such as cancer^{1, 2}. Integrating different data modalities can enhance our understanding of cancer^{3, 4}, but also paves the way for precision medicine which promises individualised diagnosis, prognosis, treatment and care^{1, 5, 6}.

Increasingly, we are moving from the traditional one-size-fits-all approach to more targeted testing and treatment. While molecular pathology revolutionised precision oncology, the first FDA-cleared companion diagnostic (CDx) assays relied on simpler molecular methods, and most assays focused on a single-gene of interest^{7, 8}. However, advances in next-generation sequencing (NGS) now allow for multi-target CDx assays which are becoming more prevalent^{8, 9}. The continuing cost reduction make it possible to simultaneously profile thousands of genomic regions hinting that soon multi-target panels could be run at a similar price point to that of testing 5 to 10 targets individually¹⁰. Multi-target tests not only conserve time and tissue, but also have the potential to identify complex genetic interactions, thereby enhancing our understanding of tumour biology. While NGS is still in full swing, a third wave of technologies featuring single-molecule, long-read and real-time sequencing is already on the rise. Pacific Biosciences (PacBio) and Oxford Nanopore Technologies allow to assemble and explore genomes at unprecedented resolution and speed¹¹. This technology was recently used in a clinical setting to diagnose rare genetic diseases with a turnaround rate of only 8 hours¹². Since cancer often is multicausal, the area of precision oncology greatly benefits from these developments.

At the same time, histopathology and radiology have been critical tools in clinical decision-making during cancer management^{13, 14}. Histopathological evaluation enables the study of tissue architecture and remains the gold standard for cancer diagnosis¹⁵. More recently, significant progress in whole slide imaging (WSI) has led to a transition from traditional histopathology methods towards digital pathology¹⁶. Digital pathology, the process of “digitising” conventional glass slides to virtual images, has many practical advantages over more traditional approaches, including speed, more straightforward data storage and management, remote access and shareability, and highly accurate, objective, and consistent readouts. On the other end of the spectrum is radiographic imaging, a non-invasive method for detecting and classifying cancer lesions. In particular Computed Tomography (CT) and Magnetic Resonance Imaging (MRI) scans are useful for generating 3D images of (pre)malignant lesions.

Ongoing improvements in artificial intelligence (AI) and advanced machine learning (ML) techniques have had major impacts on these cancer imaging ecosystems, especially in diagnostic and prognostic disciplines¹⁷. Current annotation of histopathological slides heavily relies on specialised pathologists. Leveraging image-based AI applications would not only alleviate the pathologists’ workload but also has the potential for more efficient, reproducible, and accurate spatial analysis capturing information beyond visual perception^17–19. Radiomics and pathomics refer to fields focusing on the quantitative analysis of radiological or histopathological digital images, respectively, with the aim of extracting quantitative features that can be used for clinical decision-making²⁰. This extraction used to be done with standard statistical methods, but more advanced deep learning (DL) frameworks like convolutional neural networks (CNN), deep autoencoders (DAN) and vision transformers (ViTs) are now available for automated, high-throughput feature extraction^21–24. Automatic assessment of deterministic objective features has enabled the quantification of tumour microenvironments (TME) at unprecedented speed and scale. In addition to the quantification of known hand-crafted salient features without inter-observer variability, DL also has the ability to discover unknown features and relationships that can provide biological insights and improve disease characterisation²⁵. A notable radiomics study in lung cancer found that DL features captured prognostic signatures, both within and beyond the tumour region, that correlated with cell cycle and transcriptional processes²⁶. Despite DL’s diverse capacity, one of the main challenges is the need for large datasets to train, test and validate its algorithms. But, due to ethical restrictions and the labour intensity to annotate clinical images, most studies only have limited access to large cohorts that contain ground truth labelled data²⁷.

Under the 21^st Century Cures Act²⁸, the FDA set a goal to advance precision medicine where the patient is at the centre of care. This act defines timelines for discovery, development, and delivery, and requires the fusion of evidence across modalities, with the provision that this must include real-world data and patient experience. Technological advances initiated an era where clinical data is being captured from multiple sources at unprecedented pace, ranging from medical images to genomics data and patient-generated health data (PGHD). Together with successes in AI, this opens the opportunity and necessity to analyse many data types with these advanced tools to better inform decision-making and improve patient care. To date, the FDA has cleared and approved several AI-based software as a medical device (SaMD)²⁹. Together with the publication of their recent AI/ML white paper³⁰ the FDA wants to highlight their intention to develop a regulatory framework for these highly iterative, autonomous, and continuously learning algorithms as well as for the specific data types necessary to assure safety and effectiveness. Some proposed considerations for data inclusion are (i) relevance to the clinical problem and current clinical practice, (ii) data acquisition in a consistent, generalisable, and clinically relevant manner, (iii) appropriate definition and separation of training, tuning and test sets, and (iv) appropriate level of transparency of the algorithm and its output to users.

Integration of AI functionalities in medical applications has increased in recent years³¹. However, so far most methods focused on only one specific data type at a time, leading to slow progress in approaches to integrate complementary data types with many remaining questions about the technical, analytical and clinical aspect of multimodal integration^32–35. To advance precision oncology, healthcare AI should not only inform about cancer incidence and tumour growth, but must identify the optimal treatment path, accounting for treatment-related side effects, socioeconomic factors, and care goals. Precision medicine can therefore only be achieved by merging complex and diverse multimodal data that span space and time. Single data modalities can be noisy or incomplete, but when combined with redundant signals from other modalities they can be more sensitive and robust to diagnose, prognose and assign treatments. Multimodal data are now being collected, providing a resource for biomarker discovery^36–39. For cancer, both prognostic and predictive biomarkers are of interest. While the former provides information on the patient’s diagnosis and overall outcome, the latter informs about treatment decisions and response⁴⁰.

Here, we argue that several sources of routinely collected medical data are not used to their full potential for diagnosing and treating cancer patients, because they are studied mostly in isolation instead of in an integrated fashion. These are: (i) electronic health records (EHR), (ii) molecular data, (iii) digital pathology and (iv) radiographic images. When combined, these data modalities provide a wealth of complementary, redundant, and harmonious information that can be exploited to better stratify patient populations and provide individualised care (Fig. 1). In the next sections, we discuss both challenges and opportunities for multimodal biomarker discovery as it applies to cancer patients. We cover strategies for data fusion and examine approaches to address data sparsity and scarcity, data orchestration and model interpretability.

Fig. 1: Generation and processing of routinely collected biomedical modalities in oncology.

Prior to data fusion, different steps are needed to go from the raw data to workable data representations for each modality, e.g. EHRs, molecular data and medical images.

The need for multimodal data fusion in oncology

Despite huge investments in cancer research and improved diagnosis and treatments, cancer prognosis is still bleak. Predictive models based on single modalities offer a limited view of disease heterogeneity and might not provide sufficient information to stratify patients and capture the full range of events that take place in response to treatments^{41, 42}. For example, although immunotherapeutic methods like antibody-drug conjugates (ADCs) and adoptive cell therapy (ACT) (e.g. T-cell receptor (TCR) and chimeric antigen receptor T-cell (CAR-T) therapy) have shown to be very promising, response rates vary dramatically depending on the tumour subtype⁴³ and the TME⁴⁴. Various TME elements play a role in tumour development, but also in therapeutic response. Furthermore, the cellular composition of the TME dynamically evolves with tumour progression and in response to anticancer treatments^{45, 46}. The increasing application of immunotherapy underlines the need for (i) a deeper understanding of the TME and (ii) multimodal approaches that allow longitudinal TME monitoring during disease progression and therapeutic intervention⁴⁷.

Currently, biomarker discovery is mainly based on molecular data⁴⁸. Increasing implementation of genomics and proteomic technologies in a clinical setting has led to growing availability, but also growing complexity, of molecular data⁸. Large consortia like The Cancer Genome Atlas (TCGA) and Genomic Data Commons (GDC) have gathered and standardised large datasets, accumulating petabytes of genomic, expression and proteomics data^{37, 49, 50}. Barriers for NGS assay development, validation, and routine implementation remain due to many factors, such as tumour heterogeneity, sampling bias and interpretation of the results. Clinically accepted performance requirements are also often cancer-specific and depend on where in the care trajectory and for what specific purpose (e.g. diagnostic, stratification, drug response or treatment decision) tests are used⁵¹. As relevant as molecular data are for precision medicine, they discard tissue architecture, spatial and morphological information.

Although lower in resolution than genomic information, both WSI and radiographic images potentially harness orthogonal and complementary information. Digital pathology with WSIs provides data about the cellular and morphological architecture in a visual way for pathologists to interpret and can provide key information about the TME’s spatial heterogeneity using image analysis and spatial statistics⁵². Similarly, radiographic images like MRIs or CT scans provide visual data of the tissue morphology and 3D structure⁵³.

Integration of data modalities that cover different scales of a patient has the potential to capture synergistic signals that identify both intra- and inter-patient heterogeneity critical for clinical predictions^54–56. For example, the 2016 WHO classification of tumours of the central nervous system (CNS) revisited the guidelines to classify diffuse gliomas recommending histopathological diagnosis in combination with molecular markers (e.g. IDH1/2 mutation status), as each modality alone is insufficient to explain patient outcome variance^{32, 33}. Of late, some reports also suggest the use of DNA-methylation-based classification of CNS tumors^{34, 35}.

The need for integrative modelling is increasingly emphasised. In 2015, a report from Ritchie et al. highlighted that “approaches to combine multiple data types provide a more comprehensive understanding of complex genotype-phenotype associations than analysis of one dataset”⁵⁷. The last years, there have been several attempts to develop multimodal approaches, to a great degree stimulated by community-driven competitions such as DREAM and Kaggle (i.e. http://dreamchallenges.org/ and https://www.kaggle.com/). But more work is needed to integrate routinely collected data modalities into clinical decision systems.

Data fusion strategies for multimodal biomarker discovery

The age of precision medicine demands powerful computational techniques to handle high-dimensional multimodal patient data. Each data source has strengths and limitations in its creation, analysis, and interpretation that must be addressed.

Medical images, whether 2D in histopathology or 3D in radiology, contain dense information that is encoded at multiple scales. Importantly, they contain high spatial correlation and any successful approach needs to take this into account⁵⁸. So far, the best performing methods have been based on DL, and specifically CNNs^59–61. Continuous improvement in detection, segmentation, classification, and spatial characterisation means that these methods are becoming a crucial part of cancer biomarker algorithms.

EHRs comprise various data types ranging from structured data such as medications, diagnosis codes, vital signs, or lab tests, to unstructured data in the form of clinical notes, patient emails, and detailed clinical processes. Natural language processing (NLP) algorithms that can extract useful clinical information from structured and unstructured EHR data are being developed. A recent study showed the feasibility and power of such ML tools in a lung cancer cohort to reliably extract important prognostic factors embedded in the EHRs ⁶². Structured EHR sources are the easiest to process. Usually, this data is embedded into a lower dimensional vector space and fed as input to a recurrent neural network (RNNs). Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU) are the most popular RNN architectures for this purpose^63–65. While structured EHR data have obvious value, integration with insights from unstructured clinical data has shown to greatly improve clinical phenotyping⁶⁶. Fortunately, advances in NLP now make it possible mine the unstructured narratives of patient records. One way to process this data is to convert free text to medical concepts and create lower dimensional “concept embeddings”. Older methods such as Word2Vec⁶⁷ and GloVe⁶⁸ have almost been overtaken by “contextualised embeddings” like ELMo⁶⁹ and BERT^70–72. While ELMo uses RNNs, BERT is based on transformers, a neural architecture that has revolutionised the NLP field since its inception⁷³. To unlock EHRs’ full potential, more appropriate techniques are needed combining structured and unstructured information, while accounting for the noise and inaccuracies that are common to these data⁷⁴. In this regard, the concept of transfer learning for extracting clinical information from EHRs has gained a lot of traction⁷⁵.

Effective fusion methods must integrate high-dimensional multimodal biomedical data, ranging from quantitative features to images and text⁷⁶. Representing raw data in a workable format remains challenging as ML methods do not readily accept unvectorised data. A multimodal representation thus poses many difficulties. Different modalities measure distinct unmatched features with different underlying distributions and dimensionalities. Also, not all modalities and observations have the same level of confidence, noise, or information quality⁷⁷. Multimodal fusion often suffers from dealing with wide feature matrices originating from very few samples with many features across modalities. Often advanced feature extraction methods such as kernel-based methods, graphical models, or NNs are needed prior to or as part of the data fusion process to reduce the dimensionality while preserving most of the salient biological signals^77–80. Meaningful feature descriptions are the critical backbone of any model.

A major decision that must be made is at what specific modelling stage the data fusion takes place: (i) early, (ii) intermediate or (iii) late (Fig. 2)^81–83. Early fusion is characterised by concatenating feature vectors of different data modalities and only requires the training of a single model (Fig. 2a). In contrast, late fusion is based on developing models on each data modality separately and integrating their single predictions with specific averaging, weighting, or other mechanisms (Fig. 2c). Not only does late fusion allow the use of a different, often more suitable, model for each modality but it also makes it more straightforward to handle situations when some modalities are missing in the data. However, fusion at the late stage ignores possible synergies between different modalities⁸⁴.

Fig. 2: Overview of different fusion strategies for multimodal data.

a) Raw data is processed into workable formats. b) For each modality features are extracted using dedicated encoder algorithms. c) Early fusion. d) Intermediate fusion. e) Late fusion.

While both early and late fusion approaches are model-agnostic, they are not specifically designed to cope with or take full advantage of multiple modalities. Anything between early and late fusion is defined as intermediate or joint data fusion⁸⁴. Intermediate fusion does not merge input data, nor develops separate models for each modality, but instead involves the development of inference algorithms to generate a joint multimodal low-level feature representation that retains the signal and properties of each individual modality (Fig. 2b). Although dedicated inference algorithms must be developed for each model type, this approach attempts to exploit the advantages of both early and late fusion^{79, 83}. One key difference with early fusion is that the loss is propagated back to the inference algorithms during training, thus creating updated feature representations per training iteration⁸⁴. Although this allows to model complex interactions between modalities, techniques need to be in place to prevent overfitting on the training cohort. Importantly, there is currently no decisive evidence that one fusion strategy is superior, and the choice of a specific approach is usually empirically based on the available data and task⁸⁴.

Advances in multimodal biomarkers for patient stratification

Multi-omics data fusion

Although a single omics technology provides insights into the profile of a tumour, one technique alone does not fully capture the underlying biology. The rising collection of large cohorts of multi-omics cancer data has spurred several efforts to fuse multi-omics data to fully grasp the tumour profile and several models for survival and risk prediction have been proposed^{4, 6, 56, 85–93}. The TCGA research network also published numerous papers investigating the integration of genomic, transcriptomic, epigenomic and proteomic data for multiple cancer types^94–96. Also for therapy response and drug combination predictions, multi-omics ML methods proved their value over traditional unimodal models^97–100. Although various multi-omics fusion strategies now exist, one single method will not be optimal for all research questions and data types, and sometimes adding more omics layers can even negatively impact performance¹⁰¹. Each strategy has its own strengths and weaknesses, and careful selection of effective approaches should be based on the purpose and available data types⁵⁷.

Multi-scale data fusion

Similar efforts as for multi-omics data fusion have been explored for multi-scale data^{89, 102–107}. For example, Cheerla et al. used an intermediate fusion strategy to integrate histopathology, clinical, and expression data to predict patient survival for multiple cancer types. For each modality, an unsupervised encoder compressed the data into a single feature vector per patient. These feature vectors were aggregated into a joint representation allowing possible absence of one or more modalities⁴⁸. Similarly, another study proposed a late fusion strategy to classify lung cancer. Using RNAseq, miRNAseq, WSI, copy number variation, and DNA-methylation they achieved better performance than obtained by each individual modality¹⁰⁸. A few examples exist that show the potential of radiology to further refine patient stratification^109–111. However, due to its high dimensionality and computational demands, so far most studies have avoided its inclusion¹¹².

Imaging genomics & Radiogenomics

When possible, molecular tumour information is nowadays used in cancer prognosis and treatment decisions. Interestingly, multiple studies have shown that phenotypes derived from medical images can act as proxies or biomarkers of molecular phenotypes like an EGFR mutation in lung cancer^113–115. This discovery immediately gave rise to an emerging field called “radiogenomics”, the study of directly linking image features to underlying molecular properties¹¹⁶. For example, Itakura et al. used MRI phenotypes to define subtypes of glioblastoma associated with molecular pathway activity¹¹⁷. Also for breast cancer, the value of radiogenomics for risk prediction and better subtype stratification has been shown^118–120.

Current challenges and future directions for multimodal data fusion

Use of multimodal data models is likely the only way to advance precision oncology, but many challenges exist to realise their full potential. Although data availability is the main driver of multimodal data fusion, it also poses its major barrier. DL requires large amounts of data and data sparsity and scarcity both present serious challenges, especially for biomedical data. In clinical practice there are often different types of data missing between patients as not all patients might have all modalities due to cost, insurance coverage, material availability and lack of systemic collection procedures amongst others. To become relevant in an oncology setting, methods need to be able to handle different patterns of missing modalities. Fortunately, various interpolation, imputation and matrix completion algorithms have already been successfully applied for clinical data. These can range from basic methods including mean/median substitution, regression, k-nearest neighbour, and tree-based methods to more advanced algorithms like multiple imputation, multivariate imputation by chained equations or NN like RNN, LSTM and GANs^121–123. Also, with the recent successes in DL techniques, dedicated fusion approaches are becoming available that allow joint representations that can handle incomplete or missing modalities^{48, 124–129}.

However, there are two major hurdles to advance these efforts. Firstly, the depth of data per patient, i.e. many observables per patient are routinely generated and stored, but typical cohort sizes of patients are relatively small. Emerging evidence highlights that these cohorts are often biased, representing patients from higher socioeconomic status with continuous access to care and high levels of patient engagement^{130, 131}. Limiting analyses to patients with complete data will lead to model overfitting, bias, and poor generalisation. Secondly, the lack of large ‘golden labelled’ cohorts with matched multimodal data, mainly due to the intense labour to annotate cancer datasets combined with privacy concerns. Luckily, also here DL algorithms are starting to be developed. One popular approach is data augmentation^132–135, which can include basic data transformations as well as generation of synthetic data, but also other strategies such as semi-supervised learning^136–139, active learning^{140, 141}, transfer learning^{139, 142–144}, and automated annotation^{145, 146} have shown to be promising avenues to overcome labelled data scarcity.

Despite its potential, a critical roadblock for the widespread adoption of DL in a clinical setting is the lack of well-defined methods for model interpretation. While DL can extract predictive features from complex data, these are usually abstract, and it is not always apparent if they are clinically relevant¹⁴⁷. To be useful in clinical decision-making, models need to undergo extensive testing, be interpretable, and their predictions need to be accompanied by confidence or uncertainty measures^{148, 149}. Only then will they be relevant for and adopted by clinical practitioners.

Interpretation of black box models is a heavily investigated topic and some methods for post-hoc explanations have been proposed^{147, 150}. In histopathology, most work focuses on extracting the most informative tiles by selecting those with the highest model confidence or by visualising tiles that are most relevant to the final prediction (Fig. 3a). For interpreting model predictions at higher resolution, the most relevant regions can be highlighted using gradient-based interpretation methods like Grad-CAM (Fig. 3b)¹⁵¹. Similarly, for molecular data, predictive features can be determined and visualised via Shapley Additive Explanation (SHAP)-based methods (Fig. 3d,e)^{150, 152–154}.

Fig. 3: Examples of model interpretability methods for histopathology and gene expression.

Histopathology: a) Examples of informative tiles for predicting the presence of TP53 mutations from histopathology images in prostate cancer (unpublished data). b) Visualisation of regions within tiles most relevant to the prediction, derived via Grad-CAM¹⁵¹. c) Individual cells within informative tiles are segmented and classified by Hover-Net¹⁵⁵. For a fine-grained interpretation of relevant cells (black annotations), pertinent cells within the tile are encircled by calculating the contours from regions highlighted by Grad-CAM. Gene Expression: d) Examples of SHAP visualisation¹⁵² of hypothetical gene importance according to unimodal model (top) and joint multimodal model (bottom) for cancer survival prediction. e) Example of pathway importance visualisation based on the respective gene SHAP-values in unimodal (top) versus joint multimodal (bottom) models with respect to cancer survival prediction¹⁵⁴.

Multimodal data adds additional complexity and needs careful evaluation of appropriate methods before scaling to multimodal interpretability. However, multimodal approaches are starting to emerge with encouraging solutions not only for interpretability but also for discovery of associations between modalities^{147, 150}. Note that the aforementioned methods specify why a model makes a specific decision, but do not explain the used features. Additional strategies could be leveraged to further unravel biological insights. For example, selected tiles could be overlayed with Hover-Net¹⁵⁵ to segment and classify nuclei to evaluate predominant cell types (Fig. 3c, unpublished data).

Standardisation will lead to more uniform and complete datasets, which are easier to process and fuse with other sources and will be much more interpretable on their own. TCGA is probably the best known and most used resource³⁷, but many other initiatives are underway to structurally capture clinical, genomics, imaging, and pathological data for oncology, such as The Cancer Imaging Archive (TCIA)³⁶ and the Genomics Pathology Imaging Collection (GPIC)³⁸. Together, these efforts have the shared aim to process, analyse and share data using a community-embraced standard in a FAIR (findable, accessible, interoperable, and reusable) way¹⁵⁶. This will not only promote reproducibility and transparency, but also encourages reutilisation and optimisation of existing work. However, the volume and complexity of multimodal biomedical data makes it increasingly difficult to produce and share FAIR data and current solutions often require specific expertise and resources¹⁵⁷. Furthermore, some modalities such as EHRs are not only extremely difficult to standardise and share, but also very expensive to obtain by researchers^{158, 159}. Efforts like OMOP aim at tackling this issue by harmonising EHR data across institutes and countries^{160, 161}. To make progress in multimodal studies, there is a dire need for data orchestration platforms¹⁵⁷, but also appropriate regulatory frameworks to preserve patients’ privacy¹⁶².

The importance of biomedical multimodal data fusion becomes increasingly apparent as more clinical and experimental data becomes available. To tackle the multimodal-specific obstacles, multiple methods and frameworks have been proposed and are currently heavily explored. While often still problem-specific and experimental, the field is gaining knowledge to evaluate and define what methods excel given specific conditions and data modalities. DL approaches have only touched a limited range of potential applications, mainly because of the challenges inherent to the current state of health care data as discussed above, again emphasising the need for large collaborative data standardisation and sharing efforts. In this space, competitions such as DREAM and Kaggle have been an effective concept for making standardised multimodal data available. Importantly, these initiatives also facilitate exchange of ideas and code, reproducibility, innovation, and unbiased evaluation^{163, 164}. It is our expectation that such efforts will significantly advance development of robust multimodal approaches.

Ultimately, the goal is to advance precision oncology by rigorous clinical validation of successful models in larger independent cohorts to prove any clinical utility. So far, most efforts have focused on multimodal cancer biomarkers to refine risk stratification, but with dedicated strategies multimodal data fusion could also assist in treatment decision or drug response. However, outcomes in real-world patients often lag relative to clinical trials thereby hindering to evaluate efficacies due to lack of follow-up data. Fortunately, efforts are underway to capture treatment response in automated scalable ways using NLP from clinical notes¹⁶⁵. With careful study design, ongoing improvements in data collection and sharing methods, and decreasing cost and/or availability of disease monitoring technologies, DL algorithms present a promising choice to further accelerate the field of precision oncology in this direction.