Artificial Intelligence for Multimodal Data Integration in Oncology

Abstract

In oncology, the patient state is characterized by a whole spectrum of modalities, ranging from radiology, histology, genomics to electronic-health records, each one providing additional insights. Current AI models, however, operate mainly in the realm of single modality, neglecting the broader clinical context, which inevitably diminishes their potential. Integration of different data modalities provides opportunities to increase robustness and accuracy of diagnostic and prognostic models, bringing AI closer to clinical practice. At the same time, AI models are capable of discovering novel patterns within and across modalities suitable for explaining differences in patient outcomes or treatment resistance. The insights gleaned from such models can guide exploration studies and contribute to the discovery of novel biomarkers and therapeutic targets. To support these advances, here we present a synopsis of AI methods and strategies for multimodal data fusion. We outline approaches for AI interpretability and directions for AI-driven exploration through multimodal data interconnections. We examine challenges towards clinical adoption and discuss possible emerging solutions.

1. Introduction

Cancer is a highly complex disease involving a cascade of microscopic and macroscopic changes with mechanisms and interactions that are not yet fully understood. Cancer biomarkers provide insights on the state and course of disease in the form of quantitative or qualitative measurements, which consequently guide patient management. Based on their primary use, biomarkers can be diagnostic, predictive or prognostic. Diagnostic biomarkers stand at the first line of cancer detection and diagnosis, including examples such as prostate-specific antigen (PSA) values or neoplastic changes in tissue biopsy. Predictive biomarkers determine cancer aggressiveness (e.g. grade) or predict treatment response. For instance, microsatellite instability predicts response to immune checkpoint-inhibitor therapy in colorectal cancer (Marcus et al., 2019), whereas KRAS-mutations indicate resistance to anti-EGFR treatment (Van Cutsem et al., 2009). Prognostic biomarkers forecast risks associated with clinical outcomes such as survival, recurrence or disease progression. For example, Oncotype-DX assays are used to estimate recurrence scores and survival likelihood in breast cancer (Paik et al., 2004). Despite the vital role of biomarkers, even patients with similar profiles can be met with diverse treatment responses (Shergalis et al., 2018), recurrence rates (Roy et al., 2015) or treatment toxicity (Kennedy and Salama, 2020), while the underlying reasons for such dichotomies are often unknown. Given the increasing incidence rates of cancer, there is a crucial need to identify novel and more specific biomarkers.

The biomarker discovery process typically involves manual examination of potentially informative measurements and their association with clinical endpoints. For instance, the Nottingham grading system in breast cancer was determined through dedicated examination of thousands of histopathology slides, revealing association between morphological features and patient outcome. Although identification of each new biomarker represents a milestone in oncology, this process faces several challenges. Manual assessment is time- and resource-intensive, often without the possibility of translating observations from one cancer model to another. Cancer assessment is often qualitative with substantial inter-rater variability, which hinders reproducibility and contributes to inconsistent outcomes in clinical trials. Given the large complexity of medical data, current biomarkers are almost exclusively unimodal. However, constraining the biomarkers to a single modality can significantly reduce their clinical potential. For instance, glioma patients with similar genetic or histology profile can be met with diverse outcomes caused by macroscopic factors such as tumor location preventing full resection and irradiation or disruption of blood-brain barrier altering the efficacy of drug delivery (Miller, 2002).

Over the last years, AI has demonstrated great performance in many clinically relevant tasks. AI models are able to integrate complementary information and clinical context from diverse data sources to provide more accurate patient predictions (Fig. 1A). The clinical insights identified by successful models can be further elucidated through interpretability methods and quantitative analysis to guide and accelerate the discovery of new biomarkers (Fig. 1C–D). Similarly, AI models can reveal association across diverse modalities, such as relation between certain mutations and specific changes in cellular morphology (Coudray et al., 2018) or association between radiology findings and histology-specific tumor subtypes (Ferreira-Junior et al., 2020; Hyun et al., 2019) or molecular features (Yan et al., 2021) (Fig. 1B). Such associations can identify accessible or non-invasive alternatives for existing biomarkers to support large scale population screenings or selection of patients for clinical trials (Fig. 1 E,F).

Figure 1. AI-driven multimodal data integration.

A) AI models can integrate complemenatary information and clinical context from diverse data sources to provide more accurate outcome predictions. The clinical insights identified by such models can be further elucidated through C) interpretability methods and D) quantitative analysis to guide and accelerate the discovery of new biomarkers or therapeutic targets (E-F). B) AI can reveal novel multimodal interconnestions, such as relation between certain mutations and changes in cellular morphology or associations between radiology findings and histology tumor subtypes or molecular features. Such associations can serve as non-invasive or cost-efficient alternatives to existing biomarkers to support large-scale patient screening (E-F)

2. AI Methods in Oncology

AI methods can be categorized as supervised, weakly-supervised and unsupervised. To highlight the concepts specific to each strategy we present all methods in the framework of computer vision, as illustrated in Figure 2.

Figure 2. Overview of AI methods.

A) Supervised methods use strong supervision where each data point (e.g. feature or image patch) is assigned with a label. B) Weakly-supervised methods allow to train model with weak, patient-level labels, avoiding the need for manual annotations. C) Unsupervised methods explore patterns, subgroups and structures in unlabelled data. For comparison, all methods are illustrated on binary cancer detection task.

Supervised methods

Supervised methods learn a mapping from input data to predefined labels (e.g. cancer/non-cancer) using training samples. This includes hand-crafted and deep-learning (DL) methods.

Hand-crafted methods take as input a set of predefined features (e.g. cell shape or size) extracted from the data before the training, not the data itself. The training is performed with standard machine learning (ML) models, such as random forest (RF), support-vector machine (SVM) or multi-layer perceptron (MLP) (Bertsimas and Wiberg, 2020) (Fig. 2). Since the feature extraction is not part of the learning process, the models typically have simpler architecture, lower computation cost, and may require less training data than DL models. Additional benefit is a high level of interpretability since the predictive features can be related to the data. On the other hand, the feature extraction is time consuming and can translate human bias to the models. Moreover, human perception cannot be easily captured by a set of computer rules, often leading to simpler features. Since the features are usually tailored to the specific disease, the models cannot be easily translated to other tasks or malignancies. Despite the popularity of DL methods, in many applications the hand-crafted methods are sufficient and preferred due to their simplicity and ability to learn from smaller datasets.

Deep-learning methods operate directly on raw data or in the case of large histopathology scans on image patches. Here we focus on convolutional neural networks (CNNs), the most common DL strategy for image analysis. In CNNs the predictive features are not defined, and the model alone must determine which concepts and features are useful for explaining relations between inputs and outputs, using supervision from pixel-level labels. For instance, in Figure 2 each training whole-slide image (WSI) is manually annotated to outline the tumor region. The WSI is then partitioned into rectangular patches and each patch is assigned with a label “cancer” or “no-cancer” determined by the tumor annotation. The majority of CNNs have a similar architecture, consisting of alternating convolutional, pooling and non-linear activation layers, followed by a small number of fully-connected layers. A convolution layer serves as a feature extractor, while the subsequent pooling layer condenses the features into the most relevant ones. The non-linear activation function allows the model to explore complex relations across features. Fully-connected layers then perform the end-task such as cancer classification. The main strength of CNNs is their ability to extract rich feature representations from raw data, resulting in lower pre-processing cost, higher flexibility and often superior performance over hand-crafted models. The potential limitations come from the model’s reliance on pixel-level annotations which are time-intensive and might be affected by inter-rater variability and human bias. Moreover, predictive regions for many clinical outcomes, such as survival or treatment resistance, may be unknown. CNNs are also often criticized for the lack of interpretability, since neither the model’s decision mechanisms nor the learned predictive features can (yet) be satisfyingly interpreted at the human-level. Despite these limitations, CNNs come with impressive performance, contributing to widespread usage in many medical applications.

Weakly-supervised methods

Supervised methods operate under strong supervision, where each data point (e.g. patch or pixel) is assigned a label. Such strong labels, however, are not routinely created during the clinical workflow and their construction is expensive. Weakly-supervised methods allow to train models with weak, patient-level labels (such as diagnosis or survival), avoiding the need for manual data annotations. The most common weakly-supervised methods include graph convolutional networks (GCNs), multiple-instance learning (MIL) and Vision Transformers (VITs).

Graph Convolutional Networks:

Graphs can be used to explicitly capture spatial tissue structures and encode relation between objects. A graph is defined by nodes connected by edges. In histology, a node can represent a cell, image-patch or even tissue region. Edges encode spatial relation and interaction between nodes (Zhang et al., 2019). The graph, combined with the patient-level labels, is processed by a GCN (Ahmedt-Aristizabal et al., 2021) which can be seen as a generalization of CNNs that operate on unstructured graphs. In GCNs, feature representations of a node are updated by aggregating information from neighboring nodes. The updated representations then serve as input for the final classifier. In contrast to CNNs and MIL, GCNs incorporate larger context and spatial tissue structure. This can beneficial in tasks where the clinical context span beyond the scope of a single patch (e.g. Gleason score). On the other hand, the interdependence of the nodes in GCNs come with higher training costs and memory-requirements, since the nodes cannot be processed independently. As a consequence, contemporary GCNs are relatively shallow and typically build only on sub-samples of the WSI.

In Multiple-instance learning (Carbonneau et al., 2018; Cheplygina et al., 2019), all patches from a given patient are grouped into a “bag” carrying a patient-level label. The labels of individual patches (referred to as instances) are not known. The label of a bag is assumed positive if there is at least one positive instance in the bag. The goal of the model is to predict the bag label and to identify which instances are relevant for the given prediction. The MIL models comprise of three main modules: feature extraction, aggregation, and prediction. The first module is used to embed the image patches into lower-dimensional embeddings to reduce memory requirements and facilitate fast training. This is often achieved through transfer learning (Weiss et al., 2016) and self-supervision (see next section). The patch-level embeddings are aggregated to create patient-level representations which serve as input for the final classification module. There exist several aggregation strategies. For instance, Fig. 2 shows attention-based pooling (Ilse et al., 2018) where two fully-connected networks are used to learn the relative importance of each image patch toward the model predictions – the so-called attention score. The patch-level representations, weighted by the corresponding attention score, are summed up to build the patient-level representation. The attention-scores can be also used for model interpretability (see Section 4).

Vision transformers (Dosovitskiy et al., 2020; Vaswani et al., 2017) are type of attention-based learning. In contrast to MIL, where patches are assumed independent and identically distributed, VITs account for correlation and context among patches. The main components of VITs include positional encoding, self-attention and multi-head self-attention. Positional encoding learns the spatial structure of the image and the relative distance among patches. Self-attention mechanism determines the relevance of each patch while also accounting for the context and contribution from the other patches. Multi-head self-attentions simultaneously deploys multiple self-attention blocks to account for different types of interactions between the patches and combines them into a single self-attention output. A typical VIT architecture is shown in Fig. 2. A WSI is converted into a series of patches, each coupled with positional information. Learnable encoders map each patch and its position into a single embedding vector, referred to as a token. Additional token is introduced for the classification task. The class token together with the patch tokens are fed into the transformer encoder to compute multihead self-attention and output the learnable embeddings of patches and the class. The output class token serves as a slide-level representations used for the final classification. The transformer encoder consists of several stacked identical blocks. Each block includes multi-head self-attention and MLP, along with layer-normalization and residual connections. The positional encoding and multiple self-attention heads allows to incorporate spatial information, increase the context and robustness (Li et al., 2022; Shamshad et al., 2022) of VIT methods over MIL. On the other hand, VIT tend to be more data hungry (Dosovitskiy et al., 2020).

Often it is difficult to know a priori which model is most suitable for the given task. While GCNs methods tend to perform better in tasks that require larger spatial context, such as Gleason grade, Bulten et al. (Bulten et al., 2022) showed that MIL can predict the Gleason score with clinical-grade accuracy even without the graph-based models. Without proper benchmarks offering head-to-head comparison of methods on the same data and tasks, it is difficult to make strong conclusions.

Weakly-supervised methods offer several benefits. The liberation from manual annotations reduces the cost of data pre-processing and mitigates the bias and inter-rater variability. Consequently, the models can be easily applied to large datasets, diverse tasks and also situations where the predictive regions are unknown. Since the models are free to learn from the entire scan, they can identify predictive features even beyond the regions typically evaluated by pathologists. The great performance demonstrated by weakly-supervised methods suggests that many tasks can be addressed without expensive manual annotations or hand-crafted features.

Unsupervised methods

Unsupervised methods explore structures, patterns and subgroups in data without relying on any labels. This includes self-supervised and fully-unsupervised strategies.

Self-supervised methods are designed to improve performance of fully/weakly-supervised models, where limited data are available and overfitting might be of concern. Self-supervised methods exploit available un-labeled data to learn high-quality image features and then transfer this knowledge to supervised models. To achieve this, supervised methods such as CNNs are used to solve various pretext tasks (Jing and Tian, 2019) for which the labels are generated automatically from the data. For instance, a patch can be removed from an image and a network is trained to predict the missing part of the image from its surroundings, using the actual patch as label. The patch-prediction has no direct clinical relevance, but it guides the model to learn general-purpose features of image characteristics which can be beneficial for other practical tasks. The early layers of the network usually capture general image features, while the later layers pick features relevant for the pretext task. The later layers are thus excluded, while the early layers serve as feature extractors in the fully/weakly supervised models (i.e. transfer learning).

Fully-unsupervised methods search for correlations, structures, and subgroups in the data itself, without having a prescribed task-specific outcome or label. The most common unsupervised methods include clustering and dimensionality reduction. Clustering methods (Rokach and Maimon, 2005) partition data into subgroups such that the similarity within the subgroup and separation between subgroups are maximized. Although the output clusters are not task-specific, they can reveal different cancer subtypes or patient subgroups. The aim of dimensionality reduction is to obtain low-dimensional representation capturing the main characteristics and correlations in the data. Common DL approach include autoencoders. An autoencoder (Hinton and Salakhutdinov, 2006) consists of two neural networks: encoder and decoder. The encoder learns to encode data into a lower-dimensional feature vector similar to feature extraction in CNNs. At the same time, a decoder is trained to reconstruct the original input data from the lower-dimensional vector. Along this process, the encoder learns to identify the most relevant aspects of the data. The encoder can be used as the feature extractor in supervised methods (Sharmay et al., 2021), similar to the encoder obtained through self-supervised learning. There is no guarantee that an unsupervised model will result in a desired output; rather, these models serve as tools for data exploration and feature learning.

3. Multimodal data fusion

The aim of multimodal data fusion is to extract and combine complementary, contextual information across different modalities for better decision making. This is of particular relevance in medicine where similar findings in one modality may have diverse interpretations in combination with other modalities. For instance, mass in lung X-ray scans can be interpreted as pneumonia, COVID-19, or also tumor – depending on accompanying symptoms such as fever, elevated blood count, or biopsy assessment. Similarly for cancer biomarkers, IDH1 mutation status or histology profile alone are insufficient for explaining the variance in patient outcomes, whereas the combination of both have been recently used to redefine the WHO classification of diffuse glioma (Louis et al., 2016). AI offers automated and objective way for incorporating complementary information and clinical context from diverse data for improved predictions. The models can also utilize supplementary information in modalities; if unimodal data are noisy or incomplete, supplementing redundant information from other modalities can improve the robustness and accuracy of the predictions. AI-driven data fusion strategies (Baltrušaitis et al., 2018) can be divided as early, late, and intermediate (see Fig. 3).

Figure 3. Multimodal data fusion.

A.) Early fusion builds a joint representation from raw data or features at the input level, before feeding it to the model. B.) Late fusion trains a separate model for each modality and aggregates the predictions from individual models at the decision level. (C-E) In intermediate fusion, the prediction loss is propagated back to the feature extraction layer of each modality to iteratively learn improved feature representations under the multimodal context. The unimodal data can be fused at C.) single-level or D.) gradually in different layers. E.) Information from one modality can be also used to guide feature extraction from another modality. F.) Legend of the used symbols.

Early fusion integrates information from all modalities at the input level before feeding it into a single model. The modalities can be represented as raw data, hand-crafted or deep-features. The joint representation is build through operations such as vector concatenation, element-wise sum, element-wise multiplication (Hadamard Product) or bilinear pooling (Kronecker Product) (Huang et al., 2020a; Ramachandram and Taylor, 2017). In early fusion, only one model is trained which simplify the design process. However, it is assumed that the single model is well suited for all modalities. Early fusion requires a certain level of alignment or synchronization between the modalities. Although this is more obvious in other domains, such as synchronization of audio and visual signals in speech recognition, it is also relevant in clinical settings. If the modalities come from significantly different time points, such as pre- and post-interventions, then early fusion might not be appropriate choice.

Applications of early-fusion include integration of similar modalities such as multimodal, multiview ultrasound images for breast cancer detection (Qian et al., 2021) or fusion of structural CT and/or MRI data with metabolic PET scans for cancer detection (Le et al., 2017), treatment planning (Lipková et al., 2019), or survival prediction (Nie et al., 2019). Other examples include fusion of imaging data with electronic medical records (EMRs) such as integration of dermoscopic images and patient data for skin lesion classification (Yap et al., 2018), or fusion of cervigram and EMRs for cervical dysplasia diagnosis (Xu et al., 2016). Several studies investigate the correlation between changes in gene expression and tissue morphology, integrating genomics data with histology and/or radiology images for cancer classification (Khosravi et al., 2021), survival (Chen et al., 2020b, 2021c) and treatment response (Feng et al., 2022; Sammut et al., 2022) prediction.

Late fusion, also known as decision-level fusion, trains a separate model for each modality and aggregates the prediction from individual models for the final prediction. The aggregation can be performed by averaging, majority voting, Bayes-based rules (Ramanathan et al., 2022), or learned models such as MLP. Late fusion allows to use different model architecture for each modality and does not pose any constrains on data synchronization, making it suitable for systems with large data heterogeneity or modalities from different time points. In case of missing or incomplete data, late fusion retains the ability to make predictions since each model is trained separately and aggregations, such as majority voting, can be applied even if a modality is missing. Similarly, inclusion of a new modality can be performed without the need to retrain the full model. Simple covariates, such as age or gender, are often included through late fusion due to its simplicity (see Fig. 3 B). If the unimodal data do not complement each other or do not have strong interdependencies, late fusion might be preferable thanks to the simpler architecture and smaller number of parameters compared to other fusion strategies. This is also beneficial in situations with limited data. Furthermore, errors from individual models tend to be uncorrelated, resulting in potentially lower bias and variance in late-fusion predictions. In situations when information density varies significantly across modalities, predictions from shared representations can be heavily influenced by the most dominant modality. In late fusion the contribution from each modality can be accounted for in a controlled manner by setting equal or diverse weights per modality in the aggregation step.

Examples of late fusion include integration of imaging data with non-imaging inputs, such as fusion of MRI scans and PSA-blood tests for prostate cancer diagnosis (Reda et al., 2018), integration of histology scans and patient gender for inferring origin of metastatic tumors (Lu et al., 2021), fusion of genomics and histology profiles for survival prediction (Chen et al., 2021c; Shao et al., 2019), combination of pre-treatment MRI or CT scans with EMRs for chemotherapy response prediction (Joo et al., 2021) and survival estimation (Nie et al., 2016).

Intermediate fusion is a strategy where the loss from the multimodal model propagates back to the feature extraction layer of each modality to iteratively improve feature representations under the multimodal context. For comparison, in early and late fusion the unimodal embeddings are not affected by the multimodal information. Intermediate fusion can combine individual modalities at different levels of abstractions. Moreover, in systems with three and more modalities the data can be fused either all at once (Fig. 3C) or gradually across different levels (Fig. 3D). The intermediate single-level fusion is similar to early fusion, however, in the early fusion the unimodal embeddings are not affected by the multimodal context. Gradual fusion allows to combine data from highly correlated channels at the same level, forcing the model to consider the cross-correlations between specific modalities, followed by fusion with less correlated data in later layers. For instance, in Fig. 3D, genomics and histology data are fused first, to account for the interplay between mutations and changes in the tissue morphology, while the relation with the macroscopic radiology data is considered in the later layer. Gradual fusion has shown improved performance over single-level fusion in some applications (Joze et al., 2020; Karpathy et al., 2014). Guided fusion uses information form one modality to guide feature extraction from another modality. For instance, in Fig. 2E, genomics information guides the selection of histology features. The motivation is that different tissue regions might be relevant in the presence of specific mutations. Guided fusion learns co-attention scores that reflect relevance of different histology features in the presence of specific molecular information. The co-attention scores are learned with the multimodal model, where the genomics feature and the corresponding genomics-guided histology features are combined for the final model predictions.

Examples of intermediate fusion include integration of diverse imaging modalities such as fusion of PET and CT scans in lung cancer detection(Kumar et al., 2019), fusion of MRI and ultrasound images in prostate cancer classification (Sedghi et al., 2020), or combination of multimodel MRI scan in glioma segmentation (Havaei et al., 2016). Fusion of diverse multi-omics data was used for cancer subtyping (Liang et al., 2014) or survival prediction (Lai et al., 2020). Genomics data have been used in tandem with histology (Vale-Silva and Rohr, 2021) or mammogram (Yala et al., 2019) images for improved survival prediction. Guided fusion of different radiology modalities was used to improve segmentation of liver lesions (Mo et al., 2020) and anomalies in breast tissue (Lei et al., 2020). EMRs were used to guide feature extraction from dermoscopic (Zhou and Luo, 2021) and mamography (Vo et al., 2021) images to improve detection and classification of lesions. Chen et al. (Chen et al., 2021b) used genomics information to guide selection of histology features for improved survival prediction in multiple cancer types.

There is no conclusive evidence that one fusion type is ultimately better than the others, as each type is heavily data- and task-specific.

4. Multimodal Interpretability

Interpretability is a crucial component of AI development, deployment and validation. With the ability of AI models to discover their own features, there is a worry that the models might use spurious shortcuts for predictions, instead of learning clinically relevant aspects. Such models might fail to generalize when presented with new data or discriminate against underrepresented populations (Banerjee et al., 2021; Chen et al., 2021a). On the other hand, the models can discover novel and clinical relevant insights. Here we present a brief overview of different levels of interpretability used in oncology (Fig. 4), while technical details can be find in the recent review (Arrieta et al., 2020).

Figure 4. Interpretability methods.

Histology: MIL model was trained to classify subtypes of renal-cell carcinoma in WSIs, while CNN was trained to perform the same task in image patches. A) Attention-heatmaps and patches with the lowest/highest attention score. B.) GradCAM attributions for each class. Radiology: MIL model was trained to predict survival from MRI scans using axial slides as individual instances. D.) Attention-heatmaps mapped into the 3D MRI scan and slides with the highest/lowest attention. F.) GradCAM was used to obtain pixel-level interpretability in each MRI slide. A 3D pixel-level interpretability is computed by weighting the slide-level GradCAM maps by the attention score of the respective slide. Integrated gradient attributions can be used to analyze C.) Genomics or E.) EMRs. The attribution magnitude corresponds to the importance of each feature and direction indicates feature impact towards low (left) vs. high (right) risk. The color specifies the value of the input features: copy number gain/presence of mutation are shown in red, while blue is used for copy number loss/wild-type status. E.) Attention scores can be used to analyse the importance of words in the medical text.

In histopathology, VIT or MIL can reveal relative importance of each image patch for the model predictions. The attention scores can be mapped to their spatial location to obtain slide-level attention heatmaps as shown in Fig. 4A, where a MIL model was trained to classify cancer suptyes in WSIs. Although no manual annotations were used, the model learned to identify morphology specific for each cancer type and to discriminate between normal and malignant tissues. Class activation methods (CAMs), such as GradCAM (Selvaraju et al., 2017) or GradCAM++ (Chattopadhay et al., 2018), allow to determine importance of the model inputs (e.g. pixels) by computing how the changes in the inputs affect the model outputs for each prediction class. GradCAM is often used in tandem with the guided-backpropagation method, so-called Guided-GradCAM (Selvaraju et al., 2016), where the guided-backpropagation determines the pixel-level importance inside the predictive regions specified by the GradCAM. This is illustrated in Fig. 4B, where a CNN was trained to classify cancer subtypes in image patches. For comparison, in the attention methods the importance of each instance is determine during the training while the CAM-based methods are model-agnostic i.e. independent on the model training.

In radiology, the interpretability methods are similar to those used in histology. The attention-scores can reflect the importance of slides in a 3D scan. For instance, in Fig. 4D. MIL model was trained to predict survival in glioma patients (Zhuang et al., 2022). The model considered the 3D MRI scan as a bag, where the axial slides are modeled as individual instances. Even in the absence of manual annotations, the model has placed high-attention to the slides with tumor, while low-attention was assigned to healthy tissue. CAM-based methods can be consequently deployed to localize the predictive regions within individual slides (Fig. 4F).

Molecular data can be analyzed by Integrated Gradient method (Sundararajan et al., 2017) which computes attribution values indicating how changes in specific inputs impact the model outputs. For the regression tasks, such as survival analysis, the attribution values can reflect the magnitude of the importance as well as the direction of the impact: features with positive attribution increase the predicted output (i.e. higher risk) while features with negative attribution reduce the predictive values (i.e. lower risk). At the patient-level, these is visualized as a bar plot, where the y-axis corresponds to the specific features (ordered by their absolute attribution value) and the x-axis shows the corresponding attribution values. At the population-level, the attributions plots depict the distribution of the attributions scores across all subjects. Fig. 4C shows the attribution plots for most genomics features used for survival prediction in glioma patients (Chen et al., 2021c). Other tabular data, such as hand-crafted features or values obtained from EMR can be interpreted in the same way. EMRs can be also analyzed by natural language processing (NLP) methods, such as Transformers, where the attention scores determine importance of specific words in the text (Fig. 4E.)

In multimodal models, the attribution plots can also determine contribution of each modality towards the model predictions. All previously mentioned methods can be used in multimodal models to explore interpretability within each modality. Moreover, shifts in feature importance under unimodal and multimodal settings can be investigated to analyze impact of the multimodal context.

The interpretability methods usually come without any accuracy measures and thus it is important not to over-interpret them. While CAM- or attention-based methods can localize the predictive regions they cannot specify which features are relevant, i.e. they can explain where but not why. Moreover, there is no guarantee that all high-attention/attribution region carry clinical relevance. High scores just mean that the model has considered these regions more important than others.

5. Multimodal data interconnection

The aim of multimodal data interconnection is to reveal associations and shared information across modalities. Such associations can provide new insights on cancer biology and guide discovery of novel biomarkers. Although there are many approaches for data explorations, here we illustrate few possible directions (Fig. 5).

Figure 5. Multimodal data interconnection.

A.) AI can identify associations across modalities, such as feasibility of inferring certain mutations from histology or radiology images or B.) relation between non-invasive and invasive modalities, such as prediction of histology subtype from radiomics features. C.) The models can uncover associations between clinical data and patient outcome, contributing to discovery of predictive features within and across modalities. D.)Information acquired by EMRs or wearable devices can be analyzed to identify risk factors related with cancer onset or uncover patterns related with treatment response or resistance, to support early interventions.

Morphological associates:

Malignant changes often propagate across different scales; oncogenic mutations can affect cell behaviour, which in turn reshape tissue morphology or tumor microenviroment visible in histology images. Consequently, the microscopic changes might impact tumor metabolic activity and macroscopic appearance detectable by PET or MRI scans. The feasibility of AI methods to identify associations across modalities was first demonstrated by Coudray et al. (Coudray et al., 2018) who showed that certain mutations in lung cancer can be inferred directly from hematoxylin and eosin (H&E)-stained WSIs. Other studies have followed shortly, predicting the mutation status from WSIs in liver (Chen et al., 2020a), bladder (Loeffler et al., 2021), colorectal (Jang et al., 2020), thyroid cancer (Tsou and Wu, 2019), as well as pan-cancer pan-mutation studies attempting to predict any genetic alternation in any tumor type (Fu et al., 2020; Kather et al., 2020). Additional molecular biomarkers, such as gene expression (Anand et al., 2020; Binder et al., 2021; Schmauch et al., 2020), hormone-receptor status (Naik et al., 2020), tumor mutational burden (Jain and Massoud, 2020), and microsatellite instability (Cao et al., 2020; Echle et al., 2020) have also been inferred from WSIs (Murchan et al., 2021). In radiology, AI models have predicted IDH mutation and 1p/19q co-deletion status from preoperative brain MRI scans (Bangalore Yogananda et al., 2020; Yogananda et al., 2020), BRCA1 and BRCA2 mutational status from breast mammography (Ha et al., 2017) and MRI (Vasileiou et al., 2020) scans, while EGFR and KRAS-mutation have been detected from CT scans in lung (Wang et al., 2019) and colorectal (He et al., 2020) cancer.

By discovering presence of morphological associates across modalities, AI models can enhance exploratory studies and reduce the search space for possible biomarker candidates. For instance, in Fig. 5A, the AI has revealed that one of the studied mutations can be reliably inferred from WSI. Although the predictive features used by the model might be unknown, interpretability methods can provide additional insights. Attention-heatmaps can reveal tissue regions relevant for the prediction of the specific mutation. Distinct tissue structures and cell types withing high and low-attention regions can be identified, and their properties such as nuclei shape or volume can be further extracted and analyzed. Clustering or dimensionality reduction methods can be deployed to examine the promising features, potentially revealing associations between mutation status and distinct morphological features. The identified morphological associates can serve as cost-efficient biomarker surrogates to support screening in low-to-middle income settings or reveal new therapeutic targets.

Non-invasive alternatives:

Similarly, AI can discover relationships between non-invasive and invasive modalities. For instance, AI models were used to predict histology sutypes or grades from radiomics features in lung (Sha et al., 2019), brain (Lasocki et al., 2015), liver (Brancato et al., 2022) and other cancers (Blüthgen et al., 2021). The predictive image regions can be further analyzed to identify textures and patterns with possible diagnostic values (see Fig. 5B), which in turn can serve as non-invasive surrogates for existing biomarkers.

Outcome associates:

Benefits of personalized medicine are often limited by paucity of biomarkers able to explain dichotomies in patient outcomes. On the other hand, AI models are demonstrating great performance in predicting clinical outcomes, such as survival (Lai et al., 2020), treatment response (Echle et al., 2020), recurrence (Yamamoto et al., 2019), and radiation toxicity (Men et al., 2019) using unimodal and multimodal (Chen et al., 2020b, 2021c; Joo et al., 2021; Mobadersany et al., 2018) data. These works imply the feasibility of AI models to discover relevant prognostic patterns in data, which might be elucidated by interpretability methods. For instance in Fig. 5C, a model is trained to predict survival from histology and genomics data. Attention heatmaps reveal tissue regions related to low and high risk patients groups while the molecular profiles are analyzed through attribution plots. The predictive tissue regions can be further analyzed by examining tissue morphology, cell subtypes or other human-intepretable data characteristics. Tumor-infiltrating lymphocytes can be estimated through co-localization of tumor and immune cells to specify immune hot and cold tumors. Attribution of specific modalities as well as shift in feature importance in unimodal vs. multimodal data can be explored to determine the influence of multimodal contextualization.

Such exploration studies have already provided new clinical insights. For instance, Geessink et al. (Geessink et al., 2019) showed that tumor-stroma ratio can serve as independent prognosticator in rectal cancer, while the ratio of tumor area to metastatic lymph node regions has prognostic value in gastric cancer (Wang et al., 2021). Other morphological features, such as arrangement of collagen fibers in breast histology (Li et al., 2021) or spatial tissue organization in colorectal tissue (Qi et al., 2021) have been identified as possible biomarkers for aggressiveness or recurrence.

Early predictors:

AI can also explore diverse data acquired prior to patient diagnosis to identify potential predictive risk factors. EMRs provide rich information on patient history, medication, allergies or immunization, which might contribute to patient outcome. Such diverse data can be efficiently analyzed by AI models to search for distinct patient subgroups (Fig. 5 D). Identified subgroups can be correlated with different patient outcomes, while attribution plots can identify relevance of different factors at the patient and population level. Recently, Placido et al. (Placido et al., 2021) showed the feasibility of AI to identify patients with a higher risk of developing pancreatic cancer by exploration of EMR. Similarly EMRs were used to predict treatment response prediction (Chu et al., 2020) or length of hospital stay (Alsinglawi et al., 2022). The identified novel predictive risk factors can support large-scale population screenings and early preventive care.

Outside of the hospital setting, smartphones and wearable devices offer another great opportunity for real-time and continuous patient monitoring. Changes in the measured values, such as decrease in patient step counts, have been shown as robust predictors of worse clinical outcome, treatment toxicity, and increased risk of hospitalization (Low, 2020). Furthermore, the modern wearable devices are continually expanding their functionality including measurements of temperature, stress levels, blood-oxygen saturation, or electrocardiograms. These measurements can be analyzed in tandem with clinical data to search for risk factors indicating early stages of increased toxicity or treatment resistance, to allow personalized interventions during the course of treatment. Research on personalized monitoring and nanotechnologies is investigating novel directions such as detection of patient measurements in sweat (Xu et al., 2019) or ingestible sensors to monitor medication compliance and drug absorption (Weeks et al., 2018). All these novel devices provide useful insights on the patient state, which could be analyzed in larger clinical context through AI models.

6. Challenges and path towards clinic

The path of AI into clinical practice is still laden with obstacles, many of which are amplified in the presence of multimodal data. While several recent works discus challenges, such as fairness and datasets shifts (Banerjee et al., 2021; Chen et al., 2021a; Cirillo et al., 2020; Howard et al., 2021; Mehrabi et al., 2021; Zhang et al., 2018), limited interpretability (Adebayo et al., 2018; Linardatos et al., 2020; Reyes et al., 2020) or regulatory guidelines (Cruz Rivera et al., 2020; Topol, 2020; Wu et al., 2021), here we focus on challenges specific to multimodal learning.

Missing data

The challenge of missing data refers to the absence of part of modality or complete unavailability of one or more modalities. The missing data impact both, the model training and deployment since majority of existing AI models cannot handle missing information. Moreover, the need to train models with complete multimodal data significantly constrains the size of the training datasets. For instance, The Cancer Genome Atlas (TCGA) dataset contains over 900 WSIs from glioma patients, however, only 127 cases contain corresponding radiology and genomics information. The incomplete modalities still contain valuable information, and the inability to deploy them pose a significant limitation, especially in fields like medicine where data are precious. Below we discuss two strategies for handling missing data.

Synthetic data generation can be used to complement the missing information. If part of an image is corrupted or if specific mutations are not reported, the missing information can be fabricated from the remaining data. If a whole modality is missing, its synthetic version can be derived from existing similar modalities. For instance, Haan et. al (de Haan et al., 2021) trained a supervised model for translation of H&E-stains into special stains, using the special stains as ground truth labels. The model was trained on pairs of perfectly aligned data obtained e.g through re-staining of the same slides. If paired data are not available, unsupervised methods such as cycle generative adversarial networks (GANs) (Zhu et al., 2017) can be used. While synthetic data can improve performance of detection and classification methods, they are less suitable for outcome prediction or biomarker exploration, where the predictive features are not well understood and thus there is no guarantee that the synthetic data contain the relevant disease characteristics. Moreover, the algorithms can hallucinate malignant features also into the supposedly normal synthetic images (Cohen et al., 2018) which can further hurt prediction results.

Drop-out based methods, aims to make models robust to missing information. For instance, Choi et. al(Choi and Lee, 2019) proposed EmbraceNet model which can handle incomplete or missing data during training and deployment. The EmbraceNet model probabilistically selects partial information from each modality and combines it into a single representation vector, which then serves as an input for the final decision model. When missing or invalid data is encountered, it is not sampled; instead, other more complete modalities are used to compensate for the missing data. The probabilistic data selection has also a regularization effect, similar to the dropout mechanism. The authors have tested the EmbraceNet method on natural images and videos, whereas its utility in clinical settings, although very promising, is yet to be explored.

Data alignment

To investigate cancer processes across different scales and modalities, a certain level of data alignment is required This might include alignment of: i) similar or ii) diverse modalities.

Alignment of similar modalities typically involves different imaging modalities of the same system. This is usually achieved through image registration, which is formulated as an optimization problem minimizing the difference between the modalities.

In radiology, rigid anatomical structures can guide the data alignment. For instance, registration of MRI and PET brain scans is usually achieved with high accuracy, even with simple affine registration, thanks to the rigid skull. The situation is more complex in the presence of motion and deformations, e.g. breathing in lung imaging or changes in the body posture between scanning sessions. Alignment of such data usually require deformable registrations using natural or manually placed landmarks for guidance. A particularly challenging situation is the registration of scans between interventions, e.g. registration of preoperative and postoperative scans, which exhibit lot of non-trivial changes due to tumor resection, response to treatment, or tissue compression (Haskins et al., 2020).

In histology, each stained slide usually come from different tissue cut. Even in consecutive tissue cuts there are substantial differences in the tissue appearance caused by changes in tissue microenviroment or artefacts such as tissue folding, tearing or cutting (Taqi et al., 2018), which all complicate data alignment. Automated registration of histology images is still an open problem (Borovec et al., 2020) and thus many studies deploy non-algorithmic strategies such as clearing and re-staining of the tissue slides (Hinton et al., 2019). A newly emerging direction is stainless imaging, including approaches such as ultraviolet microscopy (Fereidouni et al., 2017), stimulated Raman histology (Hollon et al., 2020) or colorimetric imaging (Balaur et al., 2021).

Alignment of diverse modalities refers to the integration of data from different scales, time points or measurements. Often an acquisition of one modality results in the destruction of the sample, preventing collection of multiple measurements of the same system. For instance, most omics measurements require tissue disintegration which inevitably affects the possibilities to study relations between cell appearance and corresponding gene expressions. Here, cross-modal autoencoders can be used to enable integration and translation between arbitrary modalities. Cross-modal autoencoders (Dai Yang et al., 2021) build a pair of encoder-decoder networks for each modality, where the encoder maps each modality into a lower-dimensional latent space, while the decoder maps it back into the original space. A discriminative objective function is used to ensure that different modalities are matched in the shared latent space. Having the shared latent space, one can combine an encoder of one modality with the decoder of another modality to translate one modality to another one. Yang et al. (Dai Yang et al., 2021) demonstrated translations between single-cell chromatin images and RNA-seq data. The feasibility and utility of the cross-modal autoencoders are yet to be tested with more advanced data. However, if proven potent, they hold great potential to address challenges with alignment and harmonization of data from diverse sources.

Transparency and prospective clinical trials

Given the complexity of AI methods, it is possible that their mechanisms will not be fully understood in the near future. However, many aspects in medicine are not fully understood either (Kirkpatrick, 2005). And thus, rather than dwelling on the full opacity of AI methods, we should advocate for their rigorous validation under randomized clinical trials, same as is done for other medical devices and drugs. Prospective trials will allow to stress-test the models under real-world conditions, compare their performance against standard-of-care practice, estimate how clinicians interact with the AI-tool and find the best way in which the models can enhance, rather than disturb the clinical workflow. In the case of biomarker surrogates discovered by AI methods, regulation paths similar to “me-too” drugs and devices (Aronson and Green, 2020) should be required to ensure comparable-level of performance. Transparency about study design and the used data are necessary to determine the intended use and conditions under which the model performance has been verified to prevent misuse under non-tested conditions. Prospective clinical trials are inevitable to truly demonstrate and quantify the added value of AI models, which will in turn increase trust and motivation of practitioners towards the AI tools.

7. Outlook and Discussion

AI has potential to impact the whole landscape of oncology, ranging from prevention to intervention. AI models can explore complex and diverse data to identify factors related with high risks of developing cancer to support large population screenings and preventive care. The models can further reveal associations across modalities to help identify diagnostic or prognostic biomarkers from easily accessible data to improve patient risk stratification or selection for clinical trials. In a similar way, the models can identify non-invasive alternatives to existing biomarkers to minimize invasive procedures. Prognostic models can predict risk factors or adverse treatment outcomes prior to interventions to guide patient management. Information acquired from personal wearable devices or nanotechnologies could be further analyzed by AI models to search for early signs of treatment toxicity or resistance, with other great application yet to come.

As with any great medical advances, there is a need for rigorous validation and examination under prospective clinical trials to verify the promises made by AI models. The role of AI in advancing the field of oncology is not autonomous, rather it is a partnership between models and human experience that will drive the further progress. AI-models come with limitations and challenges, however, these should not intimidate but rather inspire us. With increasing incidence rates of cancer, it is our obligation to capitalize on benefits offered by AI methods to accelerate discovery and translation of advances into clinic practice to serve patients and health care providers.