Multimodal learning with graphs

Abstract

Artificial intelligence for graphs has achieved remarkable success in modeling complex systems, ranging from dynamic networks in biology to interacting particle systems in physics. However, the increasingly heterogeneous graph datasets call for multimodal methods that can combine different inductive biases—the set of assumptions that algorithms use to make predictions for inputs they have not encountered during training. Learning on multimodal datasets presents fundamental challenges because the inductive biases can vary by data modality and graphs might not be explicitly given in the input. To address these challenges, multimodal graph AI methods combine different modalities while leveraging cross-modal dependencies using graphs. Diverse datasets are combined using graphs and fed into sophisticated multimodal architectures, specified as image-intensive, knowledge-grounded and language-intensive models. Using this categorization, we introduce a blueprint for multimodal graph learning, use it to study existing methods and provide guidelines to design new models.

1. Introduction

Deep learning on graphs has contributed to breakthroughs in biology [1, 2], chemistry [3, 4], physics [5, 6], and the social sciences [7]. The predominant use of graph neural networks [8] is to learn representations of various graph components—such as nodes, edges, subgraphs, and entire graphs—based on neural message passing strategies. The learned representations are used for downstream tasks, including label prediction via semi-supervised learning [9], self-supervised learning [10], and graph design and generation [11, 12]. In most existing applications, datasets explicitly describe graphs in the form of nodes, edges, and additional information representing contextual knowledge, such as node, edge, and graph attributes.

Modeling complex systems requires measurements that describe the same objects from different perspectives, at different scales, or through multiple modalities, such as images, sensor readings, language sequences, and compact mathematical statements. Multimodal learning [13] studies how such heterogeneous, complex descriptors can be optimized to create learning systems that are broadly generalizable, robust to changes in the underlying data distributions, and can train more with less labeled data. While multimodal learning has been successfully used in settings where unimodal methods fail [14, 15, 16], it presents several challenges that must be overcome to enable its broad use in AI [13, 17]. These challenges include finding representations optimized for machine learning analyses and fusing combined information from various modalities to create predictive models [18, 19, 20]. These challenges have proven difficult. For example, multimodal methods tend to focus on only a subset of modalities that are most helpful during model training while ignoring modalities that might be informative for model implementation—a pitfall known as modality collapse [21]. Moreover, in contrast to the frequent assumption that every object must exist in all modalities, the complete set of modalities is rarely available due to limitations of data collection and measurement technologies—a challenge known as missing modalities [22, 23]. Because different modalities can lead to intricate relational dependencies, simple modality fusion cannot fully leverage multimodal datasets [24]. Graph learning models such data systems [25, 26, 27] by connecting data points in different modalities as edges in optimally defined graphs [28, 29, 30] and building learning systems for a wide range of tasks [31, 32].

We introduce a blueprint for multimodal graph learning (MGL). The MGL blueprint provides a framework that can express existing algorithms and help develop new methods for multimodal learning leveraging graphs. This framework allows for learning fused graph representations and studying the aforementioned challenges of modality collapse and missing modalities [18, 13]. We apply this formulation across a broad spectrum of domains, ranging from computer vision and language processing to the natural sciences (Figure 1). We consider image-intensive graphs (IIG) for image and video reasoning (Section 3), language-intensive graphs (LIG) for processing natural and biological sequences (Section 4), and knowledge-intensive graphs (KIG) used to aid in scientific discovery (Section 5).

Fig. 1. Graph-centric multimodal learning.

Shown on the left are the different data modalities. Shown on the right are machine learning tasks for which multimodal graph learning has proved valuable. We introduce the multimodal graph learning (MGL) blueprint that serves as a unifying framework for multimodal graph neural architectures realized through learning systems in computer vision, natural language processing, and natural sciences.

2. Graph Neural Networks for Multimodal Learning

Deep learning has created a wide range of fusion approaches for multimodal learning [33, 34]. For example, recurrent neural network (RNN) and convolutional neural network (CNN) architectures have successfully been combined to fuse sound and image representations in video description problems [35, 36]. More recently, generative models have also proven very accurate for both language-dependent [37] and physics-based multimodal data [38]. Such models are based on an encoder-decoder framework, where in the encoder, the combined architectures are trained simultaneously (each one specialized for a modality), while the decoder aggregates information from individual architectures. When complex relations between modalities produce a network structure, graph neural networks (GNNs, Supplementary Note 1) provide an expressive and flexible strategy to leverage interdependencies in multimodal datasets.

2.1. Blueprint for Graph-Centric Multimodal Learning

The use of GNNs for multimodal learning is attractive because of their flexibility to model interactions both within and across different data types. However, data fusion through graph learning requires the construction of network topology and the application of inference algorithms over graphs. We present a methodology that, given a collection of multimodal input data, yields output representations that are used in downstream tasks. We refer to this methodology as multimodal graph learning (MGL). MGL can be seen as a blueprint consisting of four learning components that are connected in an end-to-end fashion. In Figure 2a,b, we highlight the difference between a conventional combination of unimodal architectures for treating multimodal data and the suggested all-in-one multimodal architecture.

Fig. 2. Overview of multimodal graph learning (MGL) blueprint.

a, A standard approach to multimodal learning involves combining different unimodal architectures, each optimized for a distinct data modality. b, In contrast, an all-in-one multimodal architecture considers inductive biases specialized for each data modality and optimizes model parameters in an end-to-end manner, enabling expressive data fusion. c, The MGL blueprint comprises four components: identifying entities, uncovering topology, propagating information, and mixing representations. These components are grouped into two phases: structure learning and learning on the structure.

The first two components of MGL, identifying entities and uncovering topology, can be grouped as the structure learning (SL) phase (Figure 2c) :

Component 1: Identifying Entities.

The first component identifies relevant entities in various data modalities and projects them into a shared namespace. For example, in precision medicine, the state of a patient might be described by matched pathology slides and clinical notes, giving rise to patient nodes with the combined image and language information. In another example from computer vision (Figure 3), entity identification entails defining superpixels in an image.

Fig. 3. Application of multimodal graph learning blueprint to images.

a, Modality identification for image comprehension where nodes represent aggregated regions of interest, or superpixels, generated by the SLIC segmentation algorithm. b, Topology uncovering for image denoising where image patches (nodes) are connected to other non-local similar patches. c, Topology uncovering in human-object interaction where two graphs are created. A human-centric graph maps body parts to their anatomical neighbors, and an interaction connects body parts relative to the distance to other objects in the image. d, Information propagation in human-object interaction where spatially conditioned graphs modify message passing to incorporate edge features that enforce the relative direction of objects in an image [50].

Component 2: Uncovering Topology.

With the entities of our problem defined, the second component discovers the interactions and interaction types among the nodes across the modalities. Interactions are often explicitly provided, so the graph is given, and this component is responsible for the combination of the already existing graph structure with the rest of modalities (e.g., in Figure 5c, the Uncovering Topology component corresponds to combining protein surface information with the protein structure itself). When the data does not have an a priori network structure, the uncovering topology component explores possible adjacency matrices based on explicit (e.g., spatial and visual characteristics) or implicit (e.g., similarities in representations) features. For the latter case, examples from the natural language processing field consider the construction of graphs from text input that express relations among words (Figure 4b).

Fig. 5. Applications of multimodal graph learning to natural sciences.

a, Information propagation in physical interactions where physics-informed neural message passing is used to update the states of particles in a system due to inter-particle interactions and other forces. b, Information propagation in molecular reasoning where a global attention mechanism is used to model the potential interaction between atoms in two molecules to predict whether two molecules will react. c, Topology uncovering in protein modeling where a multiscale graph representation is used to integrate primary, secondary, and tertiary structures of a protein with higher-level protein motifs summarized in molecular superpixels to represent a protein [26]. This robust topology provides a better prediction on tasks such as protein-ligand binding affinity prediction.

Fig. 4. Application of multimodal graph learning blueprint to language.

a, The different levels of context in text inputs from sentences to documents and the individual units identified at each context level. This is an example of modality identification’s first component of the MGL blueprint. b, The simplified construction of a language-intensive graph from text input, an application of the topology uncovering component of the MGL blueprint. c, and d, visualize examples of learning on LIGs for aspect-based sentiment analysis (ABSA), which aims to assign a sentiment (positive, negative, or neutral) to a sentence with regards to a given aspect. By grouping by relation type from within a sentence (shown in c) or modeling relations between sentences and aspects (shown in d), these methods integrate inductive biases relevant to ABSA and innovate in MGL’s third component, information propagation.

After graphs are specified or adaptively optimized (SL phase in MGL; Figure 2c), various strategies can be used to learn on the graphs. The last two MGL components, known together as the learning on structure (LoS) phase (Figure 2c), capture these strategies.

Component 3: Propagating Information.

The third component employs convolutional or message-passing steps to learn node representations based on graph adjacencies (Supplementary Note 1 for more details on graph convolutions and message passing). In the case of multiple adjacency matrices, methods use independent propagation models or assume a hypergraph formulation that fuses adjacency matrices with a single propagation model.

Component 4: Mixing Representations.

The last component transforms learned node-level representations depending on downstream tasks. The propagation models output representations over the nodes that can be combined and mixed depending on the final representation level (e.g., a graph-level or a subgraph-level label). Popular mixing strategies include simple aggregation operators (e.g., summation or averaging) or more sophisticated functions that incorporate neural network architectures.

Figure 2c shows all MGL components, going from multimodal input data to optimized representations used for downstream tasks. Mathematical formulations are in Box 1 and summaries of multimodal graph learning methods are in Supplementary Note 2.

Box 1: The blueprint for multimodal graph learning.

The blueprint for graph-centric multimodal learning has four components.

1. Identifying Entities: Information from different sources is combined and projected into a shared namespace. Nodes are identified independently as set elements, and no interactions are given yet. Let there be k modalities 𝒞 = {C₁, …,C_k}, where C_i is an information matrix of i-th modality that describes every entity by an information vector. We define Identify_i module for every modality i as:

   $$X_i\leftarrow Identif{y}_i\left({\text{C}}_i\right),$$ (1)

   that maps information of all modalities into the same namespace. If k = 1, we get a reduced unimodal variant of MGL.
2. Uncovering Topology: Let there be data modalities 𝒳 = {X₁, …, X_k}. We define Connect_j modules, j = 1, …, m, to specify connections between entities in 𝒳 based on m distance measures as:

   $$A_j\leftarrow {\mathit{\text{Connect}}}_j(𝒳).$$ (2)

   If X_i is already given as an adjacency matrix, the associated Connect_j modules specify predefined neighborhoods.
3. Propagating Information: Neural messages are exchanged along edges in the adjacency matrices 𝒜 = {A₁, …, A_m} to produce node representations:

   $$H\leftarrow Propagate(𝒳,𝒜).$$ (3)

   When multiple adjacency matrices are given, the Propagate module can specify multiple independent propagation models (Supplementary Note 1) or operate on a combined adjacency matrix.
4. Mixing Representations: Representations are mixed and transformed into latent representations optimized for a downstream task:

   $$Z\leftarrow Mix(H,𝒜).$$ (4)

   The mixing module Mix transforms node representations into final representations of entities Z on which downstream tasks are defined on. Established strategies to mix representations include aggregation operators, such as summation [39], averaging [40], multi-hop aggregation [41], and methods using adjacency information 𝒜.

3. Multimodal Graph Learning for Images

Image-intensive graphs (IIGs) are multimodal graphs where nodes represent visual features and edges represent spatial connections between image features. Structure image learning entails creating IIGs to encode geometric priors relevant to images, such as translational invariance and scale separation [42]. Translational invariance describes how the output of a CNN must not change depending on shifts in the input image and is achieved by convolutional filters with shared weights. In contrast, scale separation specifies how to decompose long-range interactions between features across scales, focusing on localized interactions that can be propagated to coarser scales. For example, pooling layers follow convolution layers in CNNs to achieve scale separation [42]. In addition, GNNs can model long-range dependencies of arbitrary shape that are important for image-related tasks [43] such as image segmentation [44, 45], image restoration [46, 47], or human object interaction [48, 49].

Visual Comprehension

Visual comprehension remains a cornerstone of visual analyses, where multimodal graph learning has proven helpful in classifying, segmenting, and enhancing images. Image classification identifies the set of object categories present in an image [51]. In contrast, image segmentation divides an image into segments and assigns each segment into a category [52, 44, 45]. Finally, image restoration and denoising transform low-quality images into high-resolution counterparts [53]. The information required for these tasks lies in objects, segments, and image patches, as well as in the long-range context surrounding them [52].

IIG construction (corresponding to MGL Components 1 and 2) begins with a segmentation algorithm such as simple linear iterative clustering (SLIC) [54] to identify meaningful regions [55, 56, 44] (Figure 3a). These regions define nodes used to extract feature maps and summary visual features for each region [45, 52], whose attributes are initialized from CNNs like FCN-16 [57] or VGG19 [58]. Moreover, the nodes are connected to their k nearest neighbors in the CNN learned feature space [55, 45, 46, 47] (Figure 3b), to spatially adjacent regions [51, 59, 44, 56], or to an arbitrary number of neighbors based on a previously defined similarity threshold between nodes [47, 56].

Once the SL phase of MGL is completed, propagation models (MGL Component 3) based on graph convolutions [52, 59, 56, 45] and graph attention [60] (GAT) are used to weigh node neighbors in the graph based on learned attention scores [51, 47]. In addition, methods such as graph denoiser networks (GCDNs) [61], internal graph neural networks (IGNNs) [46], and residualGCNs [62, 44] consider edge similarities to indicate the relative distance between image regions.

Visual Reasoning

Visual reasoning goes beyond recognizing visual elements by asking questions about the relationships between entities in images. These relationships can involve humans and objects as in human-object interaction [48] (HOI) or, more broadly, visual, semantic, and numeric entities as in visual question answering [63, 64, 65] (VQA).

In HOI, the MGL methods identify two entities, human body parts (e.g., hands, face, etc.) and objects (e.g., surfboard, bike, etc.) [48, 50], that interact in fully connected [48, 49], bipartite [50, 66], or partially connected topologies [67, 68]. MGL methods for VQA construct a new topology [69] that spans interconnected visual, semantic, and numeric graphs. Entities represent visual objects identified by an extractor, such as Faster R-CNN [70], scene text identified by optical character recognition, and number-type texts. Interactions between these entities are defined based on spatial localization: entities occurring near each other are connected by edges.

To learn about these structures (MGL Component 3), methods distinguish between propagating information between entities of the same type and entities of different types. In HOI, knowledge about entities of the same kind (i.e., intra-class neural messages) is exchanged by following edges and applying transformations defined by a GAT [60], which weighs neural messages by the similarity of latent vectors of nodes. In contrast, information between different entities (i.e., inter-class neural messages) is propagated using a GPNN [48] where the weights are adaptively learned [49]. Models can have multiple channels that reason over entities of the same class and share information across classes. For example, in HOI, relation parsing neural networks [68] use a two-channel model where human and object-centric message passing is performed before mixing these representations for the final prediction (Figure 3c). The same occurs in VQA, where visual, semantic, and numeric channels perform independent message passing before sharing information via visual-semantic aggregation and semantic-numeric aggregation [69, 71]. Other neural architectures can serve as drop-in replacements to graph-based channels [66, 67].

4. Multimodal Graph Learning for Language

With the ability to generate contextual language embeddings, language models have broadly reshaped analyses of natural language [7]. However, beyond words, structure in language exists at the level of sentences (syntax trees, dependency parsing), paragraphs (sentence-to-sentence relations), and documents (paragraph-to-paragraph links) [72]. Transformers, a prevailing class of language models [73], can capture such structure but have strict computational and data requirements. MGL methods mitigate these issues by infusing language structure into models. Specifically, these methods rely on language-intensive graphs (LIGs), explicit or implicit graphs where nodes represent semantic features linked by language dependencies.

Creating Language-Intensive Graphs

At the highest level, a language dataset can be seen as a corpus of documents, then a single document, a group of sentences, a group of mentions, a group of entities, and finally, single words (Figure 4a). Multimodal graph learning can consider these different levels of contextual information by constructing LIGs. The choice of context to include and how to create a LIG to represent this context is task specific. We describe these steps for text classification and relation extraction as these tasks underlie most language analyses.

In text classification, the model is asked to assign a label to a span of text [74] based on the usage and meaning of words (tokens). Graph structure involving words is given by the relative position of words in a document [75, 74] or document cooccurrence [76]. Relation extraction seeks to identify relations between words in a text, a capability important for other language tasks, such as question answering, summarization, and knowledge graph reasoning [77, 78]. To capture sentence meaning, the structure among word entities is based on the underlying dependency tree [79]. Beyond words, other entities are included to capture cross-sentence topology [77, 80] (Figure 4a–b).

Learning on Language-Intensive Graphs

Once a LIG is constructed, a model must be designed to learn on the LIG while incorporating inductive biases relevant to the language task. We illustrate strategies for learning on LIGs using aspect-based sentiment analysis (ABSA) as a downstream language task [81]. ABSA assigns a sentiment (positive, negative) of a text to a word/words or an aspect [81]. Models must reason over syntactic structure and long-range relations between aspects and other words in the text to perform ABSA [82, 83]. To propagate information between distant words, aspect-specific GNNs mask non-aspect words in LIGs for long-range message passing [82]. They also gate or perform element-wise multiplication between latent representations of query and aspect words [84]. To include information about the syntactic structure, GNNs distinguish between the different types of relations in the dependency tree via type-specific message passing [82, 83, 84] (Figure 4c).

The sentiment of neighboring or similar sentences is essential to determine the aspect-based sentiment of the document [81]. Cooperative graph attention networks (CoGAN) incorporate this via the cooperation between two graph-based modeling blocks: the inter- and intra-aspect modeling blocks (Figure 4d) [81]. These blocks capture the relation of sentences to other sentences with the same aspect (intra-aspect) and to neighboring sentences in the document that contain different aspects (inter-aspect). The outputs of the intra- and inter-aspect blocks are mixed in an interaction block, passing through a series of hidden layers. Finally, the intermediate representations between each hidden layer are fused via learned attention weights to create a final sentence representation (MGL Component 4).

5. Multimodal Graph Learning in Natural Sciences

In addition to computer vision and language modeling, graphs are increasingly employed in the natural sciences. We call these graphs knowledge-intensive graphs (KIGs) as they incorporate inductive biases relevant to a specific task or encode scientific knowledge in their structure.

Multimodal Graph Learning in Physics

In particle physics, GNNs have been used to identify progenitor particles causing particle jets, sprays of particles that fly out from high-energy particle collisions [85]. In these graphs, nodes are particles connected to their k-nearest neighbors. After rounds of message passing, aggregated node representations are used to identify progenitor particles [86, 87, 88, 89].

Physics-informed GNNs have emerged as a promising approach for simulating physical systems governed by multiscale processes for which conventional methods fail [90]. A typical goal is to discover hidden physics from available experimental data. GNNs are trained from available experimental data and information obtained by employing the physical laws and are then evaluated at points in the space-time domain. Such physics-informed architectures integrate multimodal data with mathematical models. For example, GNNs can express differential operators of the underlying dynamics as functions on nodes and edges [91]. GNNs can also represent physical interactions between objects, such as particles in a fluid [6], joints in a robot [5], and points in a power grid [92]. Initial node representations describe the initial state of these particles and global constants like gravity [6] with edges indicating relative particle velocity [5]. Message passing updates edge representations first to calculate the effect of relative forces in the system. It then uses the updated edge representation to update node representations and calculate the new state of particles as a result of the forces [93] (Figure 5a). This message-passing strategy advances the MGL’s third component (Section 2) and has also been employed to solve combinatorial algorithms (Bellman-Ford and Prim’s algorithms) [94, 95] and chip floorplanning to design the physical layout of computer chips [96].

Multimodal Graph Learning in Chemistry

In chemistry, MGL methods can predict intra- and inter-molecular properties from the primary molecular structure by performing message passing on molecular graphs of atoms linked by bonds [4, 97, 98, 99, 100, 101]. Present efforts incorporate 3D spatial molecular information in addition to 2D molecular details. When this information is unavailable, the MGL methods [97, 99, 100] consider stereo-chemistry to aggregate neural messages [102] and model molecules as sets of chemical substructures in addition to granular atom representations [103].

Stereoisomers are molecules with the same graph connectivity but different spatial arrangements [102]. Aggregation functions in molecular graphs aggregate the same regardless of the orientation of atoms in three-dimensional space. This can lead to poor performance, as stereoisomers can have different properties [104]. To mitigate this issue, permutation (PERM) and permutation-concatenation (PERM-CAT) aggregation [102] update every atom in a chiral group via a weighted sum of every permutation of its respective chiral group. Though the identity of the neighbors is the same in every permutation, the spatial arrangement varies. By weighing each permutation, PERM and PERM-CAT encode this inductive bias by modifying how information is propagated in the underlying graph (MGL Component 3).

Moreover, MGL can help identify chemical products produced by molecules through reactions [105, 106, 107, 108]. For example, to predict whether two molecules react, QM-GNN [105], a quantum chemistry-augmented GNN represents each reactant by its molecular graph with chemistry-informed initial representations for every atom and bond. After rounds of message passing, the atom representations are updated through a global attention mechanism (Figure 5b). The attention mechanism uncovers a novel topology where atoms can interact with atoms on other molecules. It incorporates a principle from chemistry that intermolecular interactions between particles inform reactivity. The final representations are combined with descriptors, such as atomic charges and bond lengths, and used for prediction. Such an approach integrates structural knowledge about molecules in a GNN with relevant chemistry knowledge, allowing for accurate prediction on small training datasets [105]. The inclusion of domain knowledge by fusing GNN outputs illustrates the Mix module in MGL (Section 2, Box 1). Graph learning on molecules created new opportunities for virtual drug screening [109], molecule generation and design [110, 111, 27], and drug target identification [112, 113].

Multimodal Graph Learning in Biology

Beyond individual molecules, MGL can help understand the properties of complex structures across multiple scales, the most pertinent of these structures being proteins. At the primary amino acid sequence scale, the hallmark task predicts the 3D structure from the amino acid sequence. AlphaFold constructs a KIG where nodes are amino acids with representations derived from sequence homology [25]. To propagate information in this KIG, AlphaFold introduces a triangle multiplicative update and triangle self-attention update. These triangle modifications integrate the inductive bias that learned representations must abide by the triangle inequality on distances to represent 3D structures. Multimodal graph learning, among other innovations, enabled AlphaFold to predict 3D protein structure from amino acid sequence [25].

Beyond 3D structure, molecular protein surfaces mediate critical roles in cellular function and disease, and thus modeling geometric and physical protein properties is essential [1, 114, 115]. For example, MaSIF [114] trains a GNN on molecular surfaces described as multimodal graphs to predict protein interactions. The initial representation of the nodes is based on geometric and chemical features. Next, Gaussian kernels are defined on every node to propagate information, encoding complex geometric shapes of molecular surfaces and extending the notion of a convolution. The final representations can be used to predict protein-protein interactions [114], structural configurations of protein complexes [116], and protein-ligand binding [26].

6. Outlook

Multimodal graph learning is an emerging field with applications across natural sciences, vision and language domains. We anticipate the growth in MGL be driven by fully multimodal graph architectures and new uses in the natural sciences and medicine. We also outline applications to understand when MGL is valuable or unhelpful and needs improvements to resolve challenges represented by multimodal inductive biases or a lack of explicit graphs.

Fully Multimodal Graph Architectures

Prevailing approaches use domain-specialized architectures tailored to each data modality. However, advances in general-purpose architectures provide an expressive strategy to consider dependencies between modalities irrespective of whether they are given as images, language sequences, graphs, or tabular datasets. Moreover, the MGL blueprint supports more complex graph structures, such as hypergraphs [117, 118, 119] and heterogeneous graphs [120, 121].

The blueprint can also pave the way for novel uses of graph-centric multimodal learning. For example, knowledge distillation (KD) aims to transfer knowledge from a teacher model to a smaller student model in a way that preserves performance while using fewer resources. Knowledge-intensive graphs [122, 123, 124] can be used to design more efficient KD loss functions [125, 126]. In another example, visible neural networks specify the architecture such that nodes correspond to concepts (e.g., molecules, pathways) at different scales of the cellular system, ranging from small complexes to extensive signal pathways [2, 127], connected based on biological relationships, used in forward- and back-propagation. By incorporating such inductive biases, models can be trained in a data-efficient manner as they do not have to invent relevant fundamental principles but can know these from the start and thus need fewer data for training. Harmonizing algorithm design with domain knowledge can also improve model interpretability.

Algorithmic Improvements to Resolve Multimodal Challenges

Existing methods are limited in areas without prior knowledge or relational structure. For example, in tasks such as chemical reaction prediction [105], progenitor particle classification [85], physical interaction simulation [6], and protein-ligand modeling [114], interactions relevant for the task are not a priori given, meaning that the methods must automatically capture novel, unspecified, and relevant interactions. Some applications use node feature similarity to dynamically construct local adjacencies after each layer to discover new interactions [85]. However, this cannot capture novel interactions among distant nodes since information is only passed among closely connected nodes in message passing. Methods address this limitation by incorporating attention layers with induced sparsity to discover these interactions [105]. In applications without strong relational structure, such as molecular property prediction [99, 100, 101], particle classification [85], and text classification [74], node features often have more predictive value than any encoded structure. As a result, other methods have been shown to lead to better performance than graph-based methods [129, 130].

Groundbreaking Applications in Natural Sciences and Medicine

Using deep learning in natural sciences revealed the power of graph representations for modeling small and large molecular structures. Combining different types of data can create bridges between the molecular and organism levels for modeling physical, chemical, or biological phenomena on a large scale. Recent knowledge graph applications have been introduced to enable precision medicine and make predictions across genomic, pharmaceutical, and clinical applications [121, 128]. Multi-scale learning systems are becoming valuable tools for protein structure prediction [25], protein property prediction [26], and biomolecular interaction modeling [77]. These methods can incorporate mathematical statements of physical relationships, knowledge graphs, prior distributions, and constraints by modeling predefined graph structures or modifying message-passing algorithms. When such information exists, multimodal learning can enhance image denoising [53], image restoration [53], and human-object interaction [48] in vision systems.

Data and code availability

We summarize multimodal graph learning (MGL) methods and provide a continually updated summary at https://yashaektefaie.github.io/mgl. We host a live table where future MGL methods will be added as a resource to the community.