Beyond Additive Fusion: Learning Non-Additive Multimodal Interactions

Abstract

Multimodal fusion addresses the problem of analyzing spoken words in the multimodal context, including visual expressions and prosodic cues. Even when multimodal models lead to performance improvements, it is often unclear whether bimodal and trimodal interactions are learned or whether modalities are processed independently of each other. We propose Multimodal Residual Optimization (MRO)¹ to separate unimodal, bimodal, and trimodal interactions in a multimodal model. This improves interpretability as the multimodal interaction can be quantified. Inspired by Occam’s razor, the main intuition of MRO is that (simpler) unimodal contributions should be learned before learning (more complex) bimodal and trimodal interactions. For example, bimodal predictions should learn to correct the mistakes (residuals) of unimodal predictions, thereby letting the bimodal predictions focus on the remaining bimodal interactions. Empirically, we observe that MRO successfully separates unimodal, bimodal, and trimodal interactions while not degrading predictive performance. We complement our empirical results with a human perception study and observe that MRO learns multimodal interactions that align with human judgments.

1. Introduction

Multimodal fusion integrates information from what we say, how we speak, and how we visually express ourselves. While multimodal models have led to performance improvements (Zadeh et al., 2017; Tsai et al., 2019; Zellers et al., 2021), they often have the downside of being difficult to interpret: it is unclear whether interactions between two modalities (bimodal) or three modalities (trimodal) are learned, whether modalities are processed independently of each other, or whether these models focus on only one modality (Wu et al., 2021). Quantifying multimodal interactions is an essential building block for future research: in model debugging as a step to better understand models and improve their performance (Du et al., 2019) as well as in AI applications as a step to be more interpretable (Goodman and Flaxman, 2017).

Seminal work (Hessel and Lee, 2020) observed that many multimodal models function like the sum of unimodal models, so-called additive models. In other words, these models might not be learning as many non-additive (bimodal and trimodal) interactions as expected. The non-additive interaction example in Figure 1 exemplifies how humans perceive the whole multimodal example as more than the sum of the two modalities. While the current approach of separating additive and non-additive interactions highlighted the problem of models primarily learning additive contributions, it did not provide solutions to learn non-additive interactions explicitly (Hessel and Lee, 2020). However, many multimodal tasks, such as visual question answering (Cadene et al., 2019), require learning unimodal, bimodal, and trimodal interactions.

Figure 1:

The joint assessment of language and vision (denoted as $f(L,V)$) is different from the sum of unimodal assessments (additive). This is an example for valence from the IEMOCAP dataset (Busso et al., 2008).

In this paper, we introduce Multimodal Residual Optimization (MRO) to explicitly learn and decompose predictions into the sum of unimodal, bimodal, and trimodal interactions. Inspired by Occam’s razor, to prefer simpler solutions, the main intuition of MRO is that (simpler) unimodal contributions should be learned before learning (more complex) bimodal and trimodal interactions. For example, the bimodal predictions should learn to correct the mistakes (residuals) of the unimodal predictions, thereby letting the bimodal predictions focus on the remaining bimodal interactions. Similarly, trimodal predictions should learn what is not modeled by unimodal and bimodal predictions.

We evaluate MRO on six multimodal language datasets, including tasks for intent, sentiment, and emotion recognition. MRO aims to separate multimodal interactions (unimodal, bimodal, and trimodal) without degrading predictive performance. As part of evaluating MRO, we propose a new evaluation metric that extends prior work to three modalities (Hessel and Lee, 2020). We complement our empirical results with a human perceptions study to evaluate whether MRO learns non-additive interactions that align with human judgment.

2. Related Work

We review previous research on four aspects related to multimodal interactions: the prevalence of additive interactions, model-specific and model-agnostic quantification of modality interactions, and taxonomies of multimodal interactions.

Prevalence of Additive Interactions:

Growing empirical evidence (Hessel and Lee, 2020) and annotation studies (Provost et al., 2015; Kruk et al., 2019; Wörtwein et al., 2021) highlight that additive interactions are prevalent especially on datasets that are not carefully balanced, e.g., not having the same image contextualized with different captions (Hessel and Lee, 2020). An empirical approach highlights that multimodal models can be factorized into additive models without significant loss in performance (Hessel and Lee, 2020), indicating that the examined models primarily relied on additive interactions. Similarly, multimodal perception studies indicate the importance of additive interactions: unimodal ratings of emotions are predictive of multimodal ratings (Provost et al., 2015). Further, annotations of the semiotic mode, how the multimodal meaning emerges from individual modalities (Bateman, 2014), of text-image pairs found that modalities provide mostly the same meaning (Kruk et al., 2019). Moreover, modality importance annotations for affective states found that a single modality often contains sufficient information to confirm an affective state (Wörtwein et al., 2021). While additive interactions are sufficient in many cases, non-additive interactions are still needed, especially when datasets contain the same unimodal representation in different multimodal contexts (Provost et al., 2015; Hessel and Lee, 2020).

Model-specific quantification:

Models can indicate how much they rely on potentially non-additive interactions (Zadeh et al., 2018; Tsai et al., 2020). Multimodal routing (Tsai et al., 2020) was recently proposed to interpret the relative importance of multimodal interactions. It uses the routing-by-agreement algorithm (Sabour et al., 2017) to focus more on modalities whose embedding is similar to other modalities’ embeddings. The performance gains of the routing model hint at modalities containing partially redundant information (De Gelder and Bertelson, 2003) for emotion and sentiment prediction. While most model-specific approaches cannot rule out that a multimodal model potentially uses only one modality (Wu et al., 2021), MRO encourages that a bimodal model focuses on bimodal interactions.

Model-agnostic quantification:

Multimodal interactions can be quantified after a model has been trained (Hessel and Lee, 2020; Tsang et al., 2020; Wang et al., 2021; Lyu et al., 2022). EMAP (Hessel and Lee, 2020) is based on the idea of factorizing any trained model into additive and non-additive interactions. Unfortunately, this marginalizing is very costly: with $m$ modalities and a dataset of $N$ samples, it requires $N^m$ forward passes. Compared to EMAP, MRO learns a model that directly separates multimodal interactions.

Taxonomy of Multimodal Interactions:

Many categorizations have been proposed to quantify the relationship between modalities (Kloepfer, 1976; Zhang et al., 2018; Wang et al., 2021). A recent study (Kruk et al., 2019) uses Koepfer’s parallel, amplifying, and divergent. Parallel signals that only one modality is needed for prediction as they all provide the same meaning. Amplifying is sometimes also referred to as “additive” in a non-mathematical sense: modalities provide similar information but their combined meaning is either amplified or diminished. Finally, divergent indicates that modalities provide opposing information. Figure 1 is an example of opposing information.

3. Quantifying Multimodal Interactions

To learn a multimodal model that separates unimodal, bimodal, and trimodal interactions, we first define how to quantify these three types of multimodal interactions. Further, the work presented in this section extends prior work (Hessel and Lee, 2020), which defined evaluation metrics to quantify multimodal interactions in the bimodal case.

Consider three modalities $T$ (text), $V$ (vision), and $A$ (acoustic) with corresponding features $x_T$, $x_V$, $x_A$. A bimodal function $f$ is additive when it can be factorized into the sum of two unimodal functions, $\forall x_T,x_V:f\left(x_T,x_V\right)=g\left(x_T\right)+h\left(x_V\right)$. Further, $f$ contains unimodal contributions when parts of the prediction depend on only one modality: $\exists x_T:{\mathbb{E}}_{v}f\left(x_T,v\right)\ne 0$ (Lyu et al., 2022). This equation is illustrated for the language modality but has the same formulation for the vision modality. Prior work (Hessel and Lee, 2020) proposed EMAP to quantify unimodal contributions $(UC)$ in the context of two modalities. In this paper, we generalize $UC$ to three modalities.

Claim 1.

A trimodal function $f$ contains unimodal contributions when $UC\left(f,x_T,x_V,x_A\right)\ne 0$ with

$$\begin{array}{l}UC\left(f,x_T,x_V,x_A\right)=\\ \underset{v,a}{\mathbb{E}}f\left(x_T,v,a\right)+\underset{t,a}{\mathbb{E}}f\left(t,x_V,a\right)\\ +\underset{t,v}{\mathbb{E}}f\left(t,v,x_A\right)-2\underset{t,v,a}{\mathbb{E}}f(t,v,a).\end{array}$$ (1)

The idea of $UC$ is to evaluate the model with all possible combinations of unimodal features (even feature combinations that are not in a dataset) so that the model cannot use non-additive interactions between modalities. Similarly, we can formulate a function $BI$ to quantify bimodal interactions.

Claim 2.

A trimodal function $f$ contains bimodal interactions $(BI)$ when $BI\left(f,x_T,x_V,x_A\right)\ne 0$ with

$$\begin{array}{l}BI\left(f,x_T,x_V,x_A\right)=\\ \underset{t}{\mathbb{E}}\left[f\left(t,x_V,x_A\right)-UC\left(f,t,x_V,x_A\right)\right]\\ +\underset{v}{\mathbb{E}}\left[f\left(x_T,v,x_A\right)-UC\left(f,x_T,v,x_A\right)\right]\\ +\underset{a}{\mathbb{E}}\left[f\left(x_T,x_V,a\right)-UC\left(f,x_T,x_V,x_a\right)\right].\end{array}$$ (2)

The remaining trimodal interactions $(TI)$ are then simply what is not covered by the unimodal contributions and bimodal interactions:

$$\begin{array}{l}TI\left(f,x_T,x_V,x_A\right)=f\left(x_T,x_V,x_A\right)\\ -UC\left(f,x_T,x_V,x_A\right)-BI\left(f,x_T,x_V,x_A\right).\end{array}$$ (3)

When computational feasible², $UC$, $BI$ and $TI$ are valuable tools to evaluate whether a trimodal model contains unimodal, bimodal, and trimodal interactions. We will use these metrics to evaluate our proposed approaches.

4. Multimodal Residual Optimization

The main contribution of this paper is Multimodal Residual Optimization (MRO) which has the goal of learning and decomposing predictions into unimodal, bimodal and trimodal interactions to quantify them. Inspired by Occam’s razor, the intuition of MRO is that (simpler) unimodal interactions should be prioritized before learning (more complex) bimodal and trimodal interactions. MRO has two components to separate modality interactions: an architecture and loss-function component.

4.1. MRO Architecture

Instead of using a single trimodal function to make a prediction $\widehat{y}=f\left(x_T,x_V,x_A\right)$, the goal of MRO is to make predictions as $\widehat{y}=UC\left(f,x_T,x_V,x_A\right)+BI\left(f,x_T,x_V,x_A\right)+TI\left(f,x_T,x_V,x_A\right)$ without having to compute $UC$, $BI$ and $TI$. Therefore, MRO makes predictions $\widehat{y}$ based on three components:

$$\widehat{y}={\widehat{y}}_{\text{uni}}+{\widehat{y}}_{\text{bi}}+{\widehat{y}}_{\text{tri}}$$ (4)

where ${\widehat{y}}_{\text{uni}}$, ${\widehat{y}}_{\text{bi}}$ and ${\widehat{y}}_{\text{tri}}$ model the unimodal, bimodal, and trimodal interactions respectively. It is important to note that ${\widehat{y}}_{\text{bi}}$ and ${\widehat{y}}_{\text{tri}}$ are intended to model only non-additive interactions, while ${\widehat{y}}_{\text{uni}}$ is designed to model only additive interactions. ${\widehat{y}}_{\text{uni}}$ is defined as

$${\widehat{y}}_{\text{uni}}=f_{{\theta }_T}\left(x_T\right)+f_{{\theta }_V}\left(x_V\right)+f_{{\theta }_A}\left(x_A\right)$$ (5)

where $f_{{\theta }_T}$, $f_{{\theta }_V}$ and $f_{{\theta }_A}$ are models, e.g., neural networks that use only one modality as an input. Each model has its own set of parameters (${\theta }_T$, ${\theta }_V$, and ${\theta }_A$). We parameterize the bimodal and trimodal models in a similar manner:

$$\begin{gathered} {\widehat{y}}_{\text{bi}}=f_{{\theta }_{TV}}\left(x_T,x_V\right)+f_{{\theta }_{TA}}\left(x_T,x_A\right) \\ +f_{{\theta }_{AV}}\left(x_A,x_V\right) \end{gathered}$$ (6)

$${\widehat{y}}_{\text{tri}}=f_{{\theta }_{TVA}}\left(x_T,x_V,x_A\right)$$ (7)

where $f_{{\theta }_{TV}}$, $f_{{\theta }_{TA}}$ and $f_{{\theta }_{AV}}$ are the bimodal models that take only two modalities as input, and $f_{{\theta }_{TVA}}$ takes all three modalities as input. The whole MRO model is parameterized with $\text{Θ}=\left({\theta }_T,{\theta }_V,{\theta }_A,{\theta }_{TV},{\theta }_{TA},{\theta }_{AV},{\theta }_{TVA}\right)$.

This architecture already enforces that ${\widehat{y}}_{\text{uni}}$ can only contain unimodal contributions. While dedicating unimodal, bimodal, and trimodal models was explored in prior work (Zadeh et al., 2016, 2019; Tsai et al., 2020), they did not explicitly encourage ${\widehat{y}}_{\text{bi}}$ and ${\widehat{y}}_{\text{tri}}$ not to contain unimodal contributions and similarly ${\widehat{y}}_{\text{tri}}$ not to contain bimodal interactions. The MRO loss function described in the next section addresses this issue.

4.2. MRO Loss Function

We first explain MRO for two modalities (language and vision) before presenting the more general formulation for three and more modalities.

Bimodal case:

To encourage ${\widehat{y}}_{\text{bi}}$ to not contain unimodal contributions, MRO prioritizes ${\widehat{y}}_{\text{uni}}$. MRO defines the loss function as

$$L(y,\widehat{y})=L\left(y,{\widehat{y}}_{\text{uni}}\right)+L\left(y,sg\left({\widehat{y}}_{\text{uni}}\right)+{\widehat{y}}_{\text{bi}}\right)$$ (8)

where $sg$ refers to stop-gradient (Razavi et al., 2019), which prevents back-propagation through $sg$’s arguments. The first part of Equation 8 updates ${\theta }_T$ and ${\theta }_V$ to predict $y$ using only unimodal contributions ${\widehat{y}}_{\text{uni}}=f_{{\theta }_T}\left(x_T\right)+f_{{\theta }_V}\left(x_V\right)$. The second part of Equation 8 updates ${\theta }_{TV}$ so that $L\left(y,{\widehat{y}}_{\text{uni}}+{\widehat{y}}_{\text{bi}}\right)$ is smaller; i.e., ${\widehat{y}}_{\text{bi}}$ corrects mistakes that ${\widehat{y}}_{\text{uni}}$ makes. We do not backpropagate again to ${\theta }_T$ and ${\theta }_V$ so that ${\widehat{y}}_{\text{bi}}$ does not influence ${\widehat{y}}_{\text{uni}}$; i.e., ${\widehat{y}}_{\text{uni}}$ is optimized independently of ${\widehat{y}}_{\text{bi}}$.

Figure 2 summarizes MRO in the bimodal case.

Figure 2:

Overview of MRO: bimodal model learns what cannot be predicted by the unimodal contributions.

$m$-modal case:

In the case of $m$ modalities, we have $m$ types of interactions: unimodal, bimodal, trimodal, ..., $m$-modal. Instead of separating just additive from all non-additive interactions, we want to separate these $m$ types of interactions. MRO defines the loss function as

$$L(y,\widehat{y})=\sum _{i=1}^{m}L\left(y,sg\left(\sum _{j=1}^{i-1}{\widehat{y}}_j\right)+{\widehat{y}}_i\right)$$ (9)

where ${\widehat{y}}_i$ refers to the $i$-modal predictions, i.e., ${\widehat{y}}_1={\widehat{y}}_{\text{uni}}$, ${\widehat{y}}_2={\widehat{y}}_{\text{bi}}$, ${\widehat{y}}_3={\widehat{y}}_{\text{tri}}$. For the trimodal case, ${\widehat{y}}_{\text{uni}}$, ${\widehat{y}}_{\text{bi}}$, and ${\widehat{y}}_{\text{tri}}$ were defined in subsection 4.1. When $m$ is large than three, the models can be defined following the same approach. Similar to the bimodal case, ${\widehat{y}}_{\text{bi}}$ is optimized independently of ${\widehat{y}}_{\text{tri}}$ as the gradient of ${\widehat{y}}_{\text{bi}}$ is stopped by $sg$ when optimizing ${\widehat{y}}_{\text{tri}}$.

4.3. Sequential MRO

An alternative to MRO’s approach of simultaneously optimizing all prediction components $\left({\widehat{y}}_{\text{uni}},\right.\left.{\widehat{y}}_{\text{bi}},{\widehat{y}}_{\text{tri}}\right)$, the sequential MRO (sMRO) proposes to optimize them sequentially.

First, sMRO optimizes the parameters of ${\widehat{y}}_{\text{uni}}$ using the loss $L\left(y,{\widehat{y}}_{\text{uni}}\right)$ until convergence and then freezes its parameters ${\theta }_L$, ${\theta }_V$, and ${\theta }_A$ before optimizing ${\widehat{y}}_{\text{bi}}$ and ${\widehat{y}}_{\text{tri}}$. Next, sMRO optimizes the parameters of ${\widehat{y}}_{\text{bi}}$ using the loss $L\left(y,{\widehat{y}}_{\text{uni}}+{\widehat{y}}_{\text{bi}}\right)$ until convergence and then freeze the bimodal parameters ${\theta }_{LV}$, ${\theta }_{LA}$ and ${\theta }_{VA}$. The trimodal ${\widehat{y}}_{\text{tri}}$ can then be optimized using the loss $\left.L\left(y,{\widehat{y}}_{\text{uni}}+\right.{\widehat{y}}_{\text{bi}}+{\widehat{y}}_{\text{tri}}\right)$. For cases with more than three modalities, sMRO can optimize the parameters of ${\widehat{y}}_m$ for $L\left(y,{\sum }_{i=1}^m{\widehat{y}}_i\right)$ until convergence and then freeze the parameters of ${\widehat{y}}_m$.

sMRO has similarities with gradient boosting (GB) (Friedman, 2001) when GB has, in the trimodal case, three learners that correspond to the prediction components ${\widehat{y}}_{\text{uni}}$, ${\widehat{y}}_{\text{bi}}$, and ${\widehat{y}}_{\text{tri}}$. Unlike sMRO, GB is not suitable for some loss functions, such as the mean absolute error (MAE; its gradient is not proportional to the residual), as each learner in GB estimates the gradient of the errors from the previous learners. In the case of MAE, learners will predict −1 or 1, which leads to a poor fit with only three learners.

5. Experimental Methodology

We evaluate whether we can train a model that separates unimodal, bimodal, and trimodal interactions while not degrading predictive performance.

Datasets:

We focus on five sentiment- and emotion-annotated datasets for which prior work used multimodal models, see Table 1. We also include a sixth Instagram dataset (Kruk et al., 2019) as it has modality interaction annotations (semiotic modes), which we can use to evaluate MRO.

Table 1:

Dataset overview.

Original Paper	Tasks	Abbreviation	Samples	Modalities
(Zadeh et al., 2016)	Sentiment (regression)	MOSI	2.2k	3
(Zadeh et al., 2018)	Sentiment, Polarity, Happiness (regression)	MOSEI	22.9k	3
(Busso et al., 2008)	Arousal and Valence (regression)	IEMOCAP	4.8k	3
(Valstar et al., 2016)	Arousal and Valence (regression)	SEWA	1.9k	3
(Nelson et al., 2021)	Affect categories (4-way classification)	TPOT	17.3k	3
(Kruk et al., 2019)	Intent of Instagram posts (7-way classification)	Instagram	1.3k	2

We use the same features across all sentiment and emotion datasets: RoBERTa (Liu et al., 2020) as a representation of transcribed utterances; OpenFace 2.0 (Baltrusaitis et al., 2018) to summarize face-related features, and openSMILE’s eGeMAPS (Eyben et al., 2015) to summarize acoustic features. For the Instagram dataset, we use the author-provided ResNet features (He et al., 2016) to summarize the image content and use RoBERTa to represent captions.

Evaluation:

We want that the prediction components ${\widehat{y}}_{\text{uni}}$, ${\widehat{y}}_{\text{bi}}$ and ${\widehat{y}}_{\text{tri}}$ correspond to $UC(\widehat{y})$, $BI(\widehat{y})$, and $TI(\widehat{y})$ so that the prediction components represent only unimodal, only bimodal, and only trimodal interactions. To test this, we use $\left|UC\left({\widehat{y}}_{\text{bi}}+{\widehat{y}}_{\text{tri}}\right)\right|$ to evaluate whether the bimodal and trimodal predictions contain unimodal contributions and $\left|BI\left({\widehat{y}}_{\text{tri}}\right)\right|$ whether the trimodal prediction contains bimodal contributions. Given the MRO-architecture, ${\widehat{y}}_{\text{uni}}$ cannot include bimodal and trimodal interactions and ${\widehat{y}}_{\text{bi}}$ cannot include trimodal interactions. This means, if $\left|UC\left({\widehat{y}}_{\text{bi}}+{\widehat{y}}_{\text{tri}}\right)\right|+\left|BI\left({\widehat{y}}_{\text{tri}}\right)\right|$ is 0, the model perfectly separates unimodal, bimodal, and trimodal interactions, i.e., ${\widehat{y}}_{\text{uni}}=UC(\widehat{y})$, ${\widehat{y}}_{\text{bi}}=BI(\widehat{y})$, and ${\widehat{y}}_{\text{tri}}=TI(\widehat{y})$. We use 5-fold test setup for all datasets.

Models:

We compare the MRO-architecture when optimized in different manners: with $L\left(y,{\widehat{y}}_{\text{uni}}+{\widehat{y}}_{\text{bi}}+{\widehat{y}}_{\text{tri}}\right)$ (referred to as Joint), sMRO, and MRO. For performance comparison, we include the routing model (Tsai et al., 2020) (referred to as Routing), a recently proposed model with the goal of modality interpretability. Lastly, we compare the performance against a single trimodal model $\widehat{y}=f_{{\theta }_{TVA}}\left(x_T,x_V,x_A\right)$ (referred to as Tri) to evaluate whether the larger MRO-architecture has two many parameters for smaller datasets.

Implementation Details:

The functions $f$ of Equation 4 are instantiated as multi-layer perceptrons. For each multimodal model, e.g., $f_{{\theta }_{TV}}$, we implement two popular types of fusion: early fusion (concatenating the modalities) and tensor fusion (Zadeh et al., 2017) (outer product between modalities after learning unimodal embeddings). The type of fusion is a hyper-parameter together with the number of layers, their width, learning rate, learning rate decay, L2 weight decay, dropout, and with/without prior feature selection. As a loss function, we use the mean absolute error for regression tasks and the cross-entropy loss for classification tasks.

6. Multimodal Perception Study

We conduct a multimodal perception study to evaluate whether MRO learns non-additive interactions, when humans also require non-additive interactions. We choose arousal and valence on the IEMOCAP dataset for this study as arousal and valence are two fundamental dimensions to describe emotional states (Munezero et al., 2014).

Study Design:

Crowd workers³ are asked to rate arousal and valence of video segments when being exposed to only a subset of modalities. The four subsets are: 1) the transcript of what the person says (T); 2) the muted video (V); 3) the low-pass filtered audio (A), and 4) the transcripts, the video, and the original audio (${\text{TVA}}_O$). IEMOCAP has ten speakers. We randomly select ten segments for each speaker, i.e. 100 segments.

Audio Processing:

It is challenging to disentangle speech content and how we speak (Bhargava and Bagkent, 2012). Similar to previous work, we low-pass filter the audio signal (Yang et al., 2012). Instead of using 850 Hz as a cut-off (Yang et al., 2012), we use a lower cut-off frequency, as we could understand spoken words at 850 Hz. We choose 660 Hz⁴ as it is the mean of the maximum pitch in an empirical study (Li and Yiu, 2006) and it also closely coincides with the maximum pitch of contralto singers (E5 at 659.25 Hz). We choose this pitch-focused definition as we believe that prosodic information will predict arousal and valence.

Avoiding learning effects:

Raters might be able to infer the missing multi-modal context after having rated some of the unimodal subsets for a specific segment. We therefore use two mechanisms to address learning effects across the modalities. First, each of the raters annotates only 20 randomly selected segments for each modality subset (we have eight raters per segment and modality subset). Second, we structurally randomize the order of the modality subsets by first presenting all unimodal subsets in a random order and in the end the trimodal segments.

Ratings and reliability:

Following the annotation setup from IEMOCAP, we use the ordinal arousal and valence manikins scale consisting of five levels (Bradley and Lang, 1994) to rate the two emotional dimensions. The effective reliability (Rosenthal, 2005) over $k$ raters as measured by the Intra-class Correlation Coefficient ICC(2, k-1) is excellent (above 0.9) (Koo and Li, 2016) for all modality subsets. Further, our new trimodal ratings (${\text{TVA}}_O$) correlate highly with the existing annotations on IEMOCAP r(98) = 0.88, p < 0.001 for arousal and r(98) = 0.92, p < 0.001 for valence, indicating that we can use our new annotations to inspect models trained on the original annotations.

Evaluation:

To evaluate when humans require non-additive interactions, we train a linear regression model (an additive model) that predicts ${\text{TVA}}_O$ given T, V, and A. We refer to this model as ${\widehat{y}}_{\text{uni}}^{\text{human}}$. The model fit of ${\widehat{y}}_{\text{uni}}^{\text{human}}$ shows how important the missing non-additive interactions are (Provost et al., 2015). Further, the absolute error $\left|{\text{TVA}}_O-{\widehat{y}}_{\text{uni}}^{\text{human}}\right|$ measures how important the missing non-additive interactions are to humans for each segment. We use $\left|{\text{TVA}}_O-{\widehat{y}}_{\text{uni}}^{\text{human}}\right|$ to answer the question: does MRO learn more non-additive interactions when $\left|{\text{TVA}}_O-{\widehat{y}}_{\text{uni}}^{\text{human}}\right|$ is larger, i.e., when humans require non-additive interactions?

7. Results and Discussion

Sanity Check:

Before evaluating MRO on more complex datasets, we conduct a sanity check on two simpler datasets: $x_T+x_V+x_A$ which requires only unimodal contributions (we refer to it as Sanity Check Unimodal) and $x_T{x}_V+x_T{x}_A+x_V{x}_A$ which requires only bimodal interactions (we refer to it as Sanity Check Bimodal). Figure 3 shows that the joint and the routing model do not separate unimodal, bimodal, and trimodal interactions well as $\left|UC\left({\widehat{y}}_{\text{bi}}+{\widehat{y}}_{\text{tri}}\right)\right|+\left|BI\left({\widehat{y}}_{\text{tri}}\right)\right|$ is high. As expected, sMRO and MRO separate the interacts almost perfectly as $\left|UC\left({\widehat{y}}_{\text{bi}}+{\widehat{y}}_{\text{tri}}\right)\right|+\left|BI\left({\widehat{y}}_{\text{tri}}\right)\right|$ is very close to 0.

Figure 3:

Average $\left|UC\left({\widehat{y}}_{\text{bi}}+{\widehat{y}}_{\text{tri}}\right)\right|+\left|BI\left({\widehat{y}}_{\text{tri}}\right)\right|$ for all models and datasets. Lower values indicate a better separation of unimodal, bimodal, and trimodal contributions.

To test how many epochs are needed to minimize $\left|UC\left({\widehat{y}}_{\text{bi}}+{\widehat{y}}_{\text{tri}}\right)\right|+\left|BI\left({\widehat{y}}_{\text{tri}}\right)\right|$, we evaluate it after each epoch. The results in Figure 4 show that the separation during the first epochs becomes worse as ${\widehat{y}}_{\text{uni}}$ has not yet learned much, meaning ${\widehat{y}}_{\text{bi}}$ and ${\widehat{y}}_{\text{tri}}$ try to predict unimodal contributions which increases $\left|UC\left({\widehat{y}}_{\text{bi}}+{\widehat{y}}_{\text{tri}}\right)\right|$. However, after a few epochs the separation becomes better and $\left|UC\left({\widehat{y}}_{\text{bi}}+{\widehat{y}}_{\text{tri}}\right)\right|+\left|BI\left({\widehat{y}}_{\text{tri}}\right)\right|$ reaches 0. The same can be observed for the bimodal sanity check in Figure 4.

Figure 4:

$\left|UC\left({\widehat{y}}_{\text{bi}}+{\widehat{y}}_{\text{tri}}\right)\right|+\left|BI\left({\widehat{y}}_{\text{tri}}\right)\right|$ for the same model optimized with either $L\left(y,{\widehat{y}}_{\text{uni}}+{\widehat{y}}_{\text{bi}}+{\widehat{y}}_{\text{tri}}\right)$ (Joint, in blue) or with MRO (in red). Lower values indicate a better separation of unimodal, bimodal, and trimodal interactions.

MRO significantly reduces $\left|UC\left({\widehat{y}}_{\mathbf{bi}}+{\widehat{y}}_{\mathbf{tri}}\right)\right|+\left|BI\left({\widehat{y}}_{\mathbf{tri}}\right)\right|$.

Similar to the sanity check on simpler dataset, we want that $\left|UC\left({\widehat{y}}_{\text{bi}}+{\widehat{y}}_{\text{tri}}\right)\right|+\left|BI\left({\widehat{y}}_{\text{tri}}\right)\right|$ is as small as possible. For easier comparison across datasets, we normalize $\left|UC\left({\widehat{y}}_{\text{bi}}+{\widehat{y}}_{\text{tri}}\right)\right|+\left|BI\left({\widehat{y}}_{\text{tri}}\right)\right|$ by the standard deviation of the ground truth from the training set. Figure 3 shows that sMRO and MRO significantly reduce $\mid UC\left({\widehat{y}}_{\text{bi}}+\right.\left.{\widehat{y}}_{\text{tri}}\right)\left|+\left|BI\left({\widehat{y}}_{\text{tri}}\right)\right|\right.$ compared to models optimized with $L\left(y,{\widehat{y}}_{\text{uni}}+{\widehat{y}}_{\text{bi}}+{\widehat{y}}_{\text{tri}}\right)$ (Joint) and die routing model.

As it is computationally very expensive to evaluate $\left|UC\left({\widehat{y}}_{\text{bi}}+{\widehat{y}}_{\text{tri}}\right)\right|+\left|BI\left({\widehat{y}}_{\text{tri}}\right)\right|$ after each epoch, we plot it only for arousal and valence on IEM0-CAP in Figure 4. We focus on IEMOCAP as we also conduct the perception study on it, see section 6. While the plot for arousal in Figure 4 is a bit noisy, MRO quickly reduces $\mid UC\left({\widehat{y}}_{\text{bi}}+tri\right)\mid +\left|BI\left({\widehat{y}}_{\text{tri}}\right)\right|$. The same can be observed for valence in Figure 4.

MRO does not degrade performance.

The secondary goal of MRO is not degrading performance. Table 3 lists the models’ performance. Models optimized with MRO are in no case significantly worse than any other model. However, they arc statistically significantly better than the joint model for valence on SEWA and happiness on MOSEI.

Table 3:

Average performance over the test folds. Higher is better.

	Tri	Routing	Joint	sMRO	MRO
MOSI (Pearson’s $r$)
Sentiment	0.662	0.658	0.657	0.656	0.661
MOSEI (Pearson’s $r$)
Sentiment	0.723	0.727	0.727	0.726	0.727
Polarity	0.599	0.597	0.606	0.593	0.605
Happiness	0.637	0.642	0.637	0.630	0.641
IEMOCAP (Concordance Correlation Coefficient)
Arousal	0.588	0.613	0.622	0.624	0.611
Valence	0.647	0.655	0.624	0.603	0.634
SEWA (Concordance Correlation Coefficient)
Arousal	0.317	0.263	0.293	0.292	0.304
Valence	0.268	0.335	0.268	0.310	0.337
TPOT (Accuracy)
Constructs	0.565	0.554	0.566	0.566	0.574
Instagram (macro ROC AUC)
Intent	0.876	0.731	0.891	0.888	0.891
Mean	0.588	0.595	0.589	0.589	0.599

MRO might generalizes slightly better because, similar to structural risk minimization (Vapnik, 1999), it prioritizes simpler models and relies on more complex multimodal models only when needed. Another reason is that MRO has similar’ effects as having auxiliary unimodal loss functions which seems beneficial for multimodal models (Wang et al., 2020; Zeng et al., 2021).

Ablating ${\widehat{y}}_{\mathbf{bi}}+{\widehat{y}}_{\mathbf{tri}}$ decreases performance.

We quantify the average performance impact of post-hoc removing ${\widehat{y}}_{\text{bi}}+{\widehat{y}}_{\text{tri}}$ across datasets, i.e., $\widehat{y}={\widehat{y}}_{\text{uni}}$. When comparing Table 4 with Table 3, we observe that removing ${\widehat{y}}_{\text{bi}}+{\widehat{y}}_{\text{tri}}$ (the non-additive predictions), hurts performance. While additive contributions are very important, non-additive interactions are needed for best performance.

Table 4:

Average performance when post-hoc removing ${\widehat{y}}_{\text{bi}}+{\widehat{y}}_{\text{tri}}$, i.e., $\widehat{y}={\widehat{y}}_{\text{uni}}$.

	sMRO	MRO
Mean	0.577	0.587

MRO learns more non-additive interactions when two modalities are informative.

The TPOT dataset has human judgments for how important modalities are to confirm the current affective state (Wörtwein et al., 2021). Three importance levels were annotated: 1) a modality is sufficient to confirm the affective state (while ignoring other modalities), 2) a modality contains relevant information for the affective state (information from a second modality is needed), and 3) a modality contains no information for the current affective state.

We hypothesize that MRO uses more non-additive interactions $\left({\widehat{y}}_{\text{bi}}+{\widehat{y}}_{\text{tri}}\right)$ for samples with at least two informative modalities (relevant or sufficient) compared to samples with only one informative modality. To measure whether ${\widehat{y}}_{\text{bi}}+{\widehat{y}}_{\text{tri}}$ are used more, we calculate how much the softmax probabilities (TPOT is a classification task) change when removing ${\widehat{y}}_{\text{bi}}+{\widehat{y}}_{\text{tri}}$, i.e., ${\sum }_{k=1}^4\left|\mathrm{softmax}{(\widehat{y})}^{(k)}-\mathrm{softmax}{\left({\widehat{y}}_{\text{uni}}\right)}^{(k)}\right|$ where $k$ indexes the probability vector for the four classes. The means of samples with two informative modalities (0.299) and only one informative modality (0.264) are significantly different according to an independent t-test, $t(2671)=5.059$, p < 0.001. This suggests that MRO not only mathematically separates unimodal, bimodal, and trimodal interactions but that its separation also correlates with human assessments. Further, this observation provides evidence that models are more likely to learn non-additive interactions when several modalities are themselves informative.

MRO learns more non-additive interactions when modalities amplify each other.

We included the Instagram dataset (Kruk et al., 2019) because it has modality interaction annotations (semiotic modes) that are inspired by Kloepfer (Kloepfer, 1976). To test whether ${\widehat{y}}_{\text{bi}}$ (this dataset has only two modalities) contributes more depending on the semiotic mode (parallel, amplifying, and divergent), we conduct a one-way ANOVA on the probability changes when removing ${\widehat{y}}_{\text{bi}}$. The means between the semiotic modes are significantly different, F(2,1296) = 5.059, p = 0.006, with the highest absolute average change for amplifying (0.317), followed by parallel (0.272), and then divergent (0.256). The means between amplifying and parallel are significantly different t(1297) = 2.432, p = 0.015 as well as between amplifying and divergent t(1297) = 2.874, p = 0.004. Similar to the results on TPOT, it is confirming that MRO learned significantly larger non-additive contributions $\left({\widehat{y}}_{\text{bi}}\right)$ for amplifying than for parallel. A possible explanation why diverging seems to require the least non-additive interactions is that the definition of diverging requires that only the meaning of the modalities is opposing but it does not specify how the combined meaning is formed. Even if the combined meaning of Figure 1 was neutral (additive), the semiotic mode is still divergent.

MRO learns non-additive interaction when humans need non-additive interactions.

The additive model ${\widehat{y}}_{\text{uni}}^{\text{human}}$ of predicting the mutlimodal ratings ${\text{TVA}}_O$ given the uni-modal ratings, fits very well (r² = 0.85 for arousal and r² = 0.85 for valence) which is inline with similar prior work (Provost et al., 2015). Even though our multimodal model is not on par with ${\widehat{y}}_{\text{uni}}^{\text{human}}$ (r² = 0.68 for arousal and r² = 0.66 for valence), we observe a significant correlation of r(98) = 0.202, p = 0.043 for valence between $\left|\text{TV}{\text{A}}_O-{\widehat{y}}_{\text{uni}}^{\text{human}}\right|$ (the missing non-additive interactions) and $\left|{\widehat{y}}_{\text{bi}}+{\widehat{y}}_{\text{tri}}\right|$ (non-additive contributions). This indicates that ${\widehat{y}}_{\text{bi}}+{\widehat{y}}_{\text{tri}}$ learned non-additive interactions that cannot be explained by ${\widehat{y}}_{\text{uni}}^{\text{human}}$. For arousal, we do not observe a significant correlation, potentially because the optimization seems far nosier for arousal then for valence, see Figure 4.

8. Conclusion

We proposed MRO to explicitly learn and separate unimodal, bimodal, and trimodal interactions in a multimodal model. This separation is essential for quantifying how much a model uses multimodal interactions and is a step towards more interpretable models. Based on prior work (Hessel and Lee, 2020) we proposed a new evaluation metrics to quantify whether a trimodal models uses unimodal, bimodal, and trimodal interactions. Empirically, we observed that MRO successfully separated unimodal, bimodal, and trimodal interactions while not degrading predictive performance. Beyond the empirical evaluation, MRO learns non-additive interactions in accordance with human judgments on three datasets.

Limitations

We evaluated MRO in the context of language, vision, and acoustic modalities. Future work could explore MRO’s performance on different modalities. Exploring MRO beyond three modalities will also be interesting. To address a potentially growing number of parameters for models with more than three modalities, sharing modality representation could be explored: the bi-modal models could be given access to intermediate representations from unimodal models for modalities they have in common. Sharing representations could reduce the overall model size.

While we evaluated MRO on many sentiment and emotion annotated datasets, these datasets are primarily in English, and one is in German (SEWA). More research is needed to work with a more diverse set of languages.

It will also be interesting to study MRO in tasks that require multimodal fusion and translation: generating a modality given a set of different modalities.

Table 2:

Basic demographic information about the annotators.

	Arousal	Valence
Min. age	19	21
Mean age	36	37
Max. age	79	62
Female	20	19
Male	20	21

A. Experimental Details

A.1. IEMOCAP

IEMOCAP has multiple recording conditions of two speakers interacting: acted interactions, improvised interactions, and spontaneous interactions (Busso et al., 2008). In this paper, we use the improvised interactions as they cover a diverse range of emotional expressions and are not tied to a set of fixed utterances as is the case of the acted interactions.

A.2. MOSEI

Polarity is an established dimension in ethics research (Tyagi et al., 2020) and is typically defined as the absolute value of the sentiment intensity (Hutto and Gilbert, 2014). We use this definition and apply it to MOSEI’s sentiment ratings.

A.3. SEWA

Instead of using SEWA’s time continuous ratings of valence and arousal, we take the average of the ratings for each utterance to make SEWA similar to the other datasets.

A.4. Features

We use the openSMILe configuration eGeMAPS v0.1b (Eyben et al., 2015) which extracts instantaneous low-level descriptors and summaries over a moving window. For the low-level descriptors, we calculate the median and interquartile range for each segment. For the summary features, we take the median over each segment.

OpenFace 2.0 extracts many face-related features. We summarize OpenFace’s facial action units (Ekman, 1982), head pose, and eye gaze features with the mean and standard deviation.

B. $UC$ and $BI$ for trimodal Models

Any trimodal function $f$ can be expressed as $f_T+f_V+f_A+f_{TV}+f_{TA}+f_{VA}+f_{TVA}$ such that $UC(f)=f_T+f_V+f_A$ and $BI(f)=f_{TV}+f_{TA}+f_{VA}$ where the bimodal functions do not contains unimodal contributions: $\forall x_T:{\mathbb{E}}_v\left[f_{TV}\left(x_T,v\right)\right]=0$ and similar for $x_V$ and $x_A$. Further, the trimodal function should not contains unimodal and bimodal interactions: $\forall x_T$, $x_V:{\mathbb{E}}_a\left[f_{TVA}\left(x_T,x_V,a\right)=0\right.$ and similar for the pairs $\left(x_T,x_A\right)$ and $\left(x_V,x_A\right)$.

Proof. As any trimodal function can be expressed as the above function, we show that the definition of $UC$ returns exactly the unimodal contributions $f_T+f_V+f_A$.

$$\begin{array}{l}UC(f)\\ =\underset{v,a}{\mathbb{E}}f\left(x_T,v,a\right)+\underset{t,a}{\mathbb{E}}f\left(t,x_V,a\right)\end{array}$$ (10)

$$\begin{array}{l}+\underset{t,v}{\mathbb{E}}f\left(t,v,x_A\right)-2\underset{t,v,a}{\mathbb{E}}f(t,v,a)\\ =\left(f_T\left(x_T\right)+\underset{v}{\mathbb{E}}{f}_V(v)+\underset{a}{\mathbb{E}}{f}_A(a)\right.\\ +\underset{v}{\mathbb{E}}{f}_{TV}\left(x_T,v\right)+\underset{a}{\mathbb{E}}{f}_{TA}\left(x_T,a\right)\\ \left.+\underset{v,a}{\mathbb{E}}{f}_{VA}(v,a)+\underset{v,a}{\mathbb{E}}{f}_{TVA}\left(x_T,v,a\right)\right)\\ +\left(\underset{t}{\mathbb{E}}{f}_T(t)+f_V\left(x_V\right)+\underset{a}{\mathbb{E}}{f}_A(a)\right.\\ +\underset{t}{\mathbb{E}}{f}_{TV}\left(t,x_V\right)+\underset{t,a}{\mathbb{E}}{f}_{TA}(t,a)\\ \left.+\underset{a}{\mathbb{E}}{f}_{VA}\left(x_V,a\right)+\underset{t,a}{\mathbb{E}}{f}_{TVA}\left(t,x_V,a\right)\right)\\ +\left(\underset{t}{\mathbb{E}}{f}_T(t)+\underset{v}{\mathbb{E}}{f}_V(v)+f_A\left(x_A\right)\right.\\ +\underset{tv}{\mathbb{E}}{f}_{TV}(t,v)+\underset{t}{\mathbb{E}}{f}_{TA}\left(t,x_A\right)\\ \left.+\underset{v}{\mathbb{E}}{f}_{VA}\left(v,x_A\right)+\underset{t,v}{\mathbb{E}}{f}_{TVA}\left(t,v,x_A\right)\right)\\ -2\left(\underset{t}{\mathbb{E}}{f}_T(t)+\underset{v}{\mathbb{E}}{f}_V(v)+\underset{a}{\mathbb{E}}{f}_A(a)\right.\\ +\underset{t,v}{\mathbb{E}}{f}_{TV}(t,v)+\underset{t,a}{\mathbb{E}}{f}_{TA}(t,a)\end{array}$$ (11)

$$\begin{array}{l}\left.+\underset{v,a}{\mathbb{E}}{f}_{VA}(v,a)+\underset{t,v,a}{\mathbb{E}}{f}_{TVA}(t,v,a)\right)\\ =\left(f_T\left(x_T\right)+\underset{v}{\mathbb{E}}{f}_V(v)+\underset{a}{\mathbb{E}}{f}_A(a)\right)\\ +\left(\underset{t}{\mathbb{E}}{f}_T(t)+f_V\left(x_V\right)+\underset{a}{\mathbb{E}}{f}_A(a)\right)\\ +\left(\underset{t}{\mathbb{E}}{f}_T(t)+\underset{v}{\mathbb{E}}{f}_V(v)+f_A\left(x_A\right)\right)\end{array}$$ (12)

$$-2\left(\underset{t}{\mathbb{E}}{f}_T(t)+\underset{v}{\mathbb{E}}{f}_V(v)+\underset{a}{\mathbb{E}}{f}_A(a)\right)$$ (13)

$$=f_T\left(x_T\right)+f_V\left(x_V\right)+f_A\left(x_A\right)$$ (14)

□

Compared to $BI$ in the bimodal case, we need to also remove trimodal interactions for $BI$ in the trimodal context.

Claim 3.

$BI$ is defined for three modalities as

$$\begin{array}{l}BI(f)\\ =\underset{t}{\mathbb{E}}\left[f\left(t,x_V,x_A\right)-UC\left(f,t,x_V,x_A\right)\right]\\ +\underset{v}{\mathbb{E}}\left[f\left(x_T,v,x_A\right)-UC\left(f,x_T,v,x_A\right)\right]\\ +\underset{a}{\mathbb{E}}\left[f\left(x_T,x_V,a\right)-UC\left(f,x_T,x_V,a\right)\right]\end{array}$$ (15)

$$\begin{array}{l}=\underset{t}{\mathbb{E}}f\left(t,x_V,x_A\right)+\underset{v}{\mathbb{E}}f\left(x_T,v,x_A\right)\\ +\underset{a}{\mathbb{E}}f\left(x_T,x_V,a\right)-2\underset{t,v}{\mathbb{E}}f\left(t,v,x_A\right)\\ -2\underset{t,a}{\mathbb{E}}f\left(t,x_V,a\right)-2\underset{v,a}{\mathbb{E}}f\left(x_T,v,a\right)\\ +3\underset{t,v,a}{\mathbb{E}}f(t,v,a)\end{array}$$ (16)

$$=f_{TV}\left(x_T,x_V\right)+f_{TA}\left(x_T,x_A\right)+f_{VA}\left(x_V,x_A\right)$$ (17)

The omitted steps are to apply the definition of $UC$ and cancelling terms to get to Equation 16. From there on, we write $f$ as $f_T+f_V+f_A+f_{TV}+f_{TA}+f_{VA}+f_{TVA}$ and use their properties (expected values of the bi/trimodal function are 0).

C. Study Details

In addition to the three unimodal and the trimodal combinations we explored bimodal combinations: 1) the muted video with the transcript (TV); 2) the muted video with the low-pass filtered audio (VA); 3) the transcript with the low-pass filtered audio (TA); 4) for comparison the original audio with the transcript (${\text{TA}}_O$).

C.1. Reliability

We report two types reliabilities: the averaged pairwise reliability between two random raters (ICC(2,1)) and the effective reliability of the mean over k=8 raters (ICC(2, k-1)). Pairwise and effective reliability address different purposes: pairwise is needed to determine how many raters are needed to achieve a targeted effective reliability (Rosenthal, 2005). Averaging over raters is important as emotional dimensions are subjective and difficult to annotate (especially when modalities are missing). The effective reliability describes how reliable the mean over the raters is, i.e., if we were to draw a new set of ratings and were to average them, how similar is this new mean to our current mean.

Table 5:

Pairwise and effective reliability across the eight combinations. ICC is calculated with the R package psych.

Combination	Avg. ICC(2, 1)		ICC(2, k-l)
	Arousal	Valence	Arousal	Valence
T	0.36	0.55	0.96	0.98
V	0.52	0.64	0.98	0.99
A	0.57	0.38	0.98	0.96
TV	0.48	0.62	0.97	0.98
VA	0.56	0.61	0.98	0.98
TA	0.60	0.54	0.98	0.98
${\text{TA}}_O$	0.55	0.62	0.98	0.98
${\text{TVA}}_O$	0.56	0.64	0.98	0.99

Except for transcripts-only (T) on arousal and acoustic-only (A) on valence, all pairwise reliabilities are moderate (between 0.5 and 0.75) (Koo and Li, 2016), see Table 5. The effective reliability (Rosenthal, 2005) of the mean over k raters as measured by ICC(2, k-1) is excellent (above 0.9) for all combinations. Instead of directly taking the mean over the raters, we apply, as common in affective computing, a z-normalization for each rater (Valstar et al., 2016; Busso et al., 2008) and take a weighted mean (Grimm and Kroschel, 2005) over the raters.

C.2. Compensation

All raters are paid the same fixed amount, leading to an average hourly rate of 11.14 USD/h.

D. Additional Experiments

MRO reaches $\left|UC\left({\widehat{y}}_{\mathbf{bi}}+{\widehat{y}}_{\mathbf{tri}}\right)\right|+\left|BI\left({\widehat{y}}_{\mathbf{tri}}\right)\right|=0$ when trained long enough.

As we have seen in Figure 4, a model might need to be optimized long enough to minimize $\left|UC\left({\widehat{y}}_{\text{bi}}+{\widehat{y}}_{\text{tri}}\right)\right|+\left|BI\left({\widehat{y}}_{\text{tri}}\right)\right|$. We therefore also investigate $\left|UC\left({\widehat{y}}_{\text{bi}}+{\widehat{y}}_{\text{tri}}\right)\right|+\left|BI\left({\widehat{y}}_{\text{tri}}\right)\right|$ when models are trained without early stopping. While such models are more likely to have poor generalization performance this allows us to test how much we could minimize $\left|UC\left({\widehat{y}}_{\text{bi}}+{\widehat{y}}_{\text{tri}}\right)\right|+\left|BI\left({\widehat{y}}_{\text{tri}}\right)\right|$ with MRO and sMRO. As can be seen in Table 6, MRO and sMRO are numerically very close to 0.0 demonstrating that such optimized models almost perfectly separate unimoda, bimodal, and trimodal interactions.

Table 6:

Bootstrapped average of normalized $\mid UC\left({\widehat{y}}_{\text{bi}}+\right.\left.{\widehat{y}}_{\text{tri}}\right)\left|+\left|BI\left({\widehat{y}}_{\text{tri}}\right)\right|\right.$ on the test folds (1.0 corresponds to a magnitude of one standard deviation) when models are trained without early stopping. Lower is better (ideally 0.0).

	Joint	sMRO	MRO
SEWA
Arousal	[0.81,0.87]	[0.00, 0.00]	[0.00, 0.01]
Valence	[0.29, 0.31]	[0.00, 0.00]	[0.02, 0.03]
IEMOCAP
Arousal	[0.41,0.42]	[0.00, 0.00]	[0.03, 0.03]
Valence	[0.20, 0.21]	[0.00, 0.00]	[0.05, 0.06]
MOSI
Sentiment	[0.14, 0.15]	[0.00, 0.00]	[0.00, 0.00]
MOSEI
Sentiment	[0.41,0.42]	[0.00, 0.00]	[0.00, 0.00]
Polarity	[0.44, 0.45]	[0.04, 0.04]	[0.00, 0.00]
Happiness	[0.44, 0.45]	[0.15, 0.15]	[0.01,0.01]
TPOT
Constructs	[0.29, 0.29]	[0.01,0.01]	[0.04,0.04]
Instagram Intent	[0.12, 0.13]	[0.00, 0.01]	[0.02, 0.02]

Removing stop-gradient leads to a worse separation:

Theoretically, we should be able to remove stop-gradient $\left(sg\right)$ from Equation 9 as they have the same global minima. In practice, we observe that doing so leads to worse separation of interactions, see Table 7.

MRO applies to transformers as well:

When choosing transformers (Vaswani et al., 2017) as a base model instead of multilayer perceptrons, we observe the same trend that a model trained without MRO does not separate the multimodal interactions, whereas when trained with MRO the interactions are far better separated, see Table 8.

Interactions needed for amplifiers, ambiguity, and rare behaviors.

Table 9 summarizes the five segments with the largest absolute errors $\left|{\text{TVA}}_O-{\widehat{y}}_{\text{uni}}^{\text{human}}\right|$ separately for arousal and valence (we refer to these segments with A₁ to A₅ and V₁ to V₅). Qualitatively, three groups emerge: amplifiers, ambiguities, and rare behaviors.

Unimodal amplifiers:

Amplifiers are essential for valence and sentiment as an intense expression can be very negative or positive. A modality might contain a strong amplifier (language in case of example V₃ and V₄) but the modality might not provide strong evidence for the directionality. In such cases, non-additive interactions are needed to combine the directionality from one modality with the amplifier from another modality.

Table 7:

Average performance and bootstrapped average of normalized $\left|UC\left({\widehat{y}}_{\text{bi}}+{\widehat{y}}_{\text{tri}}\right)\right|+\left|BI\left({\widehat{y}}_{\text{tri}}\right)\right|$ on the test folds (1.0 corresponds to a magnitude of one standard deviation) when models are trained without early stopping.

	Performance	$\left|UC\left({\widehat{y}}_{\text{bi}}+{\widehat{y}}_{\text{tri}}\right)\right|+\left|BI\left({\widehat{y}}_{\text{tri}}\right)\right|$
SEWA
Arousal	0.599	[0.05, 0.06]
Valence	0.638	[0.13, 0.14]
IEMOCAP
Arousal	0.316	[0.27, 0.29]
Valence	0.297	[0.30, 0.32]
MOSI
Sentiment	0.660	[0.03, 0.03]
MOSEI
Sentiment	0.724	[0.06, 0.06]
Polarity	0.605	[0.17, 0.19]
Happiness	0.644	[0.16, 0.16]
TPOT
Constructs	0.569	[0.13, 0.13]
Instagram
Intent	0.890	[0.04, 0.04]

Table 8:

Average of normalized $\left|UC\left({\widehat{y}}_{\text{bi}}+{\widehat{y}}_{\text{tri}}\right)\right|+\left|BI\left({\widehat{y}}_{\text{tri}}\right)\right|$ on the test folds (1.0 corresponds to a magnitude of one standard deviation) when using transformers as a base model instead of multilayer perceptrons.

	Joint	MRO
MOSEI
Sentiment	[0.64, 0.67]	[0.01,0.01]

Ambiguities:

When a modality might not provide information in either direction (language in case V₁, V₂, V₅, A₁, A₂), more contextual information in form of bimodal interactions is needed.

Rare behaviors:

When a typically important modality is “missing” (language in case of A₂ and vision in case of A₃) or a typically less important modality contains an important behavior (acoustic in case of V₂) it changes the relative importance of the remaining modalities. Unlike the routing model, additive models have no mechanism to re-weight how important modalities are. When a modality is unexpectedly very (un)important, a bimodal or trimodal model becomes necessary.

E. Reproducibility

Computing Resources:

All model are implemented in PyTorch and were optimized on servers with consumer-level graphic cards.

Model Information:

The validation performance, the training time, and the number of parameters for the best models as chosen based on the validation performance, are listed in Table 10.

Hyperparameter Search:

All models and datasets have the same exhaustive hyperparameter search, see Table 11. The gridsearch determined in most cases the same hyperparameter across the different optimization strategies (Joint, sMRO, and MRO), we therefore only highlight the best hyperparameters for MRO in Table 11. The performance metrics reported in Table 3 are also used to selected the best validation model.

Datasplits:

MOSI and MOSEI have an established hold-out test set, we use it for testing. SEWA has a private test set: we use the public development set for testing. IEMOCAP and Instagram have an established 5-fold test setup which we use.

Table 9:

The five segments with the largest absolute errors when predicting ${\text{TVA}}_O$ with T, V, A.

	Transcript	Non-verbal behaviors	T	V	A	${\text{TVA}}_O$	${\widehat{y}}_{\text{uni}}^{\text{human}}$
Arousal
A1	I’m gonna forget him.	gaze aversion, almost whining	−0.57	0.04	0.32	1.16	0.05
A2	What?	not attentive, quiet	−0.62	−0.83	−1.67	−2.1	−1.17
A3	Do you know how long it’s gonna take me to start all over and fill out the new form?	little movement, loud	1.11	−0.36	0.1	0.98	0.1
A4	That’s right. That’s right. I mean he would want us to, you know, celebrate the life that he- that he lived and, you know, enjoy the rest of ours as much as we can.	gaze aversion, quiet	−0.23	−1.11	−1.14	−0.17	−0.99
A5	Well, I need-I need you to be able to do this for me ‘cause I can’t do anything about it.	gaze aversion, almost whining	0.47	−0.43	0.29	0.73	0.03
Valence
V1	I mean, it’s just as hard for me, but-I know that we can do it, you know	eye-gaze aversion, fidgeting, quiet	0.25	−0.82	−0.08	−1.51	−0.41
V2	I did exactly what they told me to do.	loud, determined	−0.02	0.30	−1.45	−1.06	−0.06
V3	Oh, wow. You got in?	smile, loud/staccato- like voice	1.08	1.49	−0.86	2.04	1.08
V4	Oh, my God. That’s so dramatic.	smile, slight laughter, pitch jumps	−0.70	1.70	−0.12	1.68	0.74
V5	How can you lose my luggage like from like, in -	looking up, hand gestures	−1.39	0.88	−0.37	−0.89	−0.01

Table 10:

Validation performance (perf), training time in seconds (sec), and the number of parameters of the best validation model (params).

	Tri			Routing			Joint			sMRO			MRO
	perf	sec	params	perf	sec	params	perf	sec	params	perf	sec	params	perf	sec	params
MOSI
Sentiment	0.73	20.3	138263	0.741	68.2	556168	0.724	29.7	138263	0.735	50.8	111623	0.732	32.7	111623
MOSEI
Sentiment	0.71	97.6	141163	0.71	201.9	567768	0.713	189.9	503347	0.715	254.9	441187	0.71	171.4	114523
Polarity	0.611	99.1	441187	0.613	295.9	567768	0.617	181.8	441187	0.614	247.3	86647	0.617	290.9	441187
Happiness	0.636	87.4	441187	0.642	358.3	823208	0.638	252.6	441187	0.644	372.3	21317	0.642	380.0	503347
IEMOCAP
Arousal	0.64	41.6	111623	0.644	301.2	811608	0.643	60.3	84327	0.662	165.8	491747	0.65	69.6	4303
Valence	0.67	35.5	138263	0.66	118.8	811608	0.658	27.3	138263	0.642	64.8	22683	0.651	41.4	138263
SEWA
Arousal	0.367	174.7	22683	0.372	691.9	78518	0.388	266.3	41467	0.396	466.6	43077	0.384	484.1	43077
Valence	0.424	190.3	20283	0.427	527.3	200408	0.424	147.0	20283	0.425	461.5	11693	0.428	144.1	22683
TPOT
Constructs Instagram	0.569	105.2	80038	0.557	359.7	56036	0.568	143.4	180710	0.565	251.7	14298	0.576	243.5	14298
Intent	0.888	9.0	12098	0.757	75.7	13427	0.9	24.7	12098	0.89	24.9	12098	0.896	18.0	12098

Table 11:

Parameter search. Parameters that were determined to be best for MRO and task x are indicated by x in the superscript. We enumerate tasks in the same order as they are presented in Table 3.

Learning rate	{0.055, 0.011, 0.0052,6,10, 0.0017,8, 0.00013,4, 0.000019}
L2 weight decay	{0.1, 0.011,3,4,5,6,7,8,9,10, 0.0012, 0.0}
Hidden layers	{(5,), (10,)10, (20, 10)5,8,9, (10, 20)7, (100, 20, 10)1,2,3, (100, 100, 10)4,6}
Selection	{yes5,10, no1,2,3,4,6,7,8,9}
Feature Fusion	{concatenation3,4,7,10, tensor fusion1,2,5,6,8,9}