ICYM²I: The illusion of multimodal informativeness under missingness

Abstract

Multimodal learning is of continued interest in artificial intelligence-based applications, motivated by the potential information gain from combining different data modalities. However, modalities observed in the source environment may differ from the modalities observed in the target environment due to multiple factors, including cost, hardware failure, or the perceived informativeness of a given modality. This change in missingness patterns between the source and target environment has not been carefully studied. Naïve estimation of the information gain associated with including an additional modality without accounting for missingness may result in improper estimates of that modality’s value in the target environment. We formalize the problem of missingness, demonstrate its ubiquity, and show that the subsequent distribution shift induces bias when the missingness process is not explicitly accounted for. To address this issue, we introduce ICYM²I (In Case You Multimodal Missed It), a framework¹ for the evaluation of predictive performance and information gain under missingness through inverse probability weighting-based correction. We demonstrate the importance of the proposed adjustment to estimate information gain under missingness on synthetic, semi-synthetic, and real-world datasets.

1. Introduction

Multimodal learning is ubiquitous in machine learning as practitioners combine multiple data types to improve predictive performance in applications to healthcare (Perochon et al., 2023; Tu et al., 2024), robotics (Gao et al., 2024; Shah et al., 2023), and recommender systems (Chen et al., 2019). However, factors such as privacy concerns (Jaiswal & Provost, 2020; Zhang et al., 2021), cost-benefit tradeoffs of data-acquisition (Buck et al., 2010), and user preferences (Kossinets, 2006) imprint multimodal data with missingness. Additionally, even if modality complete data is available or curated at training, data noise (Cohen et al., 2004; Ma et al., 2023) and sensor failures (Inceoglu et al., 2021; 2023) may result in missing modalities in the target environment.

Although the missingness of modalities is a recurring challenge in real-world settings, current multi-modal machine learning methods often assume that modalities are fully observed, both in source and target environments. When missingness is considered, the literature has focused on engineering efforts (Le et al., 2025; Wu et al., 2024) such as data selection (Hosseini et al., 2022), imputation (Tran et al., 2017; Cohen Kalafut et al., 2023; Malitesta et al., 2024), and architecture design (Chen et al., 2022; Zeng et al., 2022), which implicitly assume a stable missingness process between source and target environments. When this assumption is violated, the missingness mechanism induces a distribution shift (Zhang et al., 2023; Liu et al., 2023b) that biases the estimated informativeness of a given modality. Missingness is pervasive and impacts a broad range of application domains encountered in the multimodal literature: in breast cancer screening, biopsies are only performed if there are abnormal findings in a mammogram; in autonomous vehicles, LiDAR sensor dropout can occur due to weather and lighting conditions; and in online recommender systems, reviews are only collected after certain consumer behaviors. Across these settings, ignoring the distribution shift between source and target due to missingness when quantifying modality informativeness may conflate missingness with signal, leading to flawed data collection and modeling decisions.

In this work, we propose a framework to overcome the (mis)estimation of both inherent informativeness and predictive utility under missingness in multimodal learning. Our contributions, summarized in Figure 1, are as follows:

Figure 1.

Overview of the proposed framework. Curation often discards missing data, resulting in a discrepancy between the collected $\Omega$ and source datasets ${\Omega }_{\text{source}}$ used for training. Current practice is denoted in blue: naïve training and evaluating on ${\Omega }_{\text{source}}$ leads to biased estimates of performance and informativeness on target data. The orange path illustrates the proposed ICYM²I: a double inverse probability weighting (IPW) mechanism that yields accurate performance and informativeness estimates under the target distribution.

• Framework for multimodal learning with missingness. We formalize the impact of missingness as a distribution shift intrinsic to multimodal learning, where the observed source distribution differs from the target distribution due to missingness. We show that not accounting for missingness, a common practice, may bias the estimate of a modality’s predictive and information-theoretic utility.

• ICYM²I. Under the missingness-at-random (MAR) assumption, a much more realistic assumption than the common and often implicit assumption of missingness-completely-at-random (MCAR) made by state-of-the-art multimodal strategies, we propose ICYM²I (In Case You Multimodal Missed It), a double inverse-propensity weighting correction to overcome missingness-induced distribution shifts. The proposed method does not aim to improve predictive performance; rather, it enables unbiased estimation of model performance and information gain across modalities, even under missingness. Specifically, we demonstrate that ICYM²I improves correlation in predictive and information-theoretic utility of modalities.

• Experiments on diverse data. We demonstrate the broad applicability and utility of our methods in synthetic, semi-synthetic, and real-world benchmark datasets, including a case study in multi-modal learning in health.

2. Related Work

Multimodal benchmarks suppress missingness encountered in real-world environments.

Prior work on multimodal models often assumes fully observed modalities (Ngiam et al., 2011; Zadeh et al., 2017; Hou et al., 2019). Missingness has largely been an overlooked problem (Le et al., 2025; Wu et al., 2024), to the extent that current benchmarks rarely contain samples with missing modalities. Curation often involves discarding incomplete samples, filtering them based on data-quality criteria, such as text length or file size (Sharma et al., 2018; Schuhmann et al., 2022), or imputing them using automatic tools (Miech et al., 2019). This curation implicitly assumes that rejected samples follow the same distribution as the observed ones. This assumption may not hold. For example, in autonomous driving datasets, samples with sensor failures – often caused by extreme weather or lighting conditions – may be filtered out. Models trained on complete data may consequently not generalize to these settings, posing real-world risk at deployment. When missingness is considered, previous works focused on robustness through imputation (Tran et al., 2017; Cohen Kalafut et al., 2023; Malitesta et al., 2024), representation learning (Wu et al., 2024; Liu et al., 2023a), knowledge distillation (Li et al., 2024; Wang et al., 2020a), and model ensembling (Chen et al., 2022; Zeng et al., 2022) – all ignoring the potential shift resulting from the missingness process.

Multimodal missingness in the target environment.

Prior work has explored missingness in the target environment (Lin & Hu, 2023; Zeng et al., 2022), e.g., when a captor fails at deployment (Ma et al., 2022). Broadly, two strategies have been proposed (Wu et al., 2024): (i) data preprocessing through cross-modal imputation (Cohen Kalafut et al., 2023; Malitesta et al., 2024; Tran et al., 2017), where one replaces the missing modality (Ma et al., 2021; Zhou et al., 2022), as well as (ii) model training strategies such as architecture design (Lee et al., 2023; Ge et al., 2023), distillation-based methods (Li et al., 2024; Wang et al., 2020a), and ensembling (Chen et al., 2022; Zeng et al., 2022). Through the proposed formalization, our work distinguishes between different missingness assumptions, demonstrating that the previously studied framework is only one among various plausible mechanisms for which current strategies are not well-designed.

Distribution shifts in multimodal learning.

Addressing multimodal shifts has been studied in vision-language models (Zhou et al., 2024; Verma et al., 2024) or using information-theoretic notions to understand multimodal behavior under distribution shifts (Oh et al., 2025). Augmentation and regularization strategies have been leveraged to address temporal shifts for conversation understanding (Woo et al., 2023; Lian et al., 2023). Advances in learning, such as in-context learning, have been studied to characterize adaptation to multimodal distribution shifts (Zhou et al., 2024; Xue et al., 2024). However, existing strategies aim to improve robustness under domain shifts only, while ignoring the potential shift in missingness between source and target environments. Instead, our work aims to correct estimates of performance and modality informativeness under missingness to inform modality collection at deployment.

Quantifying information-theoretic value of a modality.

Existing works often implicitly assume that additional modalities improve performance, ignoring the prohibitive cost, complexity, and potential noise added by these additional dimensions. When limited resources or constraints limit availability in the target environment, a central challenge is to quantify the information-theoretic value of a modality (Liang et al., 2024c). Liang et al. (2024a) proposed a method for recovering partial information decomposition measures of the redundancy, uniqueness, and synergy of the information provided by the different modalities (Bertschinger et al., 2014; Williams & Beer, 2010). However, these decompositions ignore the impact of missingness, a gap that our work aims to fill.

Correcting for missingness bias.

The impact of missingness in multimodal learning is understudied, in part due to the lack of formalization of missingness in this area. Ignoring this process can bias estimates of interest (Phelan et al., 2017), because the observed distribution differs from the underlying distribution that practitioners aim to model. The statistical literature has introduced strategies such as matching (Stuart, 2010) and reweighting (Jethani et al., 2022) to address missingness. We leverage these statistical tools to address the overlooked missingness in multimodal settings.

3. Multimodality and missingness

Consider two modalities, $X_1\in {𝒳}_1$ and $X_2\in {𝒳}_2$ and the state of interest $Y\in 𝒴$. We denote the joint underlying distribution $\Omega ={𝒳}_1\times {𝒳}_2\times 𝒴$. Without loss of generality, we assume an anti-causal setting for the data-generating process, in which the modalities are dependent on the states $Y$, as shown in Figure 2 (left). We use the binary indicators of missingness $M_1,M_2$, and $M_Y$, which are equal to 1 if the associated variable is missing, 0 if observed, following the convention of Mohan & Pearl (2021). Observed variables are subscripted by ‘obs’, which corresponds to the underlying modality if observed, and unobserved otherwise (denoted by $\varnothing$). Formally, the observed variable $Y_{\mathit{\text{obs}}}$ and observed modalities $X_{1,\mathit{\text{obs}}}$ and $X_{2,\mathit{\text{obs}}}$ can be defined as follows:

$$Y_{obs}=\left\{\begin{array}{ll}\varnothing & \text{if}{M}_Y=1,\\ Y& \text{otherwise.}\end{array}\right.$$

Figure 2.

Directed Acyclic Graphs of the assumed data-generating processes. On the left is the commonly assumed graph with no missingness. On the right is the proposed missingness formalism. $X_1$ and $X_2$ are two modalities of interest, $Y$ is the label of interest. The missingness process depends on $C$. Filled point nodes are observed variables, while unfilled nodes are unobserved. Gray edges indicate MAR missingness for a given modality.

In this setting, we denote the observed joint distribution ${\Omega }_{\text{obs}}:=\left(M\cdot {𝒳}_1\right)\times \left(M\cdot {𝒳}_2\right)\times (M\cdot 𝒴)$ where $M=M_1\cdot M_2\cdot M_Y$. This complete modalities analysis has been the focus of multimodal learning.

Missingness in multimodal learning.

We distinguish three mechanisms that cover potential missing modality in multimodal settings (Rubin, 1976):

• Missing Completely At Random (MCAR): A modality is missing completely at random if the missingness process is independent of any other variable.

• Missing At Random (MAR): The missingness mechanism depends on observed variables only.

• Missing Not At Random (MNAR): Missingness depends on unobserved variables.

In Figure 2 (right), we describe the missingness mechanisms as dependent on $C$, a set of covariates that determine the missingness mechanism. Note that $C$ may include one of the modalities of interest, e.g., whether $X_2$ is observed may depend on the realization of $X_1$. In general, the set $C$ may differ for each modality depending on the data-generating process.

Missingness-induced distribution shifts.

Missing modality $X_i$ and/or missing label $Y$ can induce distribution shifts between the source and the target distributions. For example, if a modality is observed in the target environment only if another one meets some criterion, then this distribution may not match the source distribution. Theoretically, we know that a non-MCAR missingness mechanism induces distribution shifts (Liu et al., 2023b; Zhang et al., 2023), i.e., the observed distribution differs from the underlying distribution. Critically, models trained and evaluated on the observed distribution are statistically biased estimates under any other missingness process.

For instance, consider an autonomous vehicle setting in which video and LiDAR are two modalities of interest. If the LiDAR sensor randomly fails, the missingness is MCAR. However, as previously mentioned, LiDAR may be more likely to malfunction under extreme weather conditions. If these conditions can be extracted from the video modality, one may assume MAR patterns. However, if the video cannot capture the variable that explains the LiDAR dysfunction (e.g., temperature), then LiDAR would be MNAR, as it is dependent on an unobserved variable. In the last two scenarios, focusing solely on complete samples would exclude all extreme-condition settings with critical real-world repercussions.

A common and often implicit assumption in the multimodal literature is the absence of missingness, which corresponds to either a MCAR mechanism (${\Omega }_{\text{obs}}=\Omega$) or a stable missingness process between the source and target environments, i.e., the source and target distributions are the same (${\Omega }_{\text{obs}}^{\text{source}}={\Omega }_{\text{obs}}^{\text{target}}$). In other words, not adjusting for missingness assumes that the missingness process is uninformative or will remain the same in the target environment.

Prior works focus on improving the robustness of multimodal models when performance may degrade due to a modality missing in the target distribution, i.e, the source distribution reflects the true distribution while the target may present missingness ${\Omega }_{\text{obs}}^{\text{source}}~\Omega \ne {\Omega }_{\text{obs}}^{\text{target}}$.

Our work questions the applicability of these assumptions where modality collection is costly. While missingness may result in a distributional shift (Zhou et al., 2023), we emphasize that demonstrating the value of a modality in the source environment may lead to increased collection of this modality in the target environment, inducing a distribution shift akin to Assumption A.

Assumption A (Multimodal analysis informs data collection). Demonstrated multimodal performance gain induces a shift in the missingness process in the target, i.e. ${\Omega }_{\mathit{\text{obs}}}^{\mathit{\text{source}}}\ne {\Omega }^{\mathit{\text{target}}}~\Omega$.

We focus on settings in which historical data used to train a model are marked by missingness. Under such settings, we aim to: (i) identify which modalities are informative and may consequently be collected in the target environment, and (ii) train models using the observed source data that generalize to the target environment where the modalities are fully observed.

4. Is this modality informative?

We aim to assess whether a partially missing modality would be informative if fully observed. To this end, we introduce ICYM²I (In Case You Multimodal Missed It), a framework for correcting model performance trained on modality complete samples where all modalities and labels are observed (${\Omega }_{\text{obs}}$) to estimate the predictive utility of the partially missing modality if it were observed for the whole population ($\Omega$). Additionally, we propose a correction to derive unbiased estimates of the information-theoretic utility of a modality, using ${\Omega }_{\text{obs}}$. We rely on Partial Information Decomposition (PID) (Williams & Beer, 2010) bounds introduced by Bertschinger et al. (2014) for this task, which quantifies the information value of a target of interest captured by two input variables.

Correction.

We propose an Inverse Probability Weighting (IPW) approach (Robins et al., 1994), which reweights samples according to their probability of being observed. Under Assumption B that relaxes the common MCAR assumption made in the multimodal literature, IPW recovers unbiased estimates of the true distribution, enabling learning and evaluation on the true distribution from observed samples. IPW-adjustment is critical for both training and evaluation of multimodal models under missingness. IPW-adjusted training results in a model trained to infer on the underlying distribution $\Omega$, while correction of the evaluation allows for measuring performance on $\Omega$, despite evaluating the model only on samples from the observed distribution ${\Omega }_{\text{obs}}$.

Assumption B (MAR and Positivity). The missingness mechanism is MAR, and $p_{\Omega }\left(M_1=0,M_2=\right.\left.0,M_Y=0\mid C\right)>0$.

4.1. A motivating example

We consider the common multimodal example of learning bit-wise logic operators (Bertschinger et al., 2014; Harder et al., 2013; Liang et al., 2024a). We generate 10, 000 points with two modalities drawn from Bernoulli distributions ($p=0.5$). The output state $Y$ is defined using the binary operators AND, OR, and XOR of input bits $X_1$ and $X_2$. In this setting, we induce missingness $M_2$ in $X_2$ and $Y$ as a function of $X_1$ (MAR): $M_2~\mathit{\text{Bern}}\left(0.6X_1+0.2\right)$, resulting in 50% missingness in $X_2$. We investigate the impact of missingness on current strategies for evaluating the predictive and information-theoretic utility of a given modality.

Estimating performance for informativeness.

A common practice to measure the predictive value of adding a modality is through ablation studies where practitioners train models using different modalities on the subset of samples where all modalities are observed (${\Omega }_{\text{obs}}$). First, unimodal models $f\left(x_i\right)$ are trained to approximate $p_{{\Omega }_{\text{obs}}}\left(y\mid x_i\right),\forall i\in \{1,2\}$ and a multimodal model $f\left(x_1,x_2\right)$ to approximate $p_{{\Omega }_{\text{obs}}}\left(y\mid x_1,x_2\right)$ on the same observed dataset. The performance is then compared in a hold-out set, sampled from ${\Omega }_{\text{obs}}$. The informativeness of a modality is measured by the relative performance gain of the multimodal model over the unimodal model. However, multimodal models can underperform their unimodal counterparts due to data characteristics (Zhang et al., 2024) and learning dynamics (Wang et al., 2020b; Zhai et al., 2024). Thus, relying solely on performance as a proxy for informativeness, particularly under distribution shifts, can be misleading.

Partial Information Decomposition (Bertschinger et al., 2014).

As an alternative to estimating performance, existing works have decomposed the informativeness associated with each modality (Liang et al., 2024a). Bertschinger et al. (2014) formalized this decomposition by analyzing the total (three-way) mutual information $I\left(Y:\left(X_1,X_2\right)\right)$ (McGill, 1954; Te Sun, 1980), a measure of dependency between the target variable $Y$ and the modalities ($X_1,X_2$). This quantity can be decomposes into four components: shared information (information both $X_1,X_2$ share about $Y$), unique information 1 (information only $X_1$ has about $Y$), unique information 2 (information only $X_2$ has about $Y$), and complementary information (information about $Y$ that requires both $X_1$ and $X_2$) as follows:

$$I\left(Y:\left(X_1,X_2\right)\right)=\underset{\text{shared information}}{\underbrace{SI\left(Y:X_1;X_2\right)}}+\underset{\text{unique information}1}{\underbrace{UI\left(Y:X_1\setminus X_2\right)}}+\underset{\text{unique information}2}{\underbrace{UI\left(Y:X_2\setminus X_1\right)}}+\underset{\text{complementary information}}{\underbrace{CI\left(Y:X_1;X_2\right)}}$$

Bertschinger et al. (2014) specifies how to estimate these quantities (see Appendix B); for instance, the unique information between $Y$ and $X_1$ can be estimated using the following:

$$\widetilde{UI}\left(Y:X_1\setminus X_2\right)=\underset{q\in {\Delta }_{\Omega }}{\text{min}}\left[I_q\left(Y:\left(X_1,X_2\right)\right)-I_q\left(Y:X_2\right)\right],$$

where ${\Delta }_{\Omega }$ is the set $q\left(X_i=x_i,Y=y\right)=p_{\Omega }\left(X_i=x_i,Y=y\right)\forall x_i\in {𝒳}_i,y\in 𝒴,i\in \{1,2\}$, that is, the set of joint distributions over ($X_1,X_2,Y$) that match the true two-way data distributions. Note that the objective function requires minimization with respect to the three-way mutual information. Importantly, Bertschinger et al. (2014) demonstrates that the solution to any one objective specifies an optimum solution for all other quantities.

Prior work that relies on this Partial Information Decomposition (PID) to attribute information-theoretic value implicitly assumes that ${\Omega }_{\text{obs}}^{\text{source}}={\Omega }^{\text{source}}={\Omega }^{\text{target}}=\Omega$. Instead, we evidence the limitations of these strategies performed on ${\Omega }_{\text{obs}}^{\text{source}}\ne \Omega$ when the target decomposition is ${\Omega }^{\text{target}}=\Omega$, i.e., the true data-generating mechanism.

As a motivating example, we analyze the impact of missingness on estimating PID for unidimensional modalities with a bitwise-logic outcome (AND, OR, and XOR). Table 1 (left) presents the discriminative performance associated with neural networks trained on each individual modality and their combination under three scenarios: (i) access to all data (Oracle), (ii) focusing only on datapoints with all covariates observed (Observed), and (iii) adequately accounting for missingness (ICYM²I using IPW to adjust ${\Omega }_{\text{obs}}\mapsto \Omega$, by modeling the missingness mechanism), as proposed in Section 4.2. Table 1 (right) presents PID, discussed in Section 4.3 under the same scenarios, demonstrating how information decomposition is also biased due to missingness.

Table 1.

Impact of missingness on multimodality information for bitwise logic operators. Parentheses denote standard deviation across batches.

		AUROC			Information Decomposition
		$X_1$	$X_2$	$X_1+X_2$	Unique 1	Unique 2	Shared	Complementary
AND	Oracle	0.83 (0.01)	0.84 (0.01)	1.00 (0.00)	0.05 (0.00)	0.03 (0.00)	0.26 (0.00)	0.47 (0.00)
	Observed	0.66 (0.01)	0.93 (0.01)	1.00 (0.00)	0.44 (0.00)	0.00 (0.00)	0.15 (0.00)	0.36 (0.00)
	ICYM2I	0.83 (0.01)	0.85 (0.02)	1.00 (0.00)	0.03 (0.00)	0.03 (0.00)	0.27 (0.00)	0.45 (0.00)
OR	Oracle	0.84 (0.01)	0.83 (0.01)	1.00 (0.00)	0.04 (0.00)	0.05 (0.00)	0.27 (0.00)	0.46 (0.00)
	Observed	0.95 (0.01)	0.77 (0.01)	1.00 (0.00)	0.01 (0.00)	0.15 (0.00)	0.10 (0.00)	0.23 (0.00)
	ICYM2I	0.85 (0.02)	0.82 (0.01)	1.00 (0.00)	0.03 (0.00)	0.02 (0.00)	0.27 (0.00)	0.50 (0.00)
XOR	Oracle	0.51 (0.02)	0.49 (0.01)	1.00 (0.00)	0.00 (0.00)	0.00 (0.00)	0.00 (0.00)	0.99 (0.00)
	Observed	0.52 (0.02)	0.80 (0.02)	1.00 (0.00)	0.34 (0.00)	0.07 (0.00)	−0.07 (0.00)	0.62 (0.00)
	ICYM2I	0.53 (0.03)	0.49 (0.03)	1.00 (0.00)	0.00 (0.00)	0.00 (0.00)	0.01 (0.00)	0.96 (0.00)

Specifically, relying on ${\Omega }_{\text{obs}}$ overestimates the performance of $X_1$ for OR but underestimates it for AND. Similarly, biased decomposition overestimates the informativeness of $X_1$ (“Unique 1” compared to “Unique 2”) for OR. As $X_1$ informs the missingness process, it indirectly informs the outcome of interest, despite the true underlying generative process being dependent on both. The use of IPW can correct for such bias under positivity, provided the IPW propensities can be estimated (i.e., the MAR assumption). We study sensitivity to this assumption in Appendix D, where we further evaluate the robustness of our method under MCAR and MNAR, demonstrating robustness under MCAR.

We now formally describe two methods for reliably inferring the informativeness of modalities using (i) unbiased estimation of unimodal versus multimodal model performance using supervised learning (ICYM²I-learn), and (ii) high-dimensional autodifferentiable partial information decomposition (ICYM²I-PID). In addition, we demonstrate the need for IPW-adjusted evaluation as a key element to determine modality informativeness using supervised learning.

4.2. ICYM²I-learn: Estimating predictive performance under missingness

Training.

Under the MAR assumption, i.e., the missingness is fully explained by observed covariates $C$; that is, the probability of a data point being missing depends only on $C$, we propose to train the model with a weighted loss using samples from ${\Omega }_{\text{obs}}$. The proposed IPW-adjusted loss accounts for the distributional shift (${\Omega }_{\text{obs}}\mapsto \Omega$) by up-weighting under-observed samples, as described in the following lemma.

Lemma 1 (IPW Training). The loss function computed on the observed data $l_{{\Omega }_{\mathit{\text{obs}}}}\left(x_1,x_2,y\right)$ can be reweighted to approximate the target loss $l_{\Omega }\left(x_1,x_2,y\right)$ as follows:

$$l_{\Omega }\left(x_1,x_2,y\right)=\frac{1}{1-p\left(m_1,m_2,m_y\mid C\right)}{l}_{{\Omega }_{obs}}\left(x_1,x_2,y\right)$$

where $p\left(m_1,m_2,m_y\mid C\right)$ is the probability of missingness, given the covariates $C$.

Evaluation.

Existing work suffers from an analogous bias in model evaluation, by relying on a hold-out set from the observed distribution (${\Omega }_{\text{obs}}$). To estimate a given metric under the true underlying distribution, one must correct this metric using a similar correction as previously described. For instance, Li et al. describes how to correct for both AUC and Brier score using IPW.

Corollary 1 (ICYM²I-learn). Consider a model $f$ trained and evaluated on data drawn from ${\Omega }_{\mathit{\text{obs}}}$. To correct the model and estimate its performance on $\Omega$, one must correct both its training and evaluation following the previous corrections.

4.3. ICYM²I-PID: Partial information decomposition for multimodal informativeness

Obtaining the partial information decomposition bounds corresponds to estimating the distribution $q\in {\Delta }_{\Omega }$ that minimizes three-way mutual information (see Bertschinger et al. (2014), Lemma 4), where ${\Delta }_{\Omega }$ is the set of all distributions over ($X_1,X_2,Y$) such that the two-way joints between $X_i$ and $Y$ match the true data-generating distribution. Formally, one can focus on solving:

$$\underset{q\in {\Delta }_{\Omega }}{\text{min}}{I}_q\left(Y:\left(X_1,X_2\right)\right)$$

To operationalize this minimization, Liang et al. (2024a) proposes to minimize this quantity using the observed samples:

$${\Delta }_{\Omega }\approx \left\{q\propto \text{exp}\left(f_1\left(x_1\right)\cdot f_2\left(x_2\right)\right):q\left(x_i,y\right)=p_{{\Omega }_{\text{obs}}\varphi }\left(x_i,y\right)\forall x_i\in {𝒳}_i,y\in 𝒴,i\in \{1,2\}\right\}$$

where $p_{{\Omega }_{\text{obs}}\varphi }$ is a re-parametrization of ${\Omega }_{\text{obs}}$ using neural networks. To enforce the marginals of $q$ to match $p_{{\Omega }_{\text{obs}}}$, Liang et al. (2024a) proposes an iterative process using a Sinkhorn–Knopp procedure (Knight, 2008).

Under missingness, we observe samples from ${\Omega }_{\text{obs}}$ rather than $\Omega$. To unbias the PID measures in this setting requires adjusting for the ${\Omega }_{\text{obs}}\mapsto \Omega$ shift. Effectively, our approach corrects the previous optimization to account for the distribution shift induced by MAR missingness.

First, we introduce a corrected mutual information computation to ensure that we optimize an unbiased estimate of the three-way mutual information under $\Omega$ using samples from ${\Omega }_{\text{obs}}$ (see Appendix A for the complete proof).

Lemma 2 (Corrected mutual information).

$$I_{\Omega }^{\text{IPW}}\left(Y:\left(X_1,X_2\right)\right)={\mathbb{E}}_{\begin{array}{c}{x}_1,x_2~p_{{\Omega }_{\mathit{\text{obs}}}}\left(x_1,x_2\right)\\ y~p_{\Omega }\left(y\mid x_1,x_2\right)\end{array}}\left[\frac{1-p\left(m_1,m_2\right)}{1-p\left(m_1,m_2\mid x_1,x_2,y\right)}\text{log}\left(\frac{p_{\Omega }\left(x_1,x_2,y\right)}{p_{\Omega }\left(x_1,x_2\right)p_{\Omega }\left(y\right)}\right)\right]$$

Then, ICYM²I-PID uses the following projection set to operationalize PID-bound estimation while accounting for missingness:

$$\begin{gathered} {{\Delta }_{\Omega }}^{{\text{ICYM}}^2\text{I}}\approx \left\{q\propto \text{exp}\left(f_1\left(x_1\right)\cdot f_2\left(x_2\right)\right):q\left(x_i,y\right)=p_{{\Omega }_{\varphi }}\left(y,x_i\right)\forall x_i\in {𝒳}_i,y\in 𝒴,i\in \left\{1,2\right\}\right\} \\ =\left\{q\propto \text{exp}\left(f_1\left(x_1\right)\cdot f_2\left(x_2\right)\right):q\left(x_i,y\right)={\text{IPW}}_{p_{\Omega \varphi }}\left(p_{{\Omega }_{\text{obs}\varphi }}\left(y,x_i\right)\right)\forall x_i\in {𝒳}_i,y\in 𝒴,i\in \left\{1,2\right\}\right\} \end{gathered}$$

where ${\text{IPW}}_q\left(p\right)$ is an IPW correction for the shift $p\mapsto q$ using samples from $p$ and $p_{\varphi }$ is a re-parametrization of $\Omega$ using neural networks and learned using samples from ${\Omega }_{\text{obs}}$ via the weighted loss introduced in Lemma 1. Importantly, the proposed correction is agnostic to the parametrisation of $q$. We note that these mutual information-based quantities can be equivalently estimated using entropy-based measures (see derivations in Appendix B).

Estimating PID under missingness therefore consists in optimizing:

$$\underset{q\in {\Delta }_{\Omega }^{\text{ICY}{\text{M}}^2\text{I}}}{\text{min}}{I}_q^{\text{IPW}}\left(Y:\left(X_1,X_2\right)\right)$$

Similarly to Liang et al. (2024a), the unbiased PID estimation framework consists of the following steps (detailed in Appendix C):

1. Model the missingness mechanism. Train a model to estimate the probability of missingness given $C$ to obtain IPW weights for correcting the distribution shift.

2. Train corrected unimodal and multimodal models. Train each model ($f_1\left(x_1\right),f_2\left(x_2\right)$, and $f\left(x_1,x_2\right)$) using the IPW-corrected loss introduced in Lemma 1. These models can be trained with flexible inductive biases tailored to the application at hand.

3. Solve PID optimization. Estimate $q\in {\Delta }_{\Omega }^{{\text{ICYM}}^2\text{I}}$ minimizing $I_q\left(Y:\left(X_1,X_2\right)\right)$, where $q$ is parameterized by the product of two unimodal networks. To match the marginal of $q$ to the estimated probabilities of the corrected unimodal neural networks, we apply a modified Sinkhorn–Knopp procedure (Knight, 2008) using IPW-corrected unimodal distributions². Our framework is agnostic to the choice of parametrization of $q$, provided that calibrated probabilistic scores are learned.

4. Estimate the PID components. Given $q$, compute PID quantities using the bounds of Bertschinger et al. (2014), corrected via the IPW-correction introduced in Appendix A.

5. Experiments

To better understand the connection between performance, information decomposition, and missingness, we propose a simulation (detailed in Appendix E), two semi-synthetic studies that reflect real-world missingness mechanisms (see Appendix F), and a real-world case study.

5.1. Simulation and semi-synthetic experiments

In Table 2, each point represents the estimated performance value for one simulation under the training and evaluation IPW-corrections and the oracle performance, i.e., a model trained and tested on $\Omega$. Specifically, columns reflect evaluation correction, while rows reflect training correction. These results underline the importance of correcting both training and evaluation, as proposed in ICYM²I, to best align with the performance one would obtain on $\Omega$, as shown by the smallest Root Mean Squared Error (RMSE) observed when both corrections are applied. This observation shows that the proposed ICYM²I best estimates the performance associated with each modality, despite relying on ${\Omega }_{\text{obs}}$. Appendix E echoes the same observation when estimating PID.

Table 2.

Comparison between estimated AUC performance under the different training and evaluation corrections and oracle performance on $\Omega .ϵ$ denotes the RMSE between estimated and oracle AUCs.

The semi-synthetic experiments examine the effect of enforcing increasing missingness on the performance and information decomposition of UR-FUNNY (Hasan et al., 2019) and hateful memes (Kiela et al., 2020), two foundational real-world datasets used in the multimodal literature for affective computing and content moderation. Table 3 summarizes the effect of enforcing 70% missingness on estimating multimodality informativeness across these datasets, demonstrating the generalizability of our proposed strategy across real-world datasets. Appendix F further illustrates the robustness of the methodology under different levels of missingness in these datasets and explores MNAR patterns.

Table 3.

Impact of 70% missingness on multimodality information for UR-FUNNY (Hasan et al., 2019) and Hateful Memes (Kiela et al., 2020). Parentheses denote standard deviation across batches.

		AUROC			Information Decomposition
		Text	Image/Video	Image + Text	Uniquetext	Uniqueimage	Shared	Complementary
UR-FU.	Oracle	0.68 (0.01)	0.60 (0.02)	0.69 (0.02)	0.10 (0.00)	0.02 (0.00)	0.00 (0.00)	0.00 (0.00)
	Observed	0.61 (0.03)	0.54 (0.04)	0.63 (0.03)	0.05 (0.00)	0.00 (0.00)	0.03 (0.00)	0.00 (0.00)
	ICYM2I	0.66 (0.03)	0.57 (0.04)	0.62 (0.04)	0.07 (0.00)	0.00 (0.00)	0.00 (0.00)	0.00 (0.00)
Memes	Oracle	0.71 (0.01)	0.57 (0.01)	0.72 (0.01)	0.09 (0.01)	0.00 (0.00)	0.04 (0.00)	0.05 (0.01)
	Observed	0.68 (0.02)	0.61 (0.02)	0.71 (0.02)	0.13 (0.00)	0.04 (0.00)	0.01 (0.00)	0.00 (0.00)
	ICYM2I	0.67 (0.02)	0.61 (0.02)	0.71 (0.02)	0.10 (0.00)	0.01 (0.00)	0.02 (0.03)	0.03 (0.01)

5.2. Chest radiographs are uninformative over electrocardiograms for structural heart disease detection.

While our core contribution is methodological, this section illustrates how ignoring missingness can lead to biased estimates of the informativeness of a given modality in a real-world setting where modalities are commonly missing. Specifically, we study structural heart disease (SHD), a set of conditions that affect the heart’s physiology, which is typically diagnosed using transthoracic echocardiograms (TTEs) (Writing Committee Members et al., 2021). However, TTEs are often underutilized in the United States due to diagnostic stewardship and competing financial incentives (Papolos et al., 2016). Prior work using unimodal models with common modalities in electrocardiograms (ECGs) (Elias et al., 2022; Ulloa-Cerna et al., 2022) and chest radiographs (CXRs) (Bhave et al., 2024) has demonstrated that non-TTE modalities can detect structural heart disease labels. However, CXRs are not systematically collected in conjunction with ECGs, leading to systematic missingness patterns. We therefore evaluate ICYM²I on this clinical task to assess the informativeness of CXRs for diagnosing SHD, despite their missingness.

Dataset.

Our study population comprises a retrospective cohort of 98,397 adult patients who received an ECG and a TTE within 1 year of each other. The population has 20.56% SHD prevalence. In this cohort, 12,587 patients (12.79%) have recorded CXRs. For subjects with multiple echocardiograms, we select the first TTE to model opportunistic screening with non-TTE modalities. All data were collected from Columbia University Medical Center, a large academic medical system in New York City, between 2008 and 2022. Data are split temporally, with subjects with TTEs collected on or after 2018 ($n=\mathrm{40,734}$) allocated to the test set. All data were de-identified, retrospective, and collected for clinical purposes within an academic hospital system, with Institutional Review Board approval. Appendix G contains further details regarding preprocessing, embedding generation, and the ICYM²Iimplementation.

Results.

Table 4 presents the performance of each uni- and multimodal model, along with the associated information decomposition. While both the observed and corrected analyses demonstrate the importance of ECG in modeling SHD, the corrected results raise questions about the information gain associated with CXR. Naive decomposition suggests the unique information in CXRs accounts for about 5% of the total information. However, ICYM²I reduces this unique contribution to 1.8% while increasing estimates of shared information between ECG and CXRs for SHD detection. In contrast to domain knowledge, where ECGs capture electrophysiology while CXRs capture structure and anatomy, two distinct aspects of cardiac health, the corrected complementary and shared results, and low unique information of CXRs suggest that CXRs are not independently useful for SHD diagnosis. Note that our results indicate that the multimodal model performs slightly worse than the unimodal ECG model, reflecting the potential risk of overfitting associated with a large number of features.

Table 4.

Informativeness of ECG and CXR modalities on model-based structural heart disease detection. Parentheses denote standard deviation across batches ($n=1024$).

	AUROC			Information Decomposition
	ECG	CXR	ECG + CXR	UniqueECG	UniqueCXR	Shared	Complementary
Observed	0.83 (0.01)	0.72 (0.02)	0.82 (0.01)	0.11 (0.00)	0.01 (0.00)	0.10 (0.00)	0.00 (0.00)
ICYM2I	0.82 (0.01)	0.73 (0.02)	0.83 (0.01)	0.07 (0.00)	0.01 (0.00)	0.48 (0.00)	0.01 (0.00)

6. Discussion

This work formalizes the issue of missing modalities in multimodal settings. We emphasize that existing work commonly overlooks missingness by discarding samples with any missing modality during curation, or implicitly assumes that the missingness mechanism remains stable when a model is deployed in the target environment. Our work formalizes this problem and demonstrates its ubiquity in the multimodality literature. Most critically, prior work ignores that any perceived informativeness of a modality may result in increased rates of data collection, inducing different missingness patterns at deployment. Our work, therefore, introduces ICYM²I, a correction to estimate the information gain associated with a partially observed modality. Our results demonstrate the methodology’s capacity to correct for biases introduced by missingness across synthetic, semi-synthetic, and real-world multimodal datasets. Finally, we demonstrate the practical utility of this methodology on a healthcare dataset, showing the divergent conclusions that one would reach if missingness were ignored. Our work highlights the critical importance of missingness in multi-modal research and urges practitioners to pay particular attention to this issue by systematically collecting data with incomplete modalities and carefully modeling and accounting for missingness to enhance robustness.

Limitations.

The key assumption in our work is that missingness is MAR. No theoretical guarantees exist under MNAR patterns. While distinguishing these assumptions is empirically untestable, practitioners should ensure that this assumption is appropriate for their data. Importantly, MAR is less restrictive than the implicit MCAR assumption made in the multimodal literature, and does not require distributional assumptions that one must assume to tackle MNAR patterns. Additionally, our work is based on Partial Information Decomposition (PID) (Bertschinger et al., 2014), which focuses on two input modalities. In practice, practitioners could consider a one-vs-all approach to inform modality informativeness using our method. However, extending the decomposition to more than two modalities remains an open challenge (Griffith & Koch, 2014; Kolchinsky, 2022), where notions of mutual information are not well-defined beyond three-way mutual information. As in prior work on PID-based measures of information gain on high-dimensional data (Liang et al., 2023; 2024a), the quality of the representations used may impact the measures returned by ICYM²I. We ensure that our probabilistic estimates are calibrated to mitigate such challenges. Additionally, PID is designed for datasets with paired modalities, where the input modalities are associated with an instance (e.g., a patient and their associated chest X-ray and electrocardiogram); consequently, our method does not extend to unpaired settings. Finally, our proposed method relies on a notion of instance for which all modalities are observable. Extending this method when there is no notion of an instance, i.e., unaligned modalities, could be considered but would require different inductive biases to model the underlying unimodal and multimodal probabilities.