Representation Retrieval Learning for Heterogeneous Data Integration

Abstract

In the era of big data, large-scale, multi-source, multi-modality datasets are increasingly ubiquitous, offering unprecedented opportunities for predictive modeling and scientific discovery. However, these datasets often exhibit complex heterogeneity, such as covariates shift, posterior drift, and blockwise missingness, which worsen predictive performance of existing supervised learning algorithms. To address these challenges simultaneously, we propose a novel Representation Retrieval ($R^2$) framework, which integrates a dictionary of representation learning modules (representer dictionary) with data source-specific sparsityinduced machine learning model (learners). Under the $R^2$ framework, we introduce the notion of integrativeness for each representer, and propose a novel Selective Integration Penalty (SIP) to explicitly encourage more integrative representers to improve predictive performance. Theoretically, we show that the excess risk bound of the $R^2$ framework is characterized by the integrativeness of representers, and SIP effectively improves the excess risk. Extensive simulation studies validate the superior performance of $R^2$ framework and the effect of SIP. We further apply our method to two real-world datasets to confirm its empirical success. Supplementary materials for this article are available online, including a standardized description of the materials available for reproducing the work.

1. Introduction

Large-scale data integration has made transformative contributions across numerous fields, including computer vision, natural language processing, biomedicine, genomics and healthcare. For example, in biomedicine, integrating randomized clinical trials and observational studies is of great interest, as it borrows information from both data sources (Yang and Ding 2020; Shi, Pan, and Miao 2023; Colnet et al. 2024). In genomics, multi-modality and multi-batch assays enable the discovery of cellular heterogeneity and development (Kriebel and Welch 2022; Du, Cai, and Roeder 2022). In healthcare, multiple types of time-series measurements, such as cardiovascular, physical activities, and sleep data can be integrated to improve real-time health and well-being monitoring (Zhang et al. 2024; Sun, Zhao, and Xue 2025; Rim et al. 2025). However, integration of large-scale data effectively remains challenging, particularly when data are collected from diverse sources or populations, and across various variables and modalities.

Integrating large-scale data is particularly challenging due to various types of heterogeneity. First, the marginal distributions of the same modality are often heterogeneous across different sources or populations, a phenomenon called “distribution heterogeneity”, or “covariates shift” in the literature (Shimodaira 2000). Second, in the context of supervised learning, the conditional distribution of responses given covariates could be heterogeneous, which is named “posterior heterogeneity”, or “posterior drift” (Widmer and Kubat 1996). Third, observed covariates or modalities are often not uniformly measured: some covariates are observed across all data sources, while others are observed in only partial data sources. We refer to this as “observation heterogeneity” or “blockwise missingness” (Yu et al. 2020; Xue and Qu 2021).

In this work, we propose a unified framework to incorporate all three types of heterogeneity in large-scale data integration. Specifically, we introduce the Representation Retrieval $\left(R^2\right)$ framework, which constructs a dictionary of representation learning modules (representer), such as neural nets (Goodfellow et al. 2016) and smoothing function bases (Ravikumar et al. 2009), to capture the complex distribution across all data sources. Then, each data source is allowed to retrieve relevant and informative representers from the dictionary for prediction to accommodate covariates shift among data sources. Finally, source-specific machine learning algorithms (learner) are employed to predict responses based on the retrieved latent representations, such that the posterior heterogeneity is incorporated. This learning algorithm is named as Representation Retrieval ($R^2$) learning hereafter. Further, we extend $R^2$ learning to Blockwise Representation Retrieval ($B{R}^2$) learning to handle the observation heterogeneity. In particular, we introduce modality-specific representer dictionaries and aggregate retrieved representer from each modality for prediction. This approach exploits all available observed data and does not require missing imputation, which is robust against untestable missing mechanism assumptions.

Moreover, we introduce a new concept of integrativeness of representers, defined as the number of data sources retrieving the specific representer. Theoretically, we provide a precise excess risk bound of the $R^2$ framework, and reveals that the upper bound is explicitly controlled by integrativeness of represeneters. Motivated by this phenomenon, we propose a novel Selective Integration Penalty (SIP), and explicitly encourage more integrative representers to improve prediction performance. Extensive simulation studies and real-world applications also confirm the superior empirical performance of our proposed method.

1.1. Related Works

In the current literature, various approaches for handling heterogeneous data have been proposed, while most of them only address one or two types of the aforementioned heterogeneity. We review the related works as follows.

Covariates shift is encountered frequently in data integration, and has been studied extensively in the literature. Related methods can be summarized into two categories: first, joint-and-individual structure, assuming a joint component that shared by all data sources, while each data source has its unique component to capture the individual information not explained by the joint component. Some representative works include (Lock et al. 2013; Klami et al. 2014; Bunte et al. 2016; Feng et al. 2018; Tang and Allen 2021; Lin and Zhang 2023; Wang et al. 2024). Second, partially sharing structure (Gaynanova and Li 2019; Lock et al. 2022; Xiao and Xiao 2024) is a more flexible framework that each component can be shared by any numbers of data sources, to accommodate greater heterogeneity. In the latent space modeling, a shared latent space is assumed and each data source has different projection onto this shared latent space (Klami et al. 2014; Bunte et al. 2016; Tang and Allen 2021), which fits into the joint-and-individual structure. In contrast, our method assumes a shared dictionary and allows each data source to retrieve any representers from the dictionary, following the partially sharing structure. Therefore, our method is more adaptive to greater heterogeneity in covariates shift. Instead of uncovering the partially sharing structure in unsupervised learning as in (Gaynanova and Li 2019; Lock et al. 2022; Xiao and Xiao 2024), our proposed method actively encourages partially sharing structure via SIP to improve predictive performance.

Posterior heterogeneity has been extensively studied in the literature of transfer learning and multi-task learning. There are various structural assumptions imposed on model parameters. For instance, regression coefficients across tasks are assumed to be similar, and can be achieved by a fusion-type penalty (Tang and Song 2016), or shrinking model parameters to a prototype (Duan and Wang 2023). Other works consider more complex structures, such as latent representation (Tian, Gu, and Feng 2023), angular similarity (Gu, Han, and Duan 2024), or partially sharing structure (Behdin et al. 2025). In these parametric models, parameters are stringent to be comparable across data sources, implying that models for all sources belong to the same function class. In contrast, our method can include representers of varying complexity and each data source can retrieve informative ones to maximize their prediction accuracy.

Observation heterogeneity, or blockwise missingness often has been treated separately from the other two heterogeneity. In multi-view learning with missing views (Xu, Tao, and Xu 2015; Zhang et al. 2018; Fan et al. 2023; Xie et al. 2023), the primary target is to impute missing views by leveraging some structural assumptions, such as low-rank structure (Cai, Cai, and Zhang 2016; Zhou, Cai, and Lu 2023). Other works target prediction with the presence of missing modalities (Yuan et al. 2012; Yu et al. 2020; Xue and Qu 2021; Yu and Hou 2022; Song, Lin, and Zhou 2024; Sell et al. 2024; Sui et al. 2025; Ma, Wang, and Samworth 2025). These methods can be summarized into imputation (Yuan et al. 2012; Xue and Qu 2021; Sui et al. 2025; Ma, Wang, and Samworth 2025) and imputation-free (Yu et al. 2020; Yu and Hou 2022; Song, Lin, and Zhou 2024; Sell et al. 2024) methods. In general, imputation approaches impose more stringent requirement on missing mechanisms than imputation-free approach. When the assumed missing mechanism is violated, or imputations are biased, imputation approaches could suffer from degraded performance. Our method is imputation-free that uses all observed data, and robust against missing mechanism misspecification.

1.2. Organization

The rest of article is structured as follows: In Section 2, we introduce the notations used throughout the article and elaborate the problem setup in this work. In Section 3.1, we first present the Representation Retrieval ($R^2$) learning for multi-task learning problem. Then, we present the notion of integrativeness and introduce the Selective Ingration Penalty (SIP) in Section 3.2. The extension to Blockwise Representation Retrieval $\left(B{R}^2\right)$ learning is elaborated in Section 3.3. Theoretical analyses of the $R^2$ framework are provided in Section 4, which recalls the important role of SIP in improving predictive performance. Afterwards, we investigate the empirical performance of our $R^2$ and $B{R}^2$ learning with simulated data in Section 5, and apply our method to two real data examples to demonstrate its superior performance in Section 6. Summary and discussion are provided in Section 7.

2. Preliminaries

In this section, we introduce our problem setup with essential notations. The related background is also introduced as a prelude to the proposed method.

In this article, we focus on integrating multi-source and multi-modality data for supervised learning problems, such as classification and regression. Suppose we collect $S$ data sources and $M$ modalities: ${𝒟}_s=\left({\mathbf{X}}^{\left(s\right)},{\mathbf{y}}^{\left(s\right)}\right)$, where ${\mathbf{X}}^{\left(s\right)}\in {𝒳}^{\left(s\right)}\subset {\mathbb{R}}^{p_s}$ denotes observed modalities, and ${\mathbf{y}}^{\left(s\right)}\in {𝒴}^{\left(s\right)}$ denotes responses or outcomes. Note that ${𝒳}^{\left(s\right)}$ lies in ${\mathbb{R}}^{p_s}$ where the dimensionality $p_s$ depends on the observing modalities of the $s$th data source. More detailed notations for multi-modality covariates will be introduced in Section 3.3. The response spaces ${𝒴}^{\left(s\right)}$’s are source-specific to accommodate heterogeneous responses, for example, ${𝒴}^{\left(s\right)}=\{\mathrm{1,2},\dots ,\}$ for classification problems and ${𝒴}^{\left(s\right)}\in \mathbb{R}$ for regression problems. A straightforward illustration of our setup is provided in the upper block of Figure 1. As noted in Introduction, our problem setup encompasses a broad spectrum of problem setups, including multi-task learning and blockwise missing learning as special cases. Our target is to learn function mappings $f^{\left(s\right)}:{𝒳}^{\left(s\right)}\to {𝒴}^{\left(s\right)}$ for $s=1,\dots ,S$. Although it is valid to learn $f^{\left(s\right)}$ from the sth data source solely, here we seek to integrate shared information across different data sources and modalities to strengthen predictive performance for each source.

Figure 1.

Targeted problem setup and special cases. The upper block illustrates our problem setup: different colors represent different covariates modalities, and white blocks denote missing modalities. Different shade levels of the same color indicate different distributions of the same covariates. The lower left block shows the multi-task learning setup, which considers only one covariate modality and ignores observation heterogeneity. The lower right block illustrates block-wise missing learning, involving multi-modality data with missing blocks without considering distribution and posterior heterogeneity.

2.1. Multi-Task Representation Learning

In the machine learning community, representation learning has received considerable attention due to its powerful and flexible capacity to learn complex and nonlinear patterns. A comprehensive review can be found in (Bengio, Courville, and Vincent 2013). For supervised learning settings, a typical procedure following representation learning is: $\mathbf{X}\underset{\mathit{f}}{\underbrace{\stackrel{\theta }{\to }\mathbf{h}\stackrel{g}{\to }}\mathbf{y}}$, where $\theta$ is a latent representation algorithm (representer) which projects the original data into a latent space, and $g$ is a regression or classification algorithm (learner) which establishes the association between latent representation and responses. In other words, the function $f$ is a composition of $g$ and $\theta$, denoted by $f=g\circ \theta$. This learning paradigm is a building block in deep learning for many applications. The representer $\theta$ is usually a complex neural net to uncover the nonlinear, entangled features, which allows a simple learner $g$ to achieve desirable empirical performance. In multi-task representation learning (Du et al. 2020; Tripuraneni et al. 2021; Watkins et al. 2023), each task is associated with a unique $g^{\left(s\right)}$, so the task-specific function mapping is formulated as $f^{\left(s\right)}=g^{\left(s\right)}\circ \theta$. In words, multi-task representation learning uses $\theta$ to learn shared representation across all tasks, and employs $g^{\left(s\right)}$ to capture all heterogeneity among tasks. In this framework, source-specific $g^{\left(s\right)}$’s are designed to capture all heterogeneity among data sources, however, it usually fail to accommodate both covariates shift and posterior drift. In this work, we employ representation learning to capture complex and nonlinear patterns, and propose a new framework to address the above drawback in multi-task representation learning.

3. Methodology

3.1. Representation Retrieval Learning

We first consider integrating multi-source datasets with the same modalities for supervised learning. This is equivalent to the multi-task learning problem with both covariates shift and posterior drift. Therefore, we use the terms “tasks” and “sources” interchangeably in this subsection.

We consider a representer dictionary as a collection of representers, denoted as $\Theta =\left\{{\theta }_1,\dots ,{\theta }_D\right\}$, where $D$ is the number of representers contained in $\Theta$. Formally, we define a representer as a univariate function mapping ${\theta }_d:𝒳\to \mathbb{R}$, so we use ${\theta }_d$ and ${\theta }_d\left(\mathbf{x}\right)$ interchangeably in later discussion. Multivariate representer outputs are also allowed, where the related extension can be found in Appendix A. To accommodate covariates shift among tasks, we assume that a subset of representers from $\Theta$ are informative to predict $y^{\left(s\right)}$ for the $s$th task, denoted as ${\Gamma }^{\left(s\right)}=\left\{{\theta }_{d_1},\dots ,{\theta }_{d_s}\right\}$. Naturally, we define ${\Gamma }^{\left(s\right)}\left(\mathbf{x}\right)=\left({\theta }_{d_1}\left(\mathbf{x}\right),\dots ,{\theta }_{d_s}\left(\mathbf{x}\right)\right):𝒳\to {\mathbb{R}}^{\left|{\Gamma }^{\left(s\right)}\right|}$. Since we do not impose any structural assumptions on ${\Gamma }^{\left(s\right)}$, both joint-and-individual (Lock et al. 2013) and partially sharing (Gaynanova and Li 2019) structures can be incorporated.

In order to learn the function mappings $f^{\left(s\right)}$ and account for posterior drift, we build the task-specific learners upon the retrieved representers as in multi-task representation learning. Suppose $g^{\left(s\right)}(\cdot )=⟨\cdot ,{\alpha }^{\left(s\right)}⟩$ is a linear function, then each function mapping $f^{\left(s\right)}$ is formulated as

$$f^{\left(s\right)}(\cdot )=⟨{\Gamma }^{\left(s\right)}(\cdot ),{\alpha }^{\left(s\right)}⟩,{\alpha }^{\left(s\right)}\in {\mathbb{R}}^{\left|{\Gamma }^{\left(s\right)}\right|}.$$ (3.1)

Mathematically, the above model is equivalent to

$$\begin{gathered} f^{\left(s\right)}\left(\cdot \right)=⟨\Theta \left(\cdot \right),{\beta }^{\left(s\right)}⟩=\sum _{d=1}^D{\beta }_d^{\left(s\right)}{\theta }_d\left(\cdot \right), \\ {\beta }^{\left(s\right)}\in {\mathbb{R}}^{\left|\Theta \right|}\text{and}{‖{\beta }^{\left(s\right)}‖}_0=\left|{\Gamma }^{\left(s\right)}\right|. \end{gathered}$$ (3.2)

Formulations (3.1) and (3.2) are equivalent because ${\beta }^{\left(s\right)}$ is a sparse coefficient vector which amounts to select $\left|{\Gamma }^{\left(s\right)}\right|$ representers from $\Theta$. In (3.1), both representers and learners are task-specific, where ${\Gamma }^{\left(s\right)}$ involves the representer retrieval and ${\alpha }^{\left(s\right)}$ is a task-specific regression coefficient. In comparison, only sparse regression coefficients ${\beta }^{\left(s\right)}$’s are task-specific in (3.2). By this means, we cast the representation retrieval framework as a model selection problem, which has been extensively studied (Tibshirani 1996; Tibshirani et al. 2005; Zou 2006; Yuan and Lin 2006). Additionally, more complex learners, such as neural nets, can be adopted to our framework, considering the sparse input features. We refer readers to (Wen et al. 2016; Feng and Simon 2017; Lemhadri et al. 2021; Fan and Gu 2024) for recent developments in sparse-input neural nets. To keep our introduction concise, we use linear learners in the following discussion.

To estimate the representers and learners, and pursue the sparsity in ${\beta }^{\left(s\right)}$’s, we can minimize the following loss function for regression problems:

$$\begin{gathered} \underset{\Theta ,{\beta }^{\left(1\right)},\dots ,{\beta }^{\left(S\right)}}{\text{min}}\frac{1}{S}\sum _{s=1}^S\frac{1}{n_s}\sum _{i=1}^{n_s}{\left(y_i^{\left(s\right)}-⟨\Theta \left({\mathbf{x}}_i^{\left(s\right)}\right),{\beta }^{\left(s\right)}⟩\right)}^2 \\ +\sum _{s=1}^S{\lambda }^{\left(s\right)}{‖{\beta }^{\left(s\right)}‖}_1, \end{gathered}$$ (3.3)

which is termed as representation retrieval ($R^2$) learning. By default, we consider representers ${\theta }_d$ as neural nets (Goodfellow et al. 2016) in this work due to its adaptation power to different modalities. Note that ${\lambda }^{\left(s\right)}$ controls the level of sparsity, so varying values of ${\lambda }^{\left(s\right)}$’s are adaptive to each data source. The optimization problem in (3.3) can be solved alternatively between $\Theta$ and ${\beta }^{\left(s\right)}$’s: Given a fixed $\Theta$, minimizing (3.3) is equivalent to solving Lasso problem for each data source; given fixed ${\beta }^{\left(s\right)}$’s, the dictionary $\Theta$ can be updated through a gradient-based algorithm. Therefore, the computation of our method is more efficient than optimizing over binary matrices as in (Gaynanova and Li 2019; Lock et al. 2022) to recover the partially sharing structure. To handle the classification problem, we can replace the least square loss with cross entropy loss without further modification.

In fact, the proposed model (3.2) covers many existing methods as special cases. For each single task, model (3.2) amounts to sparse multiple kernel learning (Koltchinskii and Yuan 2010) if ${\theta }_d$ corresponds to different kernels. In the simple scenario that ${\theta }_d\left(\mathbf{x}\right)={\mathbf{x}}_d$ and $D=p$, our model is equivalent to well-studied sparse linear models for multi-task learning (Lounici et al. 2009; Jalali et al. 2010; Craig et al. 2024). If ${\theta }_d$’s are linear mappings, our model is exactly a dictionary learning model for multi-task learning (Maurer, Pontil, and Romera-Paredes 2013). Our proposal is also closely related to mixture-of-expert for multi-task learning (Ma et al. 2018), where each representer can be treated as an expert.

3.2. Selective Integration Penalty

In this section, we introduction the notion of integrativeness of representer. Intuitively, data integration happens when the same representer is retrieved in ${\beta }^{\left(s\right)}$ by multiple tasks. Motivated by this, we define the integrativenss of a representer as follows:

$${\gamma }_d=\sum _{s=1}^S\mathbb{I}\left({\beta }_d^{\left(s\right)}\ne 0\right),$$ (3.4)

where $0\le {\gamma }_d\le S$ counts the number of data sources that retrieves the representer ${\theta }_d$ for prediction. Theoretically, we will show that more integrative representers lead to the sharper excess risk bound in Section 4. However, the optimization problem in (3.3) does not necessarily favor more integrative representers. Therefore, we introduce additional structural assumptions in the $R^2$ learning for better control of integrativeness to further improve the prediction.

Formally, we propose the following Selective Integration Penalty (SIP) to encourage the integrativeness of representers:

$$𝒫\left({\beta }^{\left(1\right)},\dots ,{\beta }^{\left(S\right)}\right)=\sum _{d=1}^D\text{min}\left(1,\frac{S-{\gamma }_d}{S-1}\right),$$ (3.5)

where $0\le 𝒫(\cdot )\le D$. For each single ${\gamma }_d$, SIP truncates at ${\gamma }_d=1$, in that the penalty is identical to 1 for $0\le {\gamma }_d\le 1$; since any $0\le {\gamma }_d\le 1$ do not integrate any information across tasks, therefore, they should be penalized at the same level. When $𝒫(\cdot )=0$, it indicates that ${\beta }_d^{\left(s\right)}\ne 0$ for all $d$ and $s$, showing that all the representers are retrieved by all $S$ data sources. On the contrary, if $𝒫(\cdot )=D$, then ${\gamma }_d\le 1$ for all $d$, suggesting that each representer is retrieved at most once. In other words, different tasks do not share any common representers. In addition, if $0<P(\cdot )<D$, a partially sharing structure is encoded in ${\beta }^{\left(s\right)}$, see Figure 2. Due to the non-smoothness of the indicator function in (3.4), it is intractable to minimize $𝒫(\cdot )$. Alternatively, we adopt the truncated-${\ell }_1$ function (Pan, Shen, and Liu 2013; Wu et al. 2016) as a continuous relaxation of the indicator function in (3.4):

$${\tilde{\gamma }}_d(\tau )=\sum _{s=1}^S\text{min}\left(1,\frac{\left|{\beta }_d^{\left(s\right)}\right|}{\tau }\right),$$

where $0\le {\tilde{\gamma }}_d\left(\tau \right)\le S$. Based on the above derivation, the loss function of our $R^2$ learning with SIP is

$$\begin{gathered} \underset{\Theta ,{\beta }^{\left(1\right)},\dots ,{\beta }^{\left(S\right)}}{\text{min}}\frac{1}{S}\sum _{s=1}^S\frac{1}{n_s}\sum _{i=1}^{n_s}{‖y_i^{\left(s\right)}-⟨\Theta \left({\mathbf{x}}_i^{\left(s\right)}\right),{\beta }^{\left(s\right)}⟩‖}_2^2 \\ +\sum _{s=1}^S{\lambda }_1^{\left(s\right)}{‖{\beta }^{\left(s\right)}‖}_1+{\lambda }_2{𝒫}_{\tau }\left({\beta }^{\left(1\right)},\dots ,{\beta }^{\left(S\right)}\right), \end{gathered}$$ (3.6)

where ${𝒫}_{\tau }(\cdot )$ is the smoothed SIP by replacing ${\gamma }_d$ with ${\tilde{\gamma }}_d\left(\tau \right)$. Here, two penalties play different roles: the ${\ell }_1$ penalty in the second term of (3.6) targets each individual task, in order to select informative representers for each specific task. SIP, on the other hand, acts on regression coefficients across tasks, and seek to improve the overall integrativeness of representers. We employ the Adam (Kingma and Ba 2014) optimizer in pytorch to minimize the (3.6) by alternating between $\Theta$ and $\beta$.

Figure 2.

(a): Black solid line: curve of SIP formula (3.5) for a single ${\gamma }_d$; blue dashed line: curve of $\sqrt{1-{\gamma }_d/S}$, which will be introduced in Section 4. (b): Illustrations of regression coefficients ${\beta }_d^{\left(s\right)}$ under $𝒫=D,0<𝒫<D$ and $𝒫=0$ scenarios. In this example, we consider $D=10$ representers, and $S=4$ tasks.

Our proposed $R^2$ learning with SIP offers unique advantages to multi-task learning. Specifically, we allow practitioners to choose flexible representers that are adaptive to their data structure. Since our SIP is defined directly on learners, representers can be different models with varying numbers of parameters. For example, linear mappings can capture linear structures of original data, while neural nets can capture high-order nonlinear structures, and we can combine linear and nonlinear representers in the dictionary as representer candidates for different data sources. This is advantageous for multi-source data where each data type could have varying complexity of structures. In contrast, most existing integrative penalties (Tang and Song 2016; Duan and Wang 2023; Tian, Gu, and Feng 2023) require model parameters comparable across tasks to apply the penalties, which could be stringent in multi-source data integration. In particular, when multi-source data have different modalities (e.g, some of them have images, while others have text data), it is ineffective to employ the same model for all sources.

3.3. Blockwise Representation Retrieval Learning

In this section, we propose the Blockwise Representation Retrieval ($B{R}^2$) learning, which extends the $R^2$ learning to multi-modality data with blockwise missingness.

For ease of our discussion, we first introduce the essential notations for multi-modality data. In total, there are $M$ modalities that are observed by at least one data source. The missing indicators, ${\mathbb{I}}_m^{\left(s\right)}$ for $s=1,\dots ,S,m=1,\dots ,M$, indicate whether the $s$th data source observing the $m$th modality. That is, ${\mathbb{I}}_m^{\left(s\right)}=1$ if the $m$th modality is observed in the $s$th data source, and ${\mathbb{I}}_m^{\left(s\right)}=0$ otherwise. Further, we define ${𝒪}_m=\left\{s:{\mathbb{I}}_m^{\left(s\right)}=1\right\}$ as the set of data sources observing the $m$th modality; Similarly, we denote ${𝒪}^{\left(s\right)}=\left\{m:{\mathbb{I}}_m^{\left(s\right)}=1\right\}$ as the observed modalities of the $s$th data source. For instance, the upper block in Figure 1 presents a $S=5$ and $M=5$ example. As an illustration, ${𝒪}^{\left(1\right)}=\{\mathrm{1,2},\mathrm{4,5}\}$ and ${𝒪}_2=\{\mathrm{1,3},4\}$ in this example.

For real applications with blockwise missingness, the missing mechanism is usually unknown and unverifiable. Due to potential covariates shift of marginal distribution of each modality or joint distribution of several modalities, imputation approaches, which learn conditional distribution from other data sources to impute missing modalities for certain data sources, can introduce significant bias. First of all, missing completely at random (MCAR) is unlikely to hold under covariates shift. Second, under missing at random (MAR) or missing not at random (MNAR) mechanisms, the conditional distribution of one modality given other modalities are different across data sources, which invalidate the aforementioned imputation approaches. Therefore, it is of great interest to investigate whether we can improve predictive performance without imputation.

In the following, we propose the Blockwise Representation retrieval $\left(B{R}^2\right)$ learning, which directly learns the function mappings from observed modalities to responses without imputation. Instead of imputing missing blocks and applying $R^2$ learning to the imputed data, we aim to borrow as much information as we can given the observed data. Specifically, we introduce representer dictionary for each single modalities. To accommodate covariates shift among data sources for each modality, a subset of representers are retrieved as informative ones to predict the response. Later, we concatenate retrieved representers from each dictionary as the latent representation of each data source. Again, we introduce source-specific sparse-induced learners for each modality to handle posterior drift. Formally, we define modality-specific representer dictionaries as ${\Theta }_m=\left\{{\theta }_{m;1},\dots ,{\theta }_{m;D_m}\right\},m=1,\dots ,M$, and each representer is a function mapping ${\theta }_{m;d}:{𝒳}_m\to \mathbb{R}$ as in $R^2$ learning. Given this setup, data sources $s\in {𝒪}_m$ share the $m$th dictionary, which enables us to borrow information across data sources for each modality. The function mapping $f^{\left(s\right)}:{𝒳}^{\left(s\right)}\to {𝒴}^{\left(s\right)}$ is then modeled as

$$\begin{gathered} f^{\left(s\right)}\left(\cdot \right)=\sum _{m=1}^M{\mathbb{I}}_m^{\left(s\right)}⟨{\Theta }_m\left(\cdot \right),{\beta }_m^{\left(s\right)}⟩, \\ {\beta }_m^{\left(s\right)}\in {\mathbb{R}}^{\left|{\Theta }_m\right|}\text{and}{‖{\beta }_m^{\left(s\right)}‖}_0=\left|{\Gamma }_m^{\left(s\right)}\right|, \end{gathered}$$ (3.7)

where ${\Gamma }_m^{\left(s\right)}$ is the subset of retrieved representers from the $m$th modality dictionary ${\Theta }_m$ for the $s$th data source. In (3.7), the missing indicator ${\mathbb{I}}_m^{\left(s\right)}$ indicates that if the $m$th modality is unobserved in the $s$th data source, then the $m$th modality has no impact on the response.

Like linear models, dictionaries ${\Theta }_m$’s represent the main effects of the $m$th modality to responses, while the interaction or association among modalities are ignored. To ease this constraint, there are two options to incorporate the association among modalities in modeling $f^{\left(s\right)}$. First, we can introduce interaction dictionaries ${\Theta }_{m_1,m_2,\dots ,m_w}$, where $m_1,\dots ,m_w\in \{1,\dots ,M\}$ represents the $w$–way interaction among modalities. Similarly, each interaction dictionary ${\Theta }_{m_1,\dots ,m_w}$ is associated with sparse learner ${\beta }_{m_1,\dots ,m_w}^{\left(s\right)}$ for each source. Second, we may choose more complex learners, such as sparse-input neural nets (Feng and Simon 2017; Lemhadri et al. 2021; Fan and Gu 2024) to learn the interaction among modalities. Again, we will only consider the modality-specific dictionaries and linear learners in the later theoretical analysis and numerical experiments.

Similar to the $R^2$ learning, the representers and learners in $B{R}^2$ learning can be estimated by minimizing the following loss function for regression problems:

$$\begin{gathered} \underset{\Theta ,\beta }{\text{min}}\frac{1}{S}\sum _{s=1}^S\frac{1}{n_s}\sum _{i=1}^{n_s}{‖y_i^{\left(s\right)}-\sum _{m=1}^M{\mathbb{I}}_m^{\left(s\right)}〈{\Theta }_m\left({\mathbf{\text{x}}}_{i;m}^{\left(s\right)}\right),{\beta }_m^{\left(s\right)}〉‖}_2^2 \\ +\sum _{s=1}^S\sum _{m=1}^M{\lambda }_{m;1}^{\left(s\right)}{‖{\beta }_m^{\left(s\right)}‖}_1,+\sum _{m=1}^M{\lambda }_{m;2}𝒫\left({\beta }_{{𝒪}_m}\right) \end{gathered}$$ (3.8)

and squared loss can be replaced with other loss functions, such as cross entropy loss for classification problems. Since ${\Theta }_m$ is modality-specific dictionary and can only be trained with data sources observing the $m$the modality, its integrativenss is defined as ${\gamma }_{m;d}=\sum _{s\in {𝒪}_m}\mathbb{I}\left({\beta }_{m;d}^{\left(s\right)}\ne 0\right)$. As a result, the SIP for the $m$th modality is only effective for data sources in ${𝒪}_m,{\beta }_{{𝒪}_m}$, which improves the modality-specific integrativeness of representers in ${\Theta }_m$..

Compared with supervised learning for individual sources ignoring the missing modalities, our $B{R}^2$ learning improves the integrativeness of each modality-specific representations that borrow information from other sources observing the same modalities, which performs at least comparably well to learning with individual sources. In comparison, supervised learning based on imputed values could diminish the predictive power if imputation models are mis-specified, which may perform even worse than learning with individual sources. Empirical comparisons among these methods are provided in Section 5.2.

4. Theoretical Results

In this section, we study the generalization performance of the $R^2$ framework. In particular, we first establish the excess risk bound of $R^2$ learning and then extend to $B{R}^2$ learning. The theoretical results in this section will reveal that the generalization performance of $R^2$ learning is impacted by the representer retrieval pattern. In words, given sparse constraints on representer retrieval, it is preferable to have less representers shared by more data sources, than more representers shared by less data sources. This finding resonates with the proposed SIP in Section 3.2. In the end, the empirical advantage of SIP over the theoretical upper bound will be discussed.

First, we formalize the function classes of (3.2) as follows: Suppose each representer ${\theta }_d\in {𝒢}_d$, where ${𝒢}_d$ is certain function class for the $d$th representer, such as linear function or neural nets with norm constraints. As for sparse learners, we consider the sparse regression coefficient class: ${\beta }^{\left(s\right)}\in {\mathscr{H}}^{\left(s\right)}=\{\beta :\left.\Vert \beta {\Vert }_1\le {\alpha }^{\left(s\right)}\right\}$. For brevity, we denote $𝒢$ as the function space for $\Theta =\left({\theta }_1,\dots ,{\theta }_D\right)$ and $\mathscr{H}$ as the space for $\beta =\left({\beta }^{\left(1\right)},\dots ,{\beta }^{\left(S\right)}\right)$. Then, the function classes of each ${ℱ}^{\left(s\right)}$ is characterized by ${𝒢}_1,\dots ,{𝒢}_D$ and ${\mathscr{H}}^{\left(s\right)}$:

$$\begin{gathered} {ℱ}^{\left(s\right)}=\left\{f^{\left(s\right)}:f^{\left(s\right)}(\cdot )=\sum _{d=1}^D{\beta }_d^{\left(s\right)}{\theta }_d(\cdot ),{\beta }^{\left(s\right)}\right. \\ \left.=\left({\beta }_1^{\left(s\right)},\dots ,{\beta }_D^{\left(s\right)}\right)\in {\mathscr{H}}^{\left(s\right)},{\theta }_d\in {ℱ}_d\right\}. \end{gathered}$$

The function classes ${𝒢}_d$’s and ${\mathscr{H}}^{\left(s\right)}$’s admits the partially sharing structures across data sources in that each data source retrieves some of representers from the dictionary, which relaxes the fully sharing structure studied in (Tripuraneni et al. 2021; Watkins et al. 2023).

Assuming each data source is endowed with distribution ${\mu }^{\left(s\right)}$, and we can observe $n_s$ iid samples $\left({\mathbf{x}}_i^{\left(s\right)},y_i^{\left(s\right)}\right),i=1,\dots ,n_s$ from ${\mu }^{\left(s\right)}$. Then, the empirical risk minimization of $R^2$ learning can be formulated as follows:

$$\begin{gathered} \underset{\Theta \in 𝒢,\beta \in \mathscr{H}}{\text{min}}\widehat{ℛ}(\Theta ,\beta ) \\ ≔\underset{\Theta \in 𝒢,\beta \in \mathscr{H}}{\text{min}}\frac{1}{S}\sum _{s=1}^S\frac{1}{n_s}\sum _{i=1}^{n_s}\ell \left\{y_i^{\left(s\right)},\sum _{d=1}^D{\theta }_d\left({\mathbf{x}}_i^{\left(s\right)}\right){\beta }_d^{\left(s\right)}\right\}, \end{gathered}$$

where the empirical risk minimizer is denoted as $\hat{\Theta }$ and $\hat{\beta }$. For simplicity, we assume $n=n_1=\cdots =n_S$ in the following theoretical development. Analogously, the population risk is defined as

$$ℛ(\Theta ,\beta )=\frac{1}{S}\sum _{s=1}^S{\mathbb{E}}_{\left({\mathbf{x}}^{\left(s\right)},y^{\left(s\right)}\right)~{\mu }^{\left(s\right)}}\left[\ell \left\{y^{\left(s\right)},\sum _{d=1}^D{\theta }_d\left({\mathbf{x}}^{\left(s\right)}\right){\beta }_d^{\left(s\right)}\right\}\right],$$

and ${\Theta }^{*},{\beta }^{*}$ denote the minimizers of population risk. Further, we denote the population risk given true $f^{\left(s\right)}$ as $ℛ\left(f\right)=\frac{1}{S}\sum _{s=1}^S{\mathbb{E}}_{\left({\mathbf{x}}^{\left(s\right)},y^{\left(s\right)}\right)~{\mu }^{\left(s\right)}}\left[\ell \left\{y^{\left(s\right)},f^{\left(s\right)}\left({\mathbf{x}}^{\left(s\right)}\right)\right\}\right]$. Since the postulated function space ${ℱ}^{\left(s\right)}$ may not contain the true function $f^{\left(s\right)},ℛ\left({\theta }^{*},{\beta }^{*}\right)-ℛ\left(f\right)$ signifies the approximation error of ${ℱ}^{\left(s\right)}$ toward $f^{\left(s\right)}$. Our target is the excess risk between the empirical risk minimizer and the true functions $f^{\left(s\right)}$’s, defined as

$$\begin{gathered} ℰ(\hat{\Theta },\hat{\beta })=ℛ(\hat{\Theta },\hat{\beta })-ℛ(f) \\ =\underset{\text{estimation error}}{\underbrace{ℛ(\hat{\Theta },\hat{\beta })-ℛ\left({\Theta }^{*},{\beta }^{*}\right)}}+\underset{\text{approximation error}}{\underbrace{ℛ\left({\Theta }^{*},{\beta }^{*}\right)-ℛ\left(f\right)}}. \end{gathered}$$

In this decomposition, the approximation error depends on several factors, such as the choice of ${𝒢}_d$’s, and properties of true function $f^{\left(s\right)}$’s and so on. In the following analysis, we focus on the estimation error from finite sample, given any choice of ${𝒢}_d$ with some mild regularity conditions. Now, we proceed with some essential assumptions:

Assumption 4.1. (a) (Lipschitz loss) The loss function: $\ell (\cdot ,\cdot )$ is bounded, and $\ell (y,\cdot )$ is L-Lipschitz for all $y\in 𝒴$.

(b) (Boundedness) The representer is bounded for $d\in \left[D\right]:{\text{sup}}_{\mathbf{x}\in 𝒳}\left|{\theta }_d\left(\mathbf{x}\right)\right|\le B$ for any ${\theta }_d\in {ℱ}_d$. In addition, if $𝒴=\mathbb{R}$, we assume $\left|y\right|\le B$.

Remark 4.1. Assumptions (a) and (b) are standard assumptions in learning theory to control the complexity of function classes, which can be satisfied with bounded $𝒴$ and $𝒢$ under common adopted loss functions, such as least square loss and cross entropy loss.

In the following, we show that the complexity of the representer dictionary is directly impacted by ${\gamma }_d$, the integrativeness of the $d$th representer.

Lemma 4.1. For any $\beta \in \mathscr{H}$ and fixed realizations ${\left({\mathbf{x}}_i^{\left(s\right)},y_i^{\left(s\right)}\right)}_{i=1}^n$, we define

$${\mathbf{G}}_{\beta }(\mathit{\sigma })=\underset{\Theta \in 𝒢}{\text{sup}}\frac{1}{nS}\sum _{s=1}^S\sum _{i=1}^n{\sigma }_{s,i}\left[\sum _{d=1}^D{\theta }_d\left({\mathbf{x}}_i^{\left(s\right)}\right){\beta }_d^{\left(s\right)}\right],$$

where ${\sigma }_{s,i}$ are i.i.d Rademacher random variables, then we have:

$${\mathbb{E}}_{\mathit{\sigma }}{\mathbf{G}}_{\beta }(\mathit{\sigma })\le \alpha \sqrt{\frac{C_{\Theta }}{n{S}^2}}\sum _{d=1}^D\sqrt{{\gamma }_d},$$ (4.1)

with ${\gamma }_d=\sum _{s=1}^S\mathbb{I}\left({\beta }_d^{\left(s\right)}\ne 0\right)$. The constant $C_{\Theta }$ is the uniform upper bound $C_{\Theta }\ge C_{{\theta }_d}$ for all $d$, where $C_{{\theta }_d}$ is the Rademacher complexity of function class ${𝒢}_d$. And $\alpha \ge {\alpha }^{\left(s\right)}$ is the uniform upper bound for all ${\alpha }^{\left(s\right)}$.

The quantity ${\mathbb{E}}_{\mathit{\sigma }}{\mathbf{G}}_{\beta }\left(\mathit{\sigma }\right)$, analogous to the Rademacher complexity, quantifies the capacity of the representation dictionary to fit random noise given any fixed $\beta \in \mathscr{H}$. A similar measure has been studied for linear dictionaries in (Maurer, Pontil, and Romera-Paredes 2013), in which they obtained a looser upper bound that relies on all $D$ representers. In contrast, we refine the analysis by leveraging the support of the given $\beta$ in Lemma 4.1, which leads to the $\sum _{d=1}^D\sqrt{{\gamma }_d}$ term in (4.1). We refer readers to Appendix H for more detailed technical derivation.

We first instantiate the upper bound (4.1) with two extreme cases for interpretation. If ${\alpha }_s$’s are large enough for all data sources, or ${\gamma }_d=S$ for all representers, the upper bound is $\alpha \sqrt{\frac{C_{\Theta }{D}^2}{nS}}$, which uses all data sources to learn all representers, and matches the order of $O\left(\sqrt{1/nS}\right)$ in existing multi-task learning literature (Tripuraneni et al. 2021). In this regime, large $D$ incurs potential overfitting that worsen the estimation error, signifying the drawback of the fully sharing structure. On the other hand, if ${\gamma }_d=1$ for all representers, the dictionary size $D$ is at least $S$, which leads the upper bound to be at least $\alpha \sqrt{\frac{C_{\Theta }}{n}}$, indicating no benefit of integrating all data sources to learn $\Theta$. In general, the complexity of $\Theta$ lies between the above two extreme cases in that shared representers can lower the complexity. Due to the concavity of $\sum _{d=1}^D\sqrt{{\gamma }_d}$, the upper bound in (4.1) is lower when nonzero ${\gamma }_d$’s are concentrated in few representers given fixed function class $𝒢$. In other words, less representers shared by more data sources lowers the upper bound in (4.1) than more representers shared by less data sources. In the following, we obtain the excess risk bound for $R^2$ learning, and provide more detailed discussion on the role of integrativeness.

Theorem 4.1. Suppose Assumption 4.1 holds, and the dictionary size is fixed as $D$. Constants $C_{\Theta }$ and $\alpha$ are defined as in Lemma 4.1. Then for empirical risk minimizer $\hat{\Theta }$ and $\hat{\beta }$, with probability $1-\delta$, we have

$$\begin{gathered} ℛ(\hat{\Theta },\hat{\beta })-ℛ\left({\Theta }^{*},{\beta }^{*}\right)\le 4L\alpha \sqrt{\frac{C_{\Theta }}{n{S}^2}}\underset{\beta \in \mathscr{H}}{\text{sup}}\sum _{d=1}^D\sqrt{{\gamma }_d} \\ +4\alpha BL\sqrt{\frac{2\text{log}\left(2D\right)}{n}}+B\sqrt{\frac{2\text{log}(2/\delta )}{nS}}. \end{gathered}$$ (4.2)

The first term in (4.2) shows that the cost of learning the representer dictionary is controlled by ${\text{sup}}_{\beta \in \mathscr{H}}\sum _{d=1}^D\sqrt{{\gamma }_d}$. Therefore, the estimation error depends not only on the sparsity level of each source-specific learner, but also on how the selected representers are shared across sources. For a fixed total number of selected source-representer pairs, the concavity of the square-root function implies that $\sum _{d=1}^D\sqrt{{\gamma }_d}$ is smaller when the nonzero coefficients are concentrated on fewer representers, that is, when selected representers are shared by more data sources. This observation provides the theoretical motivation for encouraging integrative representers.

To formalize this effect, we consider an idealized constrained coefficient class that captures the type of sharing encouraged by SIP. Recall that the truncated-${\ell }_1$ relaxation of the retrieval count is ${\tilde{\gamma }}_d\left(\tau \right)=\sum _{s=1}^S\text{min}\left\{1,\frac{\left|{\beta }_d^{\left(s\right)}\right|}{\tau }\right\}$, where $\tau >0$ specifies the threshold above which a coefficient is regarded as meaningfully nonzero. For a given function $\eta \left(\tau \right)$, define

$${\mathscr{H}}_{\eta ,\tau }=\left\{\beta \in \mathscr{H}:\sum _{d=1}^D\sqrt{{\tilde{\gamma }}_d\left(\tau \right)}\le \eta (\tau )\right\}.$$

This constrained class is not asserted to be algebraically identical to the penalized objective in (3.6). Rather, it provides a theoretical surrogate for the retrieval patterns that SIP is designed to promote: coefficient patterns in which the effective support is concentrated on more integrative representers. Under this constraint, the dictionary complexity term in Theorem 4.1 can be sharpened by replacing the worst-case quantity ${\text{sup}}_{\beta \in \mathscr{H}}\sum _{d=1}^D\sqrt{{\gamma }_d}$ with the controlled quantity $\eta \left(\tau \right)$, up to an additional approximation term caused by coefficients whose magnitudes are below the threshold $\tau$.

The quantity $\sum _{d=1}^D\sqrt{{\gamma }_d}$ is the most direct complexity measure suggested by Theorem 4.1. Equivalently, for fixed $D$ and $S$, reducing $\sum _{d=1}^D\sqrt{{\gamma }_d}$ corresponds to encouraging larger values of ${\gamma }_d$ among selected representers, or more concentrated sharing. A direct penalty based on $\sqrt{{\gamma }_d}$, however, is less convenient for gradient-based optimization: its derivative becomes singular near zero and changes rapidly for small ${\gamma }_d$. In contrast, the SIP introduced in Section 3.2 is a piecewise-linear surrogate that penalizes weakly shared representers while maintaining a stable and constant slope once a representer is used by more than one source. Thus, SIP is used as a computationally stable mechanism for encouraging the integrative retrieval patterns suggested by the upper bound.

Let $(\check{\Theta },\check{\beta })=\text{arg}{\text{min}}_{\Theta \in 𝒢,\beta \in {\mathscr{H}}_{\eta ,\tau }}\widehat{R}(\Theta ,\beta )$ and $\left({\Theta }_{\eta ,\tau }^{*},{\beta }_{\eta ,\tau }^{*}\right)=\text{arg}{\text{min}}_{\Theta \in 𝒢,\beta \in {\mathscr{H}}_{\eta ,\tau }}R(\Theta ,\beta )$, then the following corollary gives the estimation error over ${\mathscr{H}}_{\eta ,\tau }$.

Corollary 4.1. Suppose Assumption 4.1 holds, and the dictionary size is fixed as $D$. Constants $C_{\Theta }$ and $\alpha$ are defined as in Lemma 4.1.

Then for empirical risk minimizer $\check{\Theta }$ and $\check{\beta }$, with probability $1-\delta$, we have

$$\begin{gathered} ℛ(\check{\Theta },\check{\beta })-ℛ\left({\Theta }_{\eta ,\tau }^{*},{\beta }_{\eta ,\tau }^{*}\right)\le 4L\alpha \sqrt{\frac{C_{\Theta }}{n{S}^2}}\eta (\tau ) \\ +4L\tau \sqrt{\frac{C_{\Theta }{D}^2}{nS}}+4\alpha BL\sqrt{\frac{2\text{log}\left(2D\right)}{n}}+B\sqrt{\frac{2\text{log}(2/\delta )}{nS}}. \end{gathered}$$ (4.3)

The first term indicates the complexity of learning the representer when regression coefficients are meaningfully nonzero $\left(\left|{\beta }_d^{\left(s\right)}\right|>\tau \right)$, and the second term indicates the complexity of learning the representer when $\left|{\beta }_d^{\left(s\right)}\right|\le \tau$. When $\tau \to 0$, the second term vanishes, which indicates that the first term captures the main complexity of learning $\Theta$ and the second terms captures the additional cost of approximating ${\gamma }_d$ with ${\tilde{\gamma }}_d\left(\tau \right)$. In addition, since $\tau$ is usually far less than $\alpha$, the first term dominates the second term in the above upper bound.

There are several tradeoffs in the excess risk bound. First of all, the first term in (4.3) is monotonically decreasing with respect to $\tau$, while the second term is monotonically increasing with respect to $\tau$. Consequently, choosing a proper $\tau$ can effectively balance these two terms. Second, the dictionary size $D$, sparsity constraint $\alpha$ and representer complexity $C_{\Theta }$ determine the model capacity jointly. In particular, the complexity of representer $C_{\Theta }$ impact the approximation error and the complexity of learning $\Theta$ (4.1). In general, a simple representer model (small $C_{\Theta }$) would potentially worsen the approximation error and improve the generalization error, and vice versa. The sparsity constraint $\alpha$ is involved in the approximation error, and the complexity of learning both representer and regressors in the estimation error. A sparse regressor (small $\alpha$) would potentially worsen the approximation error and improve the estimation error, and vice versa. The dictionary size $D$ is slightly different, the complexity of learning $\Theta$ in (4.3) is robust with respect to $D$. In particular, since SIP encourages to minimize $\sum _{d=1}^D\sqrt{{\tilde{\gamma }}_d\left(\tau \right)}$, which leads to many representers with ${\tilde{\gamma }}_d\left(\tau \right)=0$, they do not increase the complexity of learning $\Theta$. Even though a large $D$ could increase the cost of learning regressors in $O\left(\sqrt{\text{log}\left(D\right)}\right)$, its impact is relatively minimal due to the logarithm. Therefore, in our implementation, we fix the dictionary size $D=30$ and tune other tuning parameters to optimize the generalization performance.

The theoretical property of $B{R}^2$ learning can be derived with the similar logic and derivation, which is provided in Appendix I.

5. Simulation Studies

We evaluate the performance of the proposed $R^2$ and $B{R}^2$ learning approaches under various simulation settings and compare them with several competitive methods. Section 5.1 investigates a multi-task learning (MTL) scenario with fully aligned covariates across data sources, comparing $R^2$ learning against standard MTL methods. Section 5.2 examines a multi-modality blockwise missing setting, where $B{R}^2$ learning is compared with methods designed for block-missing data. In both cases, our methods demonstrate superior performance, highlighting their effectiveness in integrating heterogeneous data.

5.1. Simulations for Multi-Task Learning

We compare our method with several baseline and state-of-the-art MTL approaches. The baseline models include Single Task Learning (STL), which fits separate models for each data source using either a linear model (STL-linear) or a feed-forward neural network (STL-NN). Another baseline is Data Pooling (Pooling), which fits a single model across all sources, thereby ignoring any types of heterogeneity. We also use either linear model (Pooling-Linear) or neural net (Pooling-NN) for prediction. These baselines represent two extremes: Pooling is optimal when all tasks are sampled from the same distribution, whereas STL is preferable when no or very few information is shared across data sources.

We also compare our method against prominent multi-task methods from the literature. First, we consider Multi-task Representation Learning (MTRL; Du et al. 2020; Tripuraneni et al. 2021), employing a uniform linear (MTRL-Linear) or neural net (MTRL-NN) representer along with source-specific learners. Second, we evaluate the penalized ERM algorithm (pERM; Tian, Gu, and Feng 2023), which allows for similar, though not identical, linear representations across sources and accommodates a subset of outlier sources. Third, Adaptive and Robust Multi-task Learning (ARMUL; Duan and Wang 2023) is included, as it encourages regression coefficients to conform to a low-rank representation. In all above methods, we specify one hidden layer with 32 neurons as neural net model or representer. The implementation details are provided in Appendix E.

We generate simulation data from $S=20$ data sources as follows:

$$y_i^{\left(s\right)}=\Theta \left({\mathbf{x}}_i^{\left(s\right)}\right){\beta }^{\left(s\right)}+{ϵ}_i^{\left(s\right)},i=1,\dots ,n_s,s=1,\dots ,S,$$ (5.1)

where covariates ${\mathbf{x}}_i^{\left(s\right)}$ follow a multivariate standard normal distribution with dimension $p=30$ and ${ϵ}_i^{\left(s\right)}$’s are sampled from standard normal distribution. The representer dictionary $\Theta (\cdot )=\left({\theta }_1,\dots ,{\theta }_{30}\right)$ contains $D=30$ representers, where the first ten representers are linear, the last twenty representers are nonlinear. The detailed specification of these representers can be found in Appendix B.1. Coefficients ${\beta }^{\left(s\right)}$’s are $D=30$-dimensional sparse vector with 5 nonzero entries, where the indices of nonzero are sampled from 30 dimension with the following five different distributions:

$$\begin{array}{ll}1.\left(\underset{\text{5 entries}}{\underbrace{1/5,\dots ,1/5}},\underset{25\text{entries}}{\underbrace{0,\dots ,0}}\right)\hfill & 2.\left(\underset{10\text{entries}}{\underbrace{1/10,\dots ,1/10}},\underset{20\text{entries}}{\underbrace{0,\dots ,0}}\right)\hfill \\ 3.\left(\underset{10\text{entries}}{\underbrace{0,\dots ,0}},\underset{5\text{entries}}{\underbrace{1/5,\dots ,1/5}},\underset{15\text{entries}}{\underbrace{0,\dots ,0}}\right)\hfill & 4.\left(\underset{10\text{entries}}{\underbrace{0,\dots ,0}},\underset{10\text{entries}}{\underbrace{1/10,\dots ,1/10}},\underset{10\text{entries}}{\underbrace{0,\dots ,0}}\right)\hfill \\ 5.\left(\underset{30\text{entries}}{\underbrace{1/30,\dots ,1/30}}\right).\hfill & \hfill \end{array}$$

In particular, the first and second distributions (linear setting) retrieve linear representers, and the third and fourth distributions (nonlinear setting) retrieve nonlinear representers. Further, the first and third distributions specify fully-sharing structure that each data source retrieves the same five representers. On the contrary, the second and fourth distributions allow partially sharing structure that each data source can flexibly retrieve 5 out of 10 representers. The last distribution (mixed setting) allows all data sources to flexibly retrieve any represeneters with uniform probability. On average, each representer will be retrieved only 10/3 times, where information across data sources are hardly borrowed. We also consider two settings for the values of nonzero entries in ${\beta }^{\left(s\right)}$’s: each nonzero entry is sampled from $𝒩\left(1,{\sigma }^2\right)$ and $\sigma =\mathrm{0,1}$ to reflect the homogeneous and heterogeneous posterior distribution of response given latent representations. In total, we consider $5\times 2=10$ simulation settings, and each simulation setting is repeated 100 times by default.

In the simulation experiment, we set sample size $n_s=100$ for each source in the training, and $n_s=1000$ for validation, and testing datasets. Model performance is evaluated using the average root mean squared error (RMSE) across all data sources. The predictive performance under nonlinear and mixed settings are presented in Figures 3 and 4, respectively, where linear setting is presented in Appendix C.1. Overall, $R^2$ learning with SIP achieves superior or comparable performance under varying settings, especially under partially sharing and posterior drift scenarios. And the effect of SIP for improving predictive performance is especially significant when neural nets are employed as representers.

Figure 3.

Simulation results for multi-task learning problem under nonlinear setting.

Figure 4.

Simulation results for multi-task learning problem under mixed setting.

Under the nonlinear setting, Pooling-NN achieves the best performance because it can correctly borrow information across all data sources in the homogeneous (fully sharing, no posterior drift) scenario. Beyond that, the $R^2$ learning with neural net representers and SIP outperforms all other competing methods if there is either partially sharing structure or posterior drift across data sources. In the mixed setting, heterogeneity across all data sources is rather significant, in that information across all data sources are hardly integrated unless true data generating procedure is known. As a result, STL-NN is a competitive baseline. Still, $R^2$ learning with neural net representers with SIP can achieve comparable and even better predictive performance.

We provide additional simulation studies in Appendix C to illustrate the performance of our method in classification problems and varying sample size settings.

5.2. Simulations for Multi-Modality Block-Mising Data

This section examines the performance of our proposed $B{R}^2$ learning method in a multi-modality block-missing data setting. We compare $B{R}^2$ learning with a benchmark Single Task Learning (STL) method that uses only the observed modalities, and with DISCOM (Yu et al. 2020), which directly predicts without imputing missing modalities by leveraging all available data for estimation of covariance and cross-covariance matrices. Note that the Pooling method is inapplicable here due to missing modalities. Hyperparameters in Yu et al. (2020) are selected via grid search in their program, and we still employ optuna package to tune the hyperparameters in our proposed $B{R}^2$ learning.

Figure 5 illustrates the blockwise missing patterns considered in the simulation where $L$ denotes the number of modalities observed per source. In this experiment, we consider $S=4$ data sources, $M=4$ modalities, and set the sample size $n_s=200$ for training, validation, and testing. Data are generated according to:

$$y^{\left(s\right)}=\sum _{m=1}^M{\Theta }_m\left({\mathbf{X}}_m^{\left(s\right)}\right){\beta }_m^{\left(s\right)}+{ϵ}^{\left(s\right)},$$

where modality-specific covariates ${\mathbf{X}}_m^{\left(s\right)}$ are sampled from a $q=40$-dimensional multivariate normal distribution with modality-specific covariance structures (details are provided in Appendix B.2). For fair comparison, we restrict the representers to the linear case (orthogonal bases in ${\mathbb{R}}^q$) as DISCOM is designed for linear models. For each modality, only two representers are active, that is, ${‖{\beta }_m^{\left(s\right)}‖}_0=2$, and the noise term ${ϵ}^{\left(s\right)}$ are iid samples of standard normal distribution.

Figure 5.

Blockwise missing patterns considered in simulation study. $L$ indicates the number of observed modalities for each data source.

Further, we consider the number of shared representers $I=\mathrm{0,1},2$ across data sources to design varying levels of covariates shift, and introduce posterior drift by setting ${\beta }_m^{\left(s\right)}={\beta }_m+{\delta }^{\left(s\right)}$ with ${\beta }_{m,\text{supp}}=\left(\mathrm{1,1}\right)$ and ${\delta }^{\left(s\right)}~𝒩\left(0,{\sigma }^2\right)$ for $\sigma \in \{\mathrm{0.1,1},3\}$. For each setting, we perform 100 Monte Carlo replications. Instead of evaluating the predictive performance on the underlying non-missing data as in Yu et al. (2020), the validation and testing sets are generated with the same missing pattern as specified in Figure 5, and predictions are made based on observed modalities. The proposed method aims to improve the predictive performance for all data sources, regardless of missing patterns, while (Yu et al. 2020) focuses on estimating the underlying homogeneous coefficients accurately.

Figure 6 shows the root mean squared error (RMSE) under various settings of $I,\sigma$ for $L=3$, where DISCOM is only applicable. The results for $L=\mathrm{1,2}$ can be found in Appendix C.2. As $\sigma$ increases, all methods perform worse, while our $B{R}^2$ learning performs consistently better than the baseline STL and DISCOM, especially in the highly heterogeneous scenarios (e.g., $I=0,\sigma =3$). In contrast, DISCOM can only outperform the baseline STL method when $I=2$ and $\sigma =0.1$, which is sensible because their homogeneous model assumption.

Figure 6.

Simulation results for blockwise missing learning under varying settings: $I\in \{\mathrm{0,1},2\},\sigma \in \{\mathrm{0.1,1},3\}$ and $L=3$.

6. Real Data Analysis

In this section, we apply our proposed method, $B{R}^2$ learning to ADNI data (Mueller et al. 2005) to evaluate its performance. Existing methods compared in Section 5 have also been assessed in these examples. Additionally, we analyze the pan-cancer example where all the features are consistently observed in all data sources and $R^2$ learning applies. The detailed analysis and results can be found in Appendix G.

Our analysis aims to learn an accurate predictor for cognitive status, measured by the Mini-Mental State Examination (MMSE, Folstein, Folstein, and McHugh 1975). The covariates are from three modalities: magnetic resonance imaging (MRI), positron emission tomography (PET) and gene expression. The block missing structure emerges in the second phase of the ADNI study at the 48th month, where the detailed blockwise missing pattern is shown in Figure 7. In our analysis, we follow the pre-processing steps in (Xue and Qu 2021) which extract the region of interest (ROI) level data in ADNI and obtain $q_1=267$ features for MRI and $q_2=113$ feature for PET. For the genetic features, sure independence screening (SIS, Fan and Lv 2008) is adopted and $q_3=300$ genetic features are retained. In total, we use $p=q_1+q_2+q_3=680$ features to build our predictor, and four data sources with varying observation patterns are considered, where the first source with $n_1$ observe all three modalities, while the rest three sources have one different missing modality, respectively.

Figure 7.

Blockwise missing pattern of ADNI data. White blocks are missing modalities.

In this analysis, we compare our method with Single Task Learning (STL), DISCOM (Yu et al. 2020), and mean-imputation followed by $R^2$ learning ($R^2$-Impute). To evaluate the performance of our proposed $B{R}^2$ learning and competing methods, we split the data into training, validation and test set randomly with proportion 60%, 20%, and 20%, respectively. Note that the sampling procedure follows stratified sampling, for example, the training set consists of 60% samples from each data source. We measure the performance with root mean squared error (RMSE) and repeat the above sampling procedure 100 times. The results are shown in Table 1.

Table 1.

RMSEs of proposed $B{R}^2$ learning and competing methods on four data sources.

Data source	$B{R}^2$ learning	$R^2$-Impute	STL	DISCOM
1	3.687(1.186)	3.763(1.042)	3.901(1.167)	12.743(7.881)
2	3.047(1.142)	2.887(1.128)	3.049(0.686)	27.082(1.755)
3	5.132(1.570)	5.203(1.588)	6.376(1.353)	14.385(6.136)
4	3.028(1.050)	3.493(1.161)	3.922(1.014)	13.172(7.149)

The bold values are minimal RMSE achieved for each data source

The proposed $B{R}^2$ learning achieves the minimal RMSE over three data sources, and clearly improves performance compared to STL and DISCOM consistently, since our $B{R}^2$ learning can effectively integrate available information for block-missing data. Given the exploratory analysis in Appendix D, covariates shift and posterior drift are non-negligible across the four data sources in this dataset, which deteriorates the performance of DISCOM. In contrast, our method is robust against these sources of heterogeneity and effectively extract shared information to achieve more accurate prediction. In addition, $R^2$-Impute also improves predicitive performance over STL across all data sources, and achieves the minimal RMSE in the second data source. Such results reflect the debate between imputation-based and imputation-free methods for handling missing data problems. As an imputation-free approach, $B{R}^2$ learning achieves significant improvement over DISCOM, however, it is still an open question that which type of methods should be adopted for a real data problem with few knowledge of missing mechanism.

7. Discussion

This article investigates data integration in supervised learning under three types of heterogeneity: covariate shift, posterior drift, and blockwise missingness. Accommodating these heterogeneity enables practitioners to integrate a larger number of data sources and improve predictive performance. Moreover, we introduce the notion of integrativeness for representers and propose a novel Selective Integration Penalty that promotes more integrative representers in the dictionary, thereby enhancing model generalization both empirically and theoretically. The effectiveness of our approach is demonstrated through extensive simulation studies and real data applications.

Several directions for future research emerge from our work. First, in the multi-modality block-missing problem, our method exploits all observed modalities to improve prediction accuracy, following the imputation-free strategy. Alternative approaches, summarized as imputation-based strategy (e.g., Xue and Qu 2021; Ma, Wang, and Samworth 2025) remains prevalent. Our numerical studies suggests there is no uniform winner between these two strategies, it is still unclear which strategy is favorable under specific conditions. Future work could explore quantitative criteria to distinguish the circumstances favoring one strategy over the other. Second, it is of great interest to develop predictive inference procedures under heterogeneous data. Although our method and existing methods show promising performance in prediction, it is still unclear how to borrow information across heterogeneous data to improve the uncertainty quantification, say, narrower predictive interval. Third, we consider a static data analysis paradigm that all data sources are collected and fixed beforehand. In real applications, it is common that data sources are collected in an online fashion. This problem is formulated as streaming data analysis (Luo and Song 2023; Luo, Wang, and Hector 2023) in statistics and continual learning (Rusu et al. 2016; Kirkpatrick et al. 2017; Van de Ven and Tolias 2019) in machine learning. It is of great interest to extend our $R^2$ or $B{R}^2$ learning to adapt online data integration.

Supplementary Material

Additional numerical experiments, data analysis, and technical details of theoretical results can be found in supplementary materials.