Robust angle-based transfer learning in high dimensions

Abstract

Transfer learning improves target model performance by leveraging data from related source populations, especially when target data are scarce. This study addresses the challenge of training high-dimensional regression models with limited target data in the presence of heterogeneous source populations. We focus on a practical setting where only parameter estimates of pretrained source models are available, rather than individual-level source data. For a single source model, we propose a novel angle-based transfer learning (angleTL) method that leverages concordance between source and target model parameters. AngleTL adapts to the signal strength of the target model, unifies several benchmark methods, and mitigates negative transfer when between-population heterogeneity is large. We extend angleTL to incorporate multiple source models, accounting for varying levels of relevance among them. Our high-dimensional asymptotic analysis provides insights into when a source model benefits the target model and demonstrates the superiority of angleTL over other methods. Extensive simulations validate these findings and highlight the feasibility of applying angleTL to transfer genetic risk prediction models across multiple biobanks.

1. Introduction

Insufficient training data present a critical challenge across various domains. In finance, the scarcity of comprehensive user credit histories hampers efforts in evaluating individual financial risk and detecting fraud (Teja, 2017). Similarly, in precision medicine, the limited availability of medical records, especially for studying rare diseases or minority and disadvantaged sub-populations, compromises both the fairness and clinical efficacy of treatments (Jia & Shi, 2017; Kim & Milliken, 2019). These constraints highlight an urgent need for innovative approaches that can enhance model performance in the face of data limitations (X. Li & Song, 2020; Viele et al., 2014; S. Yang & Kim, 2020).

When borrowing information from external sources, a major challenge is to account for the potential data heterogeneity. The external data may likely be collected from a different population with diverse characteristics (Chen et al., 2020) or be historical data where the variable definitions and measures may change over time (Mansukhani et al., 2019; Mitchell et al., 2021). Many data integration methods are proposed to leverage potentially shared information across populations while addressing data heterogeneity. For example, some methods are based on reweighting or resampling the source data such that they are more similar to the target (Huang et al., 2006; Long et al., 2013; Pan et al., 2010); some methods assume there is a unique lower dimensional representation of features across populations, which can be transferred from the source to the target (Ganin & Lempitsky, 2015; Sun & Saenko, 2016); some methods propose to use the target data to calibrate the source models (Girshick et al., 2014; Gu et al., 2022; S. Li et al., 2020). Recent research used minimax rate theory to quantify the benefit of utilizing heterogeneous source data under varying degrees of heterogeneity, study design, and privacy constraints (T. T. Cai et al., 2024). In situations where no data from the target population is available, methods have been proposed to combine source models aiming for distributional robustness by optimizing for the worst-case performance, assuming the target population can be represented as either a single source or a mixture of source populations (Meinshausen & Bühlmann, 2015). The performance of many of the aforementioned methods largely depends on whether the underlying assumptions regarding the similarity between the source and the target populations hold, which is usually unknown in practice. Therefore, it is desired to develop methods that can be adaptive to the underlying data heterogeneity or at least prevent the case where incorporating the source information leads to worse model performance than not including it, known as the ‘negative transfer’ phenomenon (Weiss et al., 2016).

In addition, there might be data-sharing constraints such that external sources cannot share individual-level data with the target study. Federated or distributed algorithms are proposed to overcome such data-sharing barriers by sharing only summary-level statistics across studies, many of which rely on sharing the gradients or higher-order derivatives of objective functions (T. Cai et al., 2021; R. Duan et al., 2018, 2020, 2022; S. Li et al., 2023) and may require iteratively sharing-updated summary statistics across datasets (S. Li et al., 2023). However, distributed and federated learning are less feasible without a collaborative environment or certain infrastructures that enable efficient computing and timely information sharing. In contrast, pretrained models from existing studies are often more accessible. With increasing attention to reproducibility and open science, more journals require studies to publish their results as online supplementary materials or to make them shareable upon request (Roobol et al., 2012; Thompson et al., 2006). Many platforms allow direct implementation (Belbasis & Panagiotou, 2022) or validation of fitted models on secure collaborative platforms such as PheKB (Kirby et al., 2016) and FATE (Liu et al., 2021). There is an increasing need for data integration methods that can directly leverage fitted models to improve the model performance of a target study.

Regression models are broadly applied in many fields, due to advantages such as simplicity, computational efficiency, and interpretability (Hafemeister & Satija, 2019; Wu et al., 2020; Wynants et al., 2020), and they are also the building blocks of many data analysis pipelines and tools (Tan, 2006; Van Buuren et al., 2006). In this paper, we consider the problem of incorporating pretrained regression models from external sources to help train a target model with limited target data. For the ith subject in the target data, let $Y_i\in \mathbb{R}$ denote the outcome variable of interest and ${\mathbf{X}}_i\in {\mathbb{R}}^p$ denote a set of p-dimensional covariates. We consider

$$Y_i={\mathbf{X}}_i^{\mathrm{\top }}\mathit{\beta }+{ϵ}_i,\text{for }i\in \{1,\dots ,n\},$$

where ${ϵ}_i\in \mathbb{R}$ is the random noise with mean zero and variance ${\sigma }^2$, $\mathit{\beta }\in {\mathbb{R}}^p$ is the regression coefficient of interest, and n is the target sample size. In addition to the target data, we observe a source model fitted on an external source dataset, where we observe the source estimate $\hat{\mathbf{w}}\in {\mathbb{R}}^p$ of the underlying regression coefficient $\mathbf{w}$ in the source population. In the case where the underlying coefficients $\mathit{\beta }$ and $\mathbf{w}$ show certain similarities, we hope to leverage $\hat{\mathbf{w}}$ to assist the estimation of the target parameter $\mathit{\beta }$.

Similar problems have been considered in the literature. In a series of transfer learning work, the source model estimate $\hat{\mathbf{w}}$ was incorporated through a regularization with the form $\Vert \mathit{\beta }-\hat{\mathbf{w}}{\Vert }_q$, for some positive constant q. For example, S. Li et al. (2020) proposed the transLASSO algorithm that leverages the source data by adding a penalty term $\Vert \mathit{\beta }-\hat{\mathbf{w}}{\Vert }_1$ when learning $\mathit{\beta }$ using the target data. Later, transLASSO has been extended to generalized linear models (Tian & Yang, 2022), functional linear regression (Lin & Reimherr, 2022), and Q-learning. Considering data sharing constraints, S. Li et al. (2023) proposed a federated learning approach by sharing gradients and Hessian matrices, and Gu et al. (2023) proposed a method to incorporate information from fitted models through a synthetic data approach. Similar ideas are also used in multitasking learning literature, where $L_2$-norm-based penalties, i.e. $\Vert \mathit{\beta }-\mathbf{w}{\Vert }_2$, are introduced to leverage the similarities between model parameters (Y. Duan & Wang, 2023; Knight & Duan, 2024). The underlying similarity assumption of these methods is that the $L_q$ distance between $\mathit{\beta }$ and $\mathbf{w}$ is small, and we refer to this class of methods as the distance-based transfer learning methods.

In real-world applications, the distance-based transfer learning methods may be less effective when $\mathit{\beta }$ and $\mathbf{w}$ are highly concordant, but their distance may not be small. For example, the source outcome may be defined differently from the target outcome (categorical versus continuous characterization of the same outcome), or there might be standardization procedures applied to the source data that are unknown to us. The source model might be fitted using a different but related outcome variable correlated with the target outcome (Miglioretti, 2003; Stearns, 2010). To leverage the concordance between the model parameters, several alternative similarity characterizations are proposed. For example, C. Li et al. (2014) proposed a bivariate ridge regression assuming the jth entries $({\beta }_j,w_j)$ follows a bivariate normal distribution with shared correlation ρ for all $j\in [p]$. Similarly, Maier et al. (2014) proposed a multitrait prediction method, assuming multivariate normally distributed random effects to leverage shared genetic architectures across traits. Qiao et al. (2019) proposed a method for multi-objective optimization problems. However, these methods require individual-level data from the source and their robustness is unclear when the level of data heterogeneity is high.

In this paper, we propose an angle-based Transfer Learning approach, named angleTL, which leverages the similarity of two regression models through a novel penalization obtained by decoupling the angle distance between $\mathit{\beta }$ and $\mathbf{w}$. As a consequence, angleTL adapts to signal strength and inherently guards against negative transfer, offering a simpler form both conceptually and computationally. Our approach unifies distance-based transfer learning, target-only and source-only estimators as specific instances, and thus is guaranteed to have superior performance. The high-dimensional asymptotic analysis provides a precise characterization of the prediction risk, illustrating the bias-variance trade-off, the influence of signal strengths, the similarity between source and target, the noise level, and the estimation error of the source to the accuracy of the final estimator. Given multiple source models, we propose methods to effectively incorporate them toward better predictive performance in the target population. The proposed methods only require parameter estimates of fitted source models, which are more accessible in practice. We perform extensive simulation studies to validate our theoretical conclusions and evaluate the performance of angleTL by training genetic risk models for low-density lipoprotein cholesterol (LDL) using data from multiple large-scale biobanks.

2. Angle-based transfer learning

Before introducing the method, we first provide a high-level introduction to the notation used in this paper. Let $\mathbf{Y}\in {\mathbb{R}}^n$ denote the outcome variable of interest and $\mathbf{X}\in {\mathbb{R}}^{n\times p}$ denote a set of p-dimensional covariates in the target data of size n. We use $\mathit{\beta }$ to represent the target parameter, while $\mathbf{w}$ represents the source parameter. When there is more than one source population, we use subscript $k\in [1,\dots ,K]$ to denote different sources, i.e. ${\mathbf{w}}_k$. Parameters without a hat represent the true population level parameter, while those with a hat represent the estimator of the true parameter. In general, subscript t represents the values with respect to the target population, and the subscript s represents the values with respect to the source population. We use $\mathbf{\Sigma }$ to denote the variance–covariance matrix of the training data $\mathbf{X}$, where ${\mathbf{\Sigma }}_{\mathbf{a}}$ with subscript $\mathbf{a}$ indicating $\mathbb{E}\{\mathbf{a}{\mathbf{a}}^{\mathrm{\top }}\}$ of a random vector $\mathbf{a}$. We denote the identity matrix with dimension p as ${\mathbf{I}}_p$. The above-mentioned ones are general notation rules, and we also summarize the specific definitions of notation in Appendix A in the online supplementary material.

Without borrowing information from the source, we can obtain a target-only estimator of $\mathit{\beta }$ through a ridge regression,

$${\tilde{\mathit{\beta }}}_{\lambda }={\displaystyle \underset{\mathbf{\beta }}{\arg min}}{\displaystyle \frac{1}{n}\Vert \mathbf{Y}-\mathbf{X}\beta {\Vert }_2^2+\lambda \Vert \mathit{\beta }{\Vert }_2^2,}$$ (1)

where λ is a tuning parameter. The penalty on $\Vert \mathit{\beta }{\Vert }_2^2$ helps reduce the overall mean squared error (MSE) of the estimator in the high-dimensional setting (Hastie et al., 2009).

Suppose that we also observe $\hat{\mathbf{w}}\in {\mathbb{R}}^p$, the parameter estimates of a source model fitted on an external source dataset with a sample size potentially much larger than n. Due to data heterogeneity, the two underlying regression coefficients, $\mathit{\beta }$ and $\mathbf{w}$, may not be the same but may share certain similarities, and hence $\hat{\mathbf{w}}$ can be used to guide the estimation of $\mathit{\beta }$. Following a series of recent transfer learning methods (Gu et al., 2023; S. Li et al., 2023, 2022), we define the $L_2$-distance-based transfer learning estimator (distTL) to be

$${\check{\beta }}_{{\lambda }_d}={\displaystyle \underset{\mathit{\beta }}{\arg min}}{\displaystyle \frac{1}{n}\Vert \mathbf{Y}-\mathbf{X}\beta {\Vert }_2^2+{\lambda }_d\Vert \mathit{\beta }-\hat{\mathbf{w}}{\Vert }_2^2,}$$ (2)

where ${\lambda }_d$ is a tuning parameter. Imposing a distance-based penalty, $\Vert \mathit{\beta }-\hat{\mathbf{w}}{\Vert }_2$, is equivalent to adding a constraint $\Vert \mathit{\beta }-\hat{\mathbf{w}}{\Vert }_2\le h$, which reduces the parameter space to an $L_2$ ball centred at $\hat{\mathbf{w}}$ as shown in Figure 1a. Anchoring on the source estimator $\hat{\mathbf{w}}$, distTL encourages estimators closer to $\hat{\mathbf{w}}$ while allowing for the calibration of potential differences.

Figure 1.

Geometric illustration of the distance-based similarity characterization (a); the angle-based characterization (b).

In real-world applications, it is possible that $\mathit{\beta }$ and $\mathbf{w}$ are concordant to some degree, but $\Vert \mathit{\beta }-\mathbf{w}{\Vert }_2$ is not small. In such cases, distTL may be less effective and we want to consider a more robust characterization of the similarity based on the angle distance between $\mathbf{w}$ and $\mathit{\beta }$, such that if $\mathbf{w}$ and $\mathit{\beta }$ are small in $L_2$ distance, it also implies that the angle between the two vectors is small, while it may not be true the other way around.

Instead of directly penalizing on $\sin \mathrm{\Theta }(\hat{\mathbf{w}},\mathit{\beta })=\sqrt{1-\frac{({\hat{\mathbf{w}}}^{\mathrm{\top }}\mathit{\beta }{)}^2}{\Vert \hat{\mathbf{w}}{\Vert }_2^2\Vert \mathit{\beta }{\Vert }_2^2}}$ which may lead to computational complexity, we consider an alternative angle-based transfer learning estimator defined as

$${\hat{\mathit{\beta }}}_{\lambda ,\eta }={\displaystyle \underset{\mathit{\beta }}{\arg min}}{\displaystyle \frac{1}{n}\Vert \mathbf{Y}-\mathbf{X}\beta {\Vert }_2^2+\lambda \Vert \mathit{\beta }{\Vert }_2^2-2\eta {\hat{\mathbf{w}}}^{\mathrm{\top }}\mathit{\beta },}$$ (3)

where $\lambda$ and $\eta$ are tuning parameters.

To see the connection between the proposed penalty terms and the $\sin \mathrm{\Theta }$ distance, we denote the projection of $\mathit{\beta }$ on to $\mathbf{w}$ by $\tilde{\mathbf{w}}$ (see Figure 1), and we see that the $L_2$ distance of $\Vert \mathit{\beta }-\tilde{\mathbf{w}}\Vert$ is expected to be small when the angle between the two vectors are small. If we penalize $\Vert \mathit{\beta }-\tilde{\mathbf{w}}{\Vert }_2$ instead of $\Vert \mathit{\beta }-\hat{\mathbf{w}}{\Vert }_2$, we essentially can leverage the angular similarity between the two models. The projection $\tilde{\mathbf{w}}$ can be expressed as $\tilde{\mathbf{w}}=\tau \hat{\mathbf{w}}$, where $\tau =\frac{\cos \theta \Vert \mathit{\beta }{\Vert }_2}{\Vert \hat{\mathbf{w}}{\Vert }_2}$. We see that

$$\lambda \Vert \mathit{\beta }-\tau \hat{\mathbf{w}}{\Vert }_2^2\propto \lambda \Vert \mathit{\beta }{\Vert }_2^2-2\lambda \tau {\hat{\mathbf{w}}}^T\mathit{\beta }.$$

The key innovation of our method is that we do not choose to express τ as a function of $\mathit{\beta }$. Instead, we consider it as an unknown scalar, and introduce another tuning parameter $\eta =\lambda \tau$. We choose the best tuning parameters that minimize the predictive loss evaluated on an independent dataset or using cross-validation, and we will show both theoretically and empirically that the proposed penalty in the form of $\lambda \Vert \mathit{\beta }{\Vert }_2^2-2\eta {\hat{\mathbf{w}}}^{\mathrm{\top }}\mathit{\beta }$ can achieve angle-based learning.

The proposed penalization has several advantages. Firstly, given λ and η, equation (3) has a closed-form solution ${\hat{\mathit{\beta }}}_{\lambda ,\eta }=({\mathbf{X}}^{\mathrm{\top }}\mathbf{X}+n\lambda {\mathbf{I}}_p{)}^{-1}({\mathbf{X}}^{\mathrm{\top }}\mathbf{Y}+n\eta \hat{\mathbf{w}})$, which ensures computational efficiency. Secondly, it is adaptive to the signal strength of the target parameter. Specifically, when $\Vert \mathit{\beta }{\Vert }_2$ is large, the constraint ${\hat{\mathbf{w}}}^{\mathrm{\top }}\mathit{\beta }>b$ can be satisfied for $\mathit{\beta }$’s with large $\sin \mathrm{\Theta }$ distance with $\hat{\mathbf{w}}$. In this scenario, the angle constraint is relatively minimal, and less information is borrowed from the source. Conversely, when $\Vert \mathit{\beta }{\Vert }_2$ is small, there is a tighter constraint on the angle distance, which results in borrowing more information from the source. Intuitively, with a sufficiently strong signal, relying more on target data can yield precise estimations. However, in situations with weak signal strength, and given that the target sample size might be limited, there is a greater need to leverage information from the source. The adaptivity to the signal strength is verified in our theoretical analysis discussed in Section 3. Thirdly, although our proposed penalty is termed as angle-based penalty, it incorporates several estimators as special cases: (i) when $\eta =0,{\hat{\mathit{\beta }}}_{\lambda ,\eta }$ reduces to the target-only estimator shown in equation (1), (ii) when $\eta =\lambda ,{\hat{\mathit{\beta }}}_{\lambda ,\eta }$ becomes distTL in equation (2), since the optimization remains the same if we add an additional $\mathit{\beta }$-independent term $\lambda \Vert \hat{\mathbf{w}}{\Vert }_2^2$, i.e. $\lambda \Vert \mathit{\beta }{\Vert }_2^2-2\lambda {\hat{\mathbf{w}}}^{\mathrm{\top }}\mathit{\beta }+\lambda \Vert \hat{\mathbf{w}}{\Vert }_2^2=\lambda \Vert \mathit{\beta }-\hat{\mathbf{w}}{\Vert }_2^2$; and (iii) if λ and η are both large, the penalty term $-2\eta {\hat{\mathbf{w}}}^{\mathrm{\top }}\mathit{\beta }$ will dominate and result in $\mathit{\beta }\approx c\hat{\mathbf{w}}$, the same as rescaling the source estimator by a constant c.

From the perspective of bias-variance trade-off, we see that when $\lambda =\eta =0$, our estimator ${\hat{\mathit{\beta }}}_{\lambda ,\eta }$ reduces to the ordinary least square estimator, which is an unbiased estimator of $\mathit{\beta }$. On the other extreme, if λ goes to infinity faster than η, ${\hat{\mathit{\beta }}}_{\lambda ,\eta }$ converges to 0, which has a large bias and small variance. The contribution of η to the bias-variance trade-off depends on the underlying similarity between $\mathbf{w}$ and $\mathit{\beta }$. When $\hat{\mathbf{w}}=\mathit{\beta }$, if choosing $\eta =\lambda$ and letting $\eta \to \mathrm{\infty }$, ${\hat{\mathit{\beta }}}_{\lambda ,\eta }$ converges to $\mathit{\beta }$. In more realistic cases when $\hat{\mathbf{w}}\ne \mathit{\beta }$, there exists optimal η and λ which minimize MSE. In Section 3, we will elaborate on these points under a rigorous theoretical framework, and provide precise characterizations of angleTL’s performance, revealing its scope of applicability and the advantages over alternative approaches.

3. Predictive risk and theoretical justifications

In this section, we study the predictive risk of ${\hat{\mathit{\beta }}}_{\lambda ,\eta }$ in equation (3). For a given choice of λ and η, we denote the prediction risk by

$$r_{\lambda ,\eta }(\mathbf{X})=\mathbb{E}\{({\mathbf{x}}^{\mathrm{\top }}{\hat{\mathbf{\beta }}}_{\lambda ,\eta }-\mathbf{y}{)}^2|\mathbf{X}\},$$

where $\mathbf{X}$ is the training data, and the expectation is taken over an independent test data point ($\mathbf{x},\mathbf{y}$), from the same distribution as the training data. Given λ and η, we denote $\mathcal{R}(\lambda ,\eta )=\underset{(n,p)\to \mathrm{\infty }}{lim}{r}_{\lambda ,\eta }(\mathbf{X})$, the limiting prediction risk as the training sample size n and the dimension p go to infinity. Our theoretical analysis is based on the following assumptions.

Assumption 1 High-Dimensional Asymptotics —

1. The dimension of the model, p, goes to infinity as n goes to infinity, with $\frac{p}{n}\to \gamma \in (0,\mathrm{\infty })$.

2. The covariate variables of each observation in the target population are identically and independently distributed with mean zero and covariance matrix $\mathbf{\Sigma }\in {\mathbb{R}}^{p\times p}$, which is positive semidefinite.

3. The cumulative distribution function of the eigenvalues, i.e. the spectral distribution, $F_{\mathbf{\Sigma }}$ of $\mathbf{\Sigma }$ converges to a limit population spectral distribution on the support $[0,\mathrm{\infty })$.

Assumption 2 Random Regression Coefficients —

The target regression coefficients, $\mathit{\beta }\in {\mathbb{R}}^p$, are random with $\mathbb{E}[\mathit{\beta }]=\mathbf{0}$, and $Var[\mathit{\beta }]=\frac{{\alpha }_t^2}{p}{\mathbf{I}}_p$. The source coefficient $\mathbf{w}$ are random with $\mathbb{E}[\mathbf{w}]=\mathbf{0}$ and $Var[\mathbf{w}]=\frac{{\alpha }_s^2}{p}{\mathbf{I}}_p$. The coefficients $\mathit{\beta }$ and $\mathbf{w}$ satisfy $\mathrm{Cov}(\mathit{\beta },\mathbf{w})=\frac{\rho {\alpha }_t{\alpha }_s}{p}{\mathbf{I}}_p$.

Assumption 3 Source Estimation Error —

The observed source estimator $\hat{\mathbf{w}}$ can be decomposed as $\hat{\mathbf{w}}=\mathbf{w}+\mathit{\delta }$, where $\mathit{\delta }$ is some sub-Gaussian estimation error vector independent of $\mathbf{w}$ and $\mathit{\beta }$, with $\mathbb{E}[\mathit{\delta }{\mathit{\delta }}^{\mathrm{\top }}]=\frac{1}{p}{\mathbf{\Sigma }}_{\delta }$. We assume that as $(n,p)\to \mathrm{\infty }$, the eigenvalues of ${\mathbf{\Sigma }}_{\delta }$ is supported on $[C_L,C_U]$ for some constants $C_U\ge C_L\ge 0$.

A few remarks on the above assumptions are in order. In general, these assumptions together help us leverage results from random matrix theory to compute the prediction risk of the proposed method, and enable rigorous comparisons with alternative methods. In particular, Assumption 1(a) concerns the limit of the ratio $\frac{p}{n}$ as $n\to \mathrm{\infty }$, reflecting the high-dimensional nature of the problem. Assumption 1(b) is a mild condition on the design matrix, which requires independent samples but allows general correlations among high-dimensional covariates. Assumption 1(c) ensures the existence of proper limit for the eigenvalues of the underlying covariance matrix $\mathbf{\Sigma }$ as $p\to \mathrm{\infty }$, which can be satisfied by a large class of covariance matrices with diagonal, exchangeable, or auto-regressive structures (Bai & Silverstein, 2010). Assumption 2 is a random-effect hypothesis assuming that the true effect size of each feature is drawn independently at random, and the overall signal strength captured in the target (source) model is characterized by the quantity ${\alpha }_t^2$ (${\alpha }_s^2$), where $\mathbb{E}\Vert \mathit{\beta }{\Vert }_2^2={\alpha }_t^2$ and $\mathbb{E}\Vert \mathbf{w}{\Vert }_2^2={\alpha }_s^2$. This hypothesis enables us to study the precise predictive risk for angleTL, distTL, and target-only estimator, and draw clear insight on factors influencing their relative performance. Such a random-effect assumption is commonly seen in high-dimensional statistics and applications to genetic risk prediction (Dobriban & Wager, 2018; C. Li et al., 2014; Vilhjálmsson et al., 2015). Finally, Assumption 3 characterizes the estimation error of the source estimate $\hat{\mathbf{w}}$ for estimating $\mathbf{w}$. Depending on the correlation structure of the features, as well as the method used to obtain $\hat{\mathbf{w}}$, the estimation error $\mathit{\delta }$ may be biased or correlated, and we quantify the magnitude of the error by the upper and lower bounds of eigenvalues, i.e. $C_L$ and $C_U$. While these assumptions facilitate the theoretical analysis, our method can be applied to various other settings.

Before stating our main theorem, we define a complex function $v(z):\mathbb{C}\setminus {\mathbb{R}}^{+}\to \mathbb{C}$, which plays an important role in our description of the prediction risk $\mathcal{R}(\lambda ,\eta )$. Specifically, according to Assumption 1, the spectral distribution of the $n\times n$ matrix $\frac{1}{n}\mathbf{X}{\mathbf{X}}^{\mathrm{\top }}$ has a limit, which we denote as $F_{\underset{\_}{\mathbf{\Sigma }}}$. In particular, $F_{\underset{\_}{\mathbf{\Sigma }}}$ is related to $F_{\mathbf{\Sigma }}$ in Assumption 1(c) via the equation

$$F_{{\displaystyle \underset{\_}{\mathbf{\Sigma }}}}(x)-\gamma F_{\mathbf{\Sigma }}(x)=(1-\gamma )1\{x=0\}.$$

We define $v(z)$ to be the Stieltjes transform of the probability measure $F_{\underset{\_}{\mathbf{\Sigma }}}$, i.e.

$$v(z):={\int }_0^{\mathrm{\infty }}\frac{d{F}_{{\displaystyle \underset{\_}{\mathbf{\Sigma }}}}(x)}{x-z},z\in \mathbb{C}\setminus {\mathbb{R}}^{+}.$$

Theorem 1

Under Assumptions 1–3, as $n\to \mathrm{\infty }$, the limiting expected predictive risk $\mathcal{R}(\lambda ,\eta )$ satisfies

$$R(\lambda ,\eta ,C_L)\le \mathcal{R}(\lambda ,\eta )\le R(\lambda ,\eta ,C_U),$$

where

$$R(\lambda ,\eta ,C):=({\lambda }^2{\alpha }_t^2+{\eta }^2{\alpha }_s^2+{\eta }^{2}C-2\lambda \eta \rho {\alpha }_t{\alpha }_s)\frac{v(-\lambda )-\lambda v^{\prime }(-\lambda )}{\gamma (\lambda v(-\lambda ){)}^2}+\frac{{\sigma }^2{v}^{\prime }(-\lambda )}{v(-\lambda )}.$$

The minimal risk ${\mathcal{R}}^{*}=\underset{\lambda ,\eta }{min}\mathcal{R}(\lambda ,\eta )$ under the optimal choice of tuning parameters satisfies

$$\frac{{\sigma }^2}{{\lambda }_L^{*}v(-{\lambda }_L^{*})}\le {\mathcal{R}}^{*}\le \frac{{\sigma }^2}{{\lambda }_U^{*}v(-{\lambda }_U^{*})},$$

where ${\lambda }_L^{*}=\frac{\gamma {\sigma }^2}{{\alpha }_t^2(1-\frac{{\rho }^2{\alpha }_s^2}{{\alpha }_s^2+C_L})}$ and ${\lambda }_U^{*}=\frac{\gamma {\sigma }^2}{{\alpha }_t^2(1-\frac{{\rho }^2{\alpha }_s^2}{{\alpha }_s^2+C_U})}$.

The proof of the above theorem can be found in Appendix B of the online supplementary material, which relies on the asymptotic theory of eigenvalues of large random matrices and, in particular, the theoretical framework developed by Dobriban Wager (2018) for the analysis of ridge-penalized linear regression and linear discrimination analysis. Similar techniques have been used for the precise analysis of many other high-dimensional problems such as covariance matrix estimation (T. T. Cai et al., 2020; Ledoit & Wolf, 2015), low-rank matrix denoizing (Donoho & Gavish, 2014), sketching and random projection (F. Yang et al., 2021), etc.

Theorem 1 provides upper and lower bounds for the limiting prediction risk $\mathcal{R}(\lambda ,\eta )$ as a function of the parameters λ and η. The possible range of $\mathcal{R}(\lambda ,\eta )$ are determined by the spectrum of ${\mathbf{\Sigma }}_{\delta }$ in Assumption 3, which in general decreases as the estimation error $\mathit{\delta }$ becomes smaller. In addition, the difference between the upper and the lower bounds $R(\lambda ,\eta ,C_U)-R(\lambda ,\eta ,C_L)=\frac{(C_U-C_L){\eta }^2[v(-\lambda )-\lambda v^{\prime }(-\lambda )]}{\gamma {[\lambda v(-\lambda )]}^2}$ depends on the spectral range $(C_U-C_L)$ of ${\mathbf{\Sigma }}_{\delta }$, which becomes smaller when the components of $\mathit{\delta }$ are less heteroskedastic or less correlated. For example, if ${\mathbf{\Sigma }}_{\delta }=I_p$, we have $C_U=C_L$. The minimal risk ${\mathcal{R}}^{*}$ lies in the interval $[\frac{{\sigma }^2}{{\lambda }_L^{*}v(-{\lambda }_L^{*})},\frac{{\sigma }^2}{{\lambda }_U^{*}v(-{\lambda }_U^{*})}]$. To better understand its implications, we consider a few special scenarios of interest.

Corollary 1

When $C_L=C_U=C$, the optimal tuning parameters satisfy ${\lambda }^{*}=\frac{\gamma {\sigma }^2}{{\alpha }_t^2(1-\frac{{\rho }^2{\alpha }_s^2}{{\alpha }_s^2+C})}$ and ${\eta }^{*}={\lambda }^{*}\frac{\rho {\alpha }_t{\alpha }_s}{{\alpha }_s^2+C}$, and the minimal risk is ${\mathcal{R}}^{*}=\frac{{\sigma }^2}{{\lambda }^{*}v(-{\lambda }^{*})}$.

This corollary addresses a special case where $C_L=C_U=C$, under which the predictive risk described in Theorem 1 is precise. We further consider the case when $C\ll {\alpha }_s^2$, which can happen if the source estimator $\hat{\mathbf{w}}$ has negligible estimation error, the optimal choice of tuning parameters satisfies ${\lambda }^{*}=\frac{\gamma {\sigma }^2}{{\alpha }_t^2(1-{\rho }^2)}$ and ${\eta }^{*}={\lambda }^{*}\rho \frac{{\alpha }_t}{{\alpha }_s}$. Geometrically, ${\alpha }_t^2=\mathbb{E}\Vert \mathit{\beta }{\Vert }_2^2$ and ${\alpha }_s^2=\mathbb{E}\Vert \mathbf{w}{\Vert }_2^2$ can be viewed as the vector lengths of $\mathit{\beta }$ and $\mathbf{w}$, respectively. The entry-wise correlation ρ is essentially the cosine between $\mathbf{w}$ and $\mathit{\beta }$. The optimal choice of the tuning parameter ${\eta }^{*}=\rho \frac{{\alpha }_t}{{\alpha }_s}\mathbf{w}$ is the projection of $\mathit{\beta }$ along $\mathbf{w}$. This is equivalent to penalizing the distance $\Vert \mathit{\beta }-\tilde{\mathbf{w}}{\Vert }_2^2$, which achieves angle-based learning although we did not explicitly penalize on the $\sin \mathrm{\Theta }$ distance. When $\hat{\mathbf{w}}$ has nonnegligible estimation errors, the optimal tuning parameter satisfies ${\eta }^{*}={\lambda }^{*}\frac{\rho {\alpha }_t{\alpha }_s}{{\alpha }_s^2+C}$. By simple derivation and with Assumption 3, we see that $C=\mathbb{E}\Vert \mathit{\delta }{\Vert }_2^2$, and $\frac{\rho {\alpha }_t{\alpha }_s}{{\alpha }_s^2+C}{\mathit{\beta }}^T\hat{\mathbf{w}}$ is the projection of $\mathit{\beta }$ on to $\hat{\mathbf{w}}$.

This corollary shows, in theory, the proposed angleTL penalty $\lambda \Vert \beta {\Vert }_2^2-2\eta {\mathit{\beta }}^{\mathrm{\top }}\hat{\mathbf{w}}$ achieves angle-based transfer learning under the optimal choices of tuning parameters. We also provide empirical evidence that the optimal tuning parameter chosen by cross-validation is reasonably close to the theoretical optimal tuning parameters across various settings (see Section 4.1 in Appendix D of the online supplementary material).

Corollary 2

When ${\alpha }_t^2\to \mathrm{\infty }$, we have that

$${\mathcal{R}}^{*}\to \frac{{\sigma }^2}{1-\gamma },\mathrm{when}\gamma <1$$ (4)

and

$${\mathcal{R}}^{*}=\frac{{\alpha }_t^2(1-\frac{{\rho }^2{\alpha }_s^2}{{\alpha }_s^2+C})}{\gamma v(0)}(1+o(1)),\mathrm{when}\gamma >1.$$ (5)

When ${\alpha }_t^2\to 0$, we have

$${\mathcal{R}}^{*}={\sigma }^2+{\sigma }^2{\alpha }_t^2(1-\frac{{\rho }^2{\alpha }_s^2}{{\alpha }_s^2+C})T(1+o(1)),$$ (6)

where $T=\underset{n\to \mathrm{\infty }}{lim}\frac{1}{p}\text{tr}(\mathbf{\Sigma })$.

From the above corollary, we see that when $\gamma <1$ ($p<n$) and ${\alpha }_t^2\to \mathrm{\infty }$, the prediction risk reduces to that of the ordinary least squares, and is independent of the covariance $\mathbf{\Sigma }$ and the source data (such as ρ, ${\alpha }_s^2$, or C). When $\gamma >1$ ($p>n$), the limiting error as ${\alpha }_t^2\to \mathrm{\infty }$ depends on the covariance $\mathbf{\Sigma }$ through $v(0)$, and the source estimator through $1-\frac{{\rho }^2{\alpha }_s^2}{{\alpha }_s^2+C}$. In the special case where $\mathbf{\Sigma }={\mathbf{I}}_p$, we have $v(0)=\frac{1}{\gamma -1}$, so that the prediction risk decreases when the sample size increases, when the source-target similarity ${\rho }^2$ increases, or when the relative source estimation error $\frac{C}{{\alpha }_s^2}$ decreases. On the other hand, when the target signal strength is very small, i.e. ${\alpha }_t^2\to 0$, the prediction risk no longer depends on γ.

Corollary 3

Denote the minimal predictive risk of the target-only estimator to be ${\mathcal{R}}^{†}$ and the minimal predictive risk of the distance-based transfer learning to be ${\mathcal{R}}^{\mathrm{\#}}$. For any ${\sigma }^2\ge 0$, $0\le \rho \le 1$, ${\alpha }_t^2\ge 0$, ${\alpha }_s^2\ge 0$, and $0\le C_L\le C_U$, we have ${\mathcal{R}}^{*}\le {\mathcal{R}}^{†}$ and ${\mathcal{R}}^{*}\le {\mathcal{R}}^{\mathrm{\#}}$. In the case when ${\alpha }_s^2+C_L<2\rho {\alpha }_t{\alpha }_s$, we have ${\mathcal{R}}^{†}<{\mathcal{R}}^{\mathrm{\#}}$.

This corollary suggests that angleTL can avoid negative transfer due to imprecise or low-quality source estimators, with performance no worse than the target-only estimator. Specifically, the limiting predictive risk of the target-only estimator ${\tilde{\mathit{\beta }}}_{\lambda }$ has been obtained in Dobriban and Wager (2018), which has a minimum risk ${\mathcal{R}}^{†}=\frac{{\sigma }^2}{{\lambda }^{†}v(-{\lambda }^{†})}$ under the optimal choice of tuning parameters ${\lambda }^{†}=\gamma \frac{{\sigma }^2}{{\alpha }_t^2}$. As a comparison, the minimal risk ${\mathcal{R}}^{*}$ of the angleTL estimator is always smaller than ${\mathcal{R}}^{†}$. To see this, we first notice that ${\mathcal{R}}^{*}$ is bounded by ${\mathcal{R}}_U^{*}=\frac{{\sigma }^2}{{\lambda }_U^{*}v(-{\lambda }_U^{*})}$. Since $\frac{1}{\lambda v(-\lambda )}$ is monotonically decreasing with λ, and that ${\lambda }^{†}=\gamma \frac{{\sigma }^2}{{\alpha }_t^2}\le {\lambda }^{*}=\frac{\gamma {\sigma }^2}{{\alpha }_t^2(1-\frac{{\rho }^2{\alpha }_s^2}{{\alpha }_s^2+C_U})}$ for any $\rho \ge 0$, ${\alpha }_s^2\ge 0$ and $C_U>0$, we have that ${\mathcal{R}}^{*}\le {\mathcal{R}}_U^{*}\le {\mathcal{R}}^{†}$. This indicates that the angleTL estimator safeguards against negative transfer because angleTL is always no worse than the target-only estimator, even when the source has a large distance to the target, or the estimation error is large. Similarly, the minimal risk of the distance-based transfer learning satisfies ${\mathcal{R}}_L^{\mathrm{\#}}=\frac{{\sigma }^2}{{\lambda }_L^{\mathrm{\#}}v(-{\lambda }_L^{\mathrm{\#}})}\le {\mathcal{R}}^{\mathrm{\#}}$, with ${\lambda }_L^{\mathrm{\#}}=\frac{\gamma {\sigma }^2}{{\alpha }_s^2+{\alpha }_t^2+C_L-2\rho {\alpha }_t{\alpha }_s}$. When ${\alpha }_s^2+C_L<2\rho {\alpha }_t{\alpha }_2$, we have ${\mathcal{R}}^{†}<{\mathcal{R}}_L^{\mathrm{\#}}\le {\mathcal{R}}^{\mathrm{\#}}$, implying the distance-based transfer learning performs worse than the target-only approach which cannot prevent negative transfer. In related work using distance-based transfer penalization (S. Li et al., 2023, 2020), an independent validation dataset is often used to prevent negative transfer, which is not required in our method.

Furthermore, angleTL is expected to have performance no worse than distTL. The precise difference between the minimal risks of angleTL and distTL can be characterized under the special case when $C_U=C_L=C$, where ${\mathcal{R}}^{*}=\frac{{\sigma }^2}{{\lambda }^{*}v(-{\lambda }^{*})}$ with ${\lambda }^{*}=\frac{\gamma {\sigma }^2}{{\alpha }_t^2(1-\frac{{\rho }^2{\alpha }_s^2}{{\alpha }_s^2+C})}$ and ${\mathcal{R}}^{\mathrm{\#}}=\frac{{\sigma }^2}{{\lambda }^{\mathrm{\#}}v(-{\lambda }^{\mathrm{\#}})}$, with ${\lambda }^{\mathrm{\#}}=\frac{\gamma {\sigma }^2}{{\alpha }_s^2+{\alpha }_t^2+C-2\rho {\alpha }_t{\alpha }_s}$. We visualize the difference in the following Figure 2 with $C=0,\rho =0.9,{\alpha }_t/{\alpha }_s=2$, and different choices of γ. With $C=0$, ${\lambda }^{*}={\eta }^{*}$ if and only if $\rho \frac{{\alpha }_t}{{\alpha }_s}=1$. Geometrically, this is when the difference $\mathit{\beta }-\mathbf{w}$ is orthogonal to $\mathbf{w}$.

Figure 2.

The difference of prediction error between disTL and angleTL over increasing sample size to dimension ratio γ, when the source estimate is noiseless $C=0$, correlation between $\mathit{\beta }$ and $\mathbf{w}$  $\rho =0.9$, and signal strength ratio ${\alpha }_t/{\alpha }_s=2$.

4. Incorporating multiple source models

In this section, we consider the setting where multiple source models are available, denoted as $\{{\hat{\mathbf{w}}}_k{\}}_{k=1,\dots ,K}$, and we want to incorporate these source models when fitting a model using the internal target data. In the existence of multiple sources, we hope that our estimator obtained from transfer learning is robust to the cases where some source models are less helpful than others.

One direct extension of angleTL is to include multiple penalty terms, one for each source estimates ${\hat{\mathbf{w}}}_k$, and obtain

$${\hat{\mathit{\beta }}}_{\lambda ,{\eta }_1,\dots ,{\eta }_K}={\displaystyle \underset{\mathit{\beta }}{\arg min}}{\displaystyle \frac{1}{n}\Vert \mathbf{Y}-\mathbf{X}\mathit{\beta }{\Vert }_2^2+\lambda \Vert \mathit{\beta }{\Vert }_2^2-2{\displaystyle \sum _{k=1}^K}{\eta }_k{\hat{\mathbf{w}}}_k^{\mathrm{\top }}\mathit{\beta },}$$

which consists of $K+1$ tuning parameters, λ and $\{{\eta }_k{\}}_{k=1,\dots ,K}$. However, this might be computationally less feasible when K is large, since optimization involving a large number of tuning parameters will be challenging in terms of computational speed and numerical stability. Intuitively, if we can derive an aggregated source estimator, denoted as $\hat{\mathbf{w}}$, that resides within the convex cone formed by the source models, i.e. $\hat{\mathbf{w}}\in \{\sum _{i=1}^K{\theta }_k{\hat{\mathbf{w}}}_k:{\theta }_k\ge 0\}$, and aligns more closely with the target parameter $\mathit{\beta }$ than any individual ${\hat{\mathbf{w}}}_k$, then we can directly utilize the single source angleTL method detailed in Section 2.

Following this idea, we present in this section two approaches to obtain the aggregated estimator $\hat{\mathbf{w}}$ from the K source estimators $\{{\hat{\mathbf{w}}}_k{\}}_{k=1,\dots ,K}$, where the main difference between them lies in whether an independent validation dataset in addition to the training data is needed for the aggregation step. In the existence of an independent validation dataset, we can use it to learn the weights $\mathit{\theta }=({\theta }_1,\dots ,{\theta }_k)$ that linearly combine $\{{\hat{\mathbf{w}}}_k{\}}_{k=1,\dots ,K}$ to obtain the aggregated estimator $\hat{\mathbf{w}}$. This can be achieved via methods such as the Q-aggregation, summarized in Algorithm 1, which is shown to approximate the best linear combination of all models when the validation data is sufficiently large (Lecué & Rigollet, 2014; Rigollet & Tsybakov, 2011; Tsybakov, 2014). If we have evidence where some of the source models can be highly different from the target, we can further impose sparse structures on $\mathit{\theta }$.

Algorithm 1.

Obtain $\hat{\mathbf{w}}$ from a validation dataset

Data: Source estimates $\{{\hat{\mathbf{w}}}_k{\}}_{k=1,\dots ,K}$ and a validation dataset $\{\stackrel{\circ }{\mathbf{X}},\stackrel{\circ }{\mathbf{Y}}\}$

Obtain the weights $\hat{\mathit{\theta }}=({\hat{\theta }}_1,\dots ,{\hat{\theta }}_K{)}^{\mathrm{\top }}$ by:

$\hat{\mathit{\theta }}={\arg min}_{\theta }\Vert \stackrel{\circ }{\mathbf{Y}}-\stackrel{\circ }{\mathbf{X}}(\sum _{k=1}^K{\theta }_k{\hat{\mathbf{w}}}_k){\Vert }_2^2+{\lambda }_{\theta }\mathcal{P}\mathcal{(}\theta \mathcal{)}$ , subject to $\mathit{\theta }\ge 0$, where ${\lambda }_{\theta }\mathcal{P}(\theta )$ can be added to enforce sparsity or other desirable structures.

Obtain $\hat{\mathbf{w}}=\sum _{k=1}^K{\hat{\mathbf{\theta }}}_k{\hat{\mathbf{w}}}_k$.

Result:  $\hat{\mathbf{w}}$

Alternatively, one could consider using the following spectral approach, summarized in Algorithm 2, to obtain an aggregated $\hat{\mathbf{w}}$ without a validation dataset. Specifically, we first normalize each ${\hat{\mathbf{w}}}_k$ to obtain ${\overline{\mathbf{w}}}_k=\frac{{\hat{\mathbf{w}}}_k}{\Vert {\hat{\mathbf{w}}}_k{\Vert }_2}$. Let ${\mathbf{u}}_1$ be the first eigenvector of $[{\overline{\mathbf{w}}}_1^{\mathrm{\top }},{\overline{\mathbf{w}}}_2^{\mathrm{\top }},\dots ,{\overline{\mathbf{w}}}_K^{\mathrm{\top }}{]}^{\mathrm{\top }}[{\overline{\mathbf{w}}}_1^{\mathrm{\top }},{\overline{\mathbf{w}}}_2^{\mathrm{\top }},\dots ,{\overline{\mathbf{w}}}_K^{\mathrm{\top }}]\in {\mathbb{R}}^{K\times K}$, and $|{\mathbf{u}}_1|$ is the absolute value of ${\mathbf{u}}_1$. We propose to use $|{\mathbf{u}}_1|$ as the weights for aggregating the K normalized source estimates, where the kth entry of $|{\mathbf{u}}_1|$ will be the weight for the kth source estimator. Then the final adaptive weighted source estimator is defined as $\hat{\mathbf{w}}=\sum _{k=1}^K|{\mathbf{u}}_1{|}_k{\overline{\mathbf{w}}}_k$. Intuitively, this approach can be considered as carrying out a principal component (PC) analysis on the matrix $[{\overline{\mathbf{w}}}_1^{\mathrm{\top }},{\overline{\mathbf{w}}}_2^{\mathrm{\top }},\dots ,{\overline{\mathbf{w}}}_K^{\mathrm{\top }}{]}^{\mathrm{\top }}$ that combines all the source estimators, where ${\mathbf{u}}_1$ is its first PC loadings, under which the linear combination ${\mathbf{u}}_1^{\mathrm{\top }}[{\overline{\mathbf{w}}}_1^{\mathrm{\top }},{\overline{\mathbf{w}}}_2^{\mathrm{\top }},\dots ,{\overline{\mathbf{w}}}_K^{\mathrm{\top }}{]}^{\mathrm{\top }}$ of ${\hat{\mathbf{w}}}_k$’s has the largest variance, or summarizes the most information in $\{{\hat{\mathbf{w}}}_k{\}}_{k=1,\dots ,K}$. In particular, under suitable conditions (see Theorem 2) on the overall quality of the source estimators, the first PC loadings are all nonnegative (in this case $|{\mathbf{u}}_1{|}_k={\mathbf{u}}_{1k}$), so that the first PC, $\sum _{k=1}^K{\mathbf{u}}_{1k}{\overline{\mathbf{w}}}_k$, is exactly the final aggregated source estimator $\hat{\mathbf{w}}$. Moreover, whenever $\{{\hat{\mathbf{w}}}_k{\}}_{k=1,\dots ,K}$ together contain a sufficient amount of information about the direction of $\mathit{\beta }$, it can also be shown that the components of $|{\mathbf{u}}_1|$ reflect the true discrepancy between each ${\hat{\mathbf{w}}}_k$ and $\mathit{\beta }$, and that the final aggregated source estimator is asymptotically no worse than the best candidate source estimator (see Theorem 2). A similar spectral weighting idea has been considered for combining multiple classifiers without labelled data (Parisi et al., 2014). Such a strategy avoids splitting extra samples from already limited target data for aggregation so that it does not affect the prediction performance.

Algorithm 2.

Obtain $\hat{\mathbf{w}}$ without a validation dataset

Data: Source estimates $\{{\hat{\mathbf{w}}}_k{\}}_{k=1,\dots ,K}$

for  $k=1,\dots ,K$  do

Normalize ${\hat{\mathbf{w}}}_k$ and obtain ${\overline{\mathbf{w}}}_k=\frac{{\hat{\mathbf{w}}}_k}{\Vert {\hat{\mathbf{w}}}_k{\Vert }_2}$.

end

Let ${\mathbf{u}}_1$ be the first eigenvector of $[{\overline{\mathbf{w}}}_1^{\mathrm{\top }},{\overline{\mathbf{w}}}_2^{\mathrm{\top }},\dots ,{\overline{\mathbf{w}}}_K^{\mathrm{\top }}]\mathrm{\top }[{\overline{\mathbf{w}}}_1^{\mathrm{\top }},{\overline{\mathbf{w}}}_2^{\mathrm{\top }},\dots ,{\overline{\mathbf{w}}}_K^{\mathrm{\top }}]\in {\mathbb{R}}^{K\times K}$.

Obtain the aggregated source estimates $\hat{\mathbf{w}}=\sum _{k=1}^K|{\mathbf{u}}_1{|}_k\frac{{\hat{\mathbf{w}}}_k}{\Vert {\hat{\mathbf{w}}}_k{\Vert }_2}$, where $|.{|}_k$ is the k-th entry of the absolute value.

Result:  $\hat{\mathbf{w}}$

The theoretical guarantee for Algorithm 1 has been carefully studied, e.g. in Tsybakov (2014). Here, we provide theoretical justification for Algorithm 2. We start with some definitions. Similarly as in Section 2, for each $k\in [K]$, we consider ${\hat{\mathbf{w}}}_k={\mathbf{w}}_k+{\mathit{\delta }}_k$, and assume ${\mathbf{w}}_k$ and ${\mathit{\delta }}_k$ satisfy Assumptions 2 and 3. Moreover, we define ${\mathbf{R}}_{\mathbf{w}}=({\rho }_{ij}^{\mathbf{w}})\in {\mathbb{R}}^{K\times K}$, where $\frac{{\rho }_{ij}^{\mathbf{w}}}{p}$ is the largest eigenvalue of

$$\mathbb{E}\left[(\frac{{\mathbf{w}}_i}{\Vert {\mathbf{w}}_i{\Vert }_2}-\frac{\mathit{\beta }}{\Vert \mathit{\beta }{\Vert }_2}){(\frac{{\mathbf{w}}_j}{\Vert {\mathbf{w}}_j{\Vert }_2}-\frac{\mathit{\beta }}{\Vert \mathit{\beta }{\Vert }_2})}^{\mathrm{\top }}\right],1\le i,j\le K,$$ (7)

which characterizes the pairwise covariance between the discrepancies in $\frac{{\mathbf{w}}_i}{\Vert {\mathbf{w}}_i{\Vert }_2}$ and $\frac{{\mathbf{w}}_j}{\Vert {\mathbf{w}}_j{\Vert }_2}$ with respect to $\frac{\mathit{\beta }}{\Vert \mathit{\beta }{\Vert }_2}$, and define ${\mathbf{R}}_{\mathit{\delta }}=({\rho }_{ij}^{\mathit{\delta }})\in {\mathbb{R}}^{K\times K}$ where $\frac{{\rho }_{ij}^{\mathit{\delta }}}{p}$ is the largest eigenvalue of

$$\mathbb{E}\left[\frac{{\mathit{\delta }}_i{\mathit{\delta }}_j^{\mathrm{\top }}}{\Vert {\mathbf{w}}_i{\Vert }_2\Vert {\mathbf{w}}_j{\Vert }_2}\right],1\le i,j\le K$$ (8)

which characterizes the pairwise covariance between the normalized source estimation errors $\frac{{\mathit{\delta }}_i}{\Vert {\mathbf{w}}_i{\Vert }_2}$ and $\frac{{\mathit{\delta }}_j}{\Vert {\mathbf{w}}_j{\Vert }_2}$. In a special case where the source datasets are mutually independent, we have ${\rho }_{ij}^{\mathit{\delta }}=0$ for $i\ne j$, and ${\mathbf{R}}_{\mathit{\delta }}$ is a diagonal matrix. Finally, we define the true similarity between ${\hat{\mathbf{w}}}_i$’s and $\mathit{\beta }$ as $\mathbf{s}=(\cos \mathrm{\angle }({\hat{\mathbf{w}}}_1,\mathit{\beta }),\dots ,\cos \mathrm{\angle }({\hat{\mathbf{w}}}_K,\mathit{\beta }){)}^{\mathrm{\top }}.$ With the above preparation, we state our main results concerning Algorithm 2.

Theorem 2

Suppose for each $k\in \{1,\dots ,K\}$, ${\mathbf{w}}_k$ and ${\mathit{\delta }}_k$ satisfy Assumptions 2 and 3 with positive ρ being bounded away from 0, and that $\Vert {\mathit{\delta }}_k{\Vert }_2^2=\mathbb{E}\Vert {\mathit{\delta }}_k{\Vert }_2^2\cdot (1+o(1))$ with high probability.¹ Then, for any small constant $ϵ>0$, there exists some sufficiently small constant $C_1(ϵ)>0$ such that whenever $max\{\Vert {\mathbf{R}}_{\mathit{\delta }}\Vert ,\Vert {\mathbf{R}}_{\mathbf{w}}\Vert \}/K\le C_1(ϵ)$, the following holds:

1. For the spectral weights $\hat{\mathbf{s}}:=|{\mathbf{u}}_1|$ and the consensus source estimator $\hat{\mathbf{w}}$ defined in Algorithm 2, it holds that $\cos \mathrm{\angle }(\hat{\mathbf{s}},\mathbf{s})=\frac{{\hat{\mathbf{s}}}^{\mathrm{\top }}\mathbf{s}}{\Vert \hat{\mathbf{s}}{\Vert }_2\Vert \mathbf{s}{\Vert }_2}\ge 1-ϵ$ and $\cos \mathrm{\angle }(\hat{\mathbf{w}},\mathit{\beta })>\rho -ϵ$ in probability as $n\to \mathrm{\infty }$.

2. There exists some constant $C_2(ϵ)>0$ such that, whenever $\frac{1}{p}\text{tr}({\mathbf{\Sigma }}_{\delta })\ge C_2(ϵ){\alpha }_s^2$, we have $\cos \mathrm{\angle }(\hat{\mathbf{w}},\mathit{\beta })>\underset{1\le k\le K}{max}\cos \mathrm{\angle }({\hat{\mathbf{w}}}_k,\mathit{\beta })+ϵ$ in probability as $n\to \mathrm{\infty }$.

Theorem 2 yields the effectiveness of the spectral weighting approach in three aspects. Firstly, it ensures that the estimated weight vector $\hat{\mathbf{s}}$ will converge to the true similarity measure $\mathbf{s}=(\cos \mathrm{\angle }({\hat{\mathbf{w}}}_1,\mathit{\beta }),\dots ,\cos \mathrm{\angle }({\hat{\mathbf{w}}}_K,\mathit{\beta }){)}^{\mathrm{\top }}$ as $n\to \mathrm{\infty }$. As a result, more weights will be given to the source estimators closer to $\mathit{\beta }$ and less weights to those with smaller $\cos \mathrm{\angle }({\hat{\mathbf{w}}}_k,\mathit{\beta })$. Secondly, it indicates the guaranteed performance of the final consensus source estimator $\hat{\mathbf{w}}$, as long as the original correlation ρ between the source and target regression coefficients is sufficiently strong. Thirdly, and interestingly, part two of Theorem 2 shows that, whenever the source estimators have weak signal-to-noise ratio in the sense that $\frac{\mathbb{E}\Vert {\mathbf{w}}_k{\Vert }_2^2}{\mathbb{E}\Vert {\mathit{\delta }}_k{\Vert }_2^2}=\frac{p{\alpha }_s^2}{\text{tr}({\mathbf{\Sigma }}_{\delta })}$ is small, the consensus estimator has strictly better performance than all the source estimators in the large sample limit, demonstrating the consensus power of the spectral weighted estimator.

The validity of the method relies on the additional condition that $max\{\Vert {\mathbf{R}}_{\mathit{\delta }}\Vert ,\Vert {\mathbf{R}}_{\mathbf{w}}\Vert \}$ is small compared with K, which essentially requires the source estimators to be sufficiently diverse and less correlated in terms of both the respective true coefficients ${\mathbf{w}}_k$, and the estimation errors ${\mathit{\delta }}_k$. Specifically, from our proof of Theorem 2 in Appendix B of online the supplementary material, it can be seen as an ‘approximate rank-one’ assumption for the matrix of normalized source estimators $\{{\overline{\mathbf{w}}}_k{\}}_{k=1}^K$. As such, the validity of Theorem 2 would effectively rely on the ‘spectral gap’ between the singular values of the matrix $[{\overline{\mathbf{w}}}_1^{\mathrm{\top }},{\overline{\mathbf{w}}}_2^{\mathrm{\top }},\dots ,{\overline{\mathbf{w}}}_K^{\mathrm{\top }}{]}^{\mathrm{\top }}$, which can be accessed empirically. In other words, when the ratio between the first two singular values is significantly larger than those between any other consecutive singular values, the assumptions behind Algorithm 2 and Theorem 2 are likely satisfied. In this connection, several data-driven methods such as ScreeNot (Donoho et al., 2023) and BEMA (Ke et al., 2023) can be adopted to help assess the appropriateness of this ‘approximate rank-one’ assumption. Second, due to the robustness of our transfer learning estimator against negative transfer, even when multiple sources are inappropriately aggregated for some reason, the final estimator would not be driven astray. Third, in addition to our spectral approach, one can alternatively use the supervised proposed in Algorithm 1, when the validity of Algorithm 2 is unclear. In Section 6, we show the effectiveness and superior performance of such spectral weighting approach in combining multiple source estimators learned from different genetic risk models.

5. Simulation study

5.1. Comparing empirical and theoretical predictive risks

We perform a simulation study to verify the theoretical predictive risks shown in Theorem 1. For $j\in \{1,\dots ,p\}$, we generate

$$({\mathbf{w}}_j,{\mathit{\beta }}_j{)}^{\mathrm{\top }}\stackrel{i.i.d}{\sim }N(0,\frac{1}{p}\left(\begin{array}{cc}{\alpha }_s^2& \rho {\alpha }_s{\alpha }_t\\ \rho {\alpha }_s{\alpha }_t& {\alpha }_t^2\end{array}\right)).$$ (9)

We set $n=50$ and $p=\gamma n$, and generate $\mathbf{X}$ from independent standard normal distributions. We then generate $\mathbf{Y}\sim N(\mathbf{X}\mathit{\beta },0.5{\mathbf{I}}_n)$ given $\mathbf{X}$. We vary the correlation between $\mathit{\beta }$ and $\mathbf{w}$, i.e. $\rho \in \{0.3,0.6,0.9\}$, and the signal strength ratio $\frac{{\alpha }_t}{{\alpha }_s}\in \{\frac{10}{9},2\}$, as well as the dimension-to-sample ratio $\gamma =\frac{p}{n}\in \{2,\frac{1}{2}\}$.

We first consider the noiseless case where the source estimator $\hat{\mathbf{w}}=\mathbf{w}$. In the noiseless case, we have a precise prediction risk for angleTL as shown in Corollary 1. For a given λ chosen from a grid, we set $\eta =\rho \frac{{\alpha }_t}{{\alpha }_s}$, which is the optimal choice of η, and calculate the theoretical risk. We also obtain the theoretical risk of distTL by setting $C_L=C_U=C$ and $\eta =\lambda$ in Theorem 1, and obtain the optimal $\lambda =\gamma \frac{{\sigma }^2}{{\alpha }_t^2+{\alpha }_s^2-2\rho \sqrt{{\alpha }_t{\alpha }_s}}$. For the target-only approach, we obtain the theoretical risk by setting $C_L=C_U=C$ and $\eta =0$ in Theorem 1, and obtain the optimal $\lambda =\gamma \frac{{\sigma }^2}{{\alpha }_t^2}$. To obtain the empirical predictive risks, we apply each method on the training data for each λ in the grid, and then obtain the corresponding MSE evaluated on an independent testing dataset of size 100.

Figure 3 compares the empirical risks and theoretical risks under the noiseless case. For all methods, the empirical risk aligns perfectly with the empirical risks, demonstrating the theoretical risk we obtained in Corollary 1 is precise in the noiseless case. As expected, angleTL is no worse than the target-only estimator and distTL across all settings. Moreover, the optimal λ in theory (vertical line) precisely falls on the lowest point of the curve in all settings. Note that in the bottom right panel, we have $\rho \frac{{\alpha }_t}{{\alpha }_s}=1$ which represents the orthogonal case shown in Figure 1c. We see that distTL reaches the same performance as angleTL. However, distTL can be worse than the target-only approach when $\rho \frac{{\alpha }_t}{{\alpha }_s}$ is far from 1, e.g. the first column, indicating that it cannot prevent negative transfer.

Figure 3.

Empirical prediction error over 500 simulation replicates (solid curve) versus theoretical prediction error given in Corollary 1 (dashed curve). The vertical line shows the optimal $\sqrt{\lambda }$ in theory that leads to the minimum prediction error. We generate target and source estimates, $\mathit{\beta }$ and $\mathbf{w}$, through multivariate Gaussian distribution, by varying their correlation $\rho \in \{0.3,0.6,0.9\}$ (in columns), signal strength ratio $\frac{{\alpha }_t}{{\alpha }_s}\in \{\frac{10}{9},2\}$ and dimension-to-sample ratio $\gamma =\frac{p}{n}\in \{2,\frac{1}{2}\}$ (in rows). We sequentially select 100 λ values, and for each λ we train on 50 samples and test on 100 samples. We report the average test error over 500 simulations.

Similarly, we perform a simulation study to verify the predictive risks when there are estimation errors $\mathit{\delta }$. We generate $\mathit{\delta }\sim N(\mathbf{0},{\mathbf{\Sigma }}_{\delta })$, where the covariance matrix ${\mathbf{\Sigma }}_{\delta }$ is generated by setting an exchangeable correlation of 0.1 and the variances are uniformly generated between 0 and 0.05. We compute $C_L$ and $C_U$ using ${\mathbf{\Sigma }}_{\delta }$ and obtain the theoretical upper and lower bounds for the predictive risk of angleTL. Figure 4 shows the empirical predictive risk of the target-only estimator, distTL, and angleTL. The shaded area represents the theoretical lower and upper bounds of the predictive risk of angleTL obtained in Theorem 1. We can see that the empirical risk of angleTL falls between the theoretical risk bounds. The width of the risk bounds increases with γ and ρ. Across all scenarios, the upper bounds are lower than the empirical risk of the target-only approach, which shows that angleTL can protect against negative transfer.

Figure 4.

Empirical prediction error over 500 simulation replicates (solid curve) and theoretical prediction error bound of the proposed method following Theorem 1 (filled area). We generate target and source estimates, $\mathit{\beta }$ and $\mathbf{w}$, through multivariate normal distribution, by varying their correlation $\rho \in \{0.3,0.6,0.9\}$ (in columns), signal strength ratio $\frac{{\alpha }_t}{{\alpha }_s}\in \{\frac{10}{9},2\}$, and dimension-to-sample ratio $\gamma =\frac{p}{n}\in \{2,\frac{1}{2}\}$ (in rows). We add randomly generated and uniformly distributed noise $\mathit{\delta }$ to $\mathbf{w}$. We sequentially take 100 λ values, and for each λ we train on 50 samples and test on 100 samples. We report the average test error over 500 simulations.

5.2. Evaluate the predictive performance across different settings

5.2.1. Single source study

We first evaluate the empirical predictive performance of angleTL in the case with one source study. To mimic the practical situation, we generate data for both the source and target populations based on the corresponding model parameters $\mathbf{w}$ and $\mathit{\beta }$ where the corresponding entries are generated from equation (9). We set the sample sizes to be $N=5,000$ for the source population, and $n=50$ for the target population. The dimension of the model is set to $p\in \{25,50,100\}$. For the source population, we generate the predictors $\tilde{\mathbf{X}}\sim N(\mathbf{0},\stackrel{\mathbf{~}}{\mathbf{\Sigma }})$, where the covariance matrix $\stackrel{\mathbf{~}}{\mathbf{\Sigma }}$ is set to have variances of 1 and exchangeable correlation of 0.2, and we generate the outcome variable $\tilde{\mathbf{Y}}\sim N(\tilde{\mathbf{X}}\mathbf{w},0.5{\mathbf{I}}_N)$. For the target population, we generate $\mathbf{X}\sim N(\mathbf{0},\mathbf{\Sigma })$, where $\mathbf{\Sigma }$ contains variances of 1 and exchangeable correlation of 0.1, and we generate the outcome variable $\mathbf{Y}\sim N(\mathbf{X}\mathit{\beta },0.5{\mathbf{I}}_n)$.

We apply an ordinary least-square approach on the generated source data to obtain $\hat{\mathbf{w}}$, which is then used together with the target data. We compare the predictive performance of angleTL to three state-of-art methods: (i) target-only: target-only estimator through equation (1); (ii) source-only: directly apply the source estimates on the testing data; (iii) distTL: distance-based transfer learning approach via equation (2). The predictive performance is evaluated by the root mean squared error (RMSE) calculated from an independent testing data set of size 200, following the same data generating mechanism as the target data. The code for fitting target-only estimator, distTL, and angleTL is the same with different choices of η and λ selected by a three-fold cross-validation, which ensures comparability. Specifically, when running target-only ridge regression, we set $\eta =0$ and only search λ from a grid containing 100 points randomly selected between 0.0001 and 0.5. When running distTL, we set $\eta =\lambda$ and search λ from a grid containing 100 points randomly selected between 0.0001 and 0.5. When running angleTL, η and λ are chosen from a two-dimensional grid, with λ ranges between 0.0001 and 0.5 while η ranges between 0.0001 and 0.2.

In Figure 5, each panel of RMSE is summarized from 200 independent simulations, where a lower value represents better prediction accuracy. The pattern aligns with what we see in Figure 3, where the performance of angleTL improves over the increasing correlation between the target and the source estimates increases. When $\frac{{\alpha }_t}{{\alpha }_s}=2$, we see that distTL overlaps with angleTL around $\rho =\frac{1}{2}$ where $\rho \frac{{\alpha }_t}{{\alpha }_s}$ is close to 1. When $\frac{{\alpha }_t}{{\alpha }_s}=\frac{1}{2}$, for all $\rho \in (0,1)$, since $\rho \frac{{\alpha }_t}{{\alpha }_s}<\frac{1}{2}$, distTL shows consistently higher RMSE than angleTL; it also underperforms the target-only estimator, indicating that distTL fails to capture the similarity between the target and the source estimates and fails to prevent negative transfer. For a given $\gamma =\frac{p}{n}$, the performance of angleTL is stable across different values of $\frac{{\alpha }_t}{{\alpha }_s}$. For a given $\frac{{\alpha }_t}{{\alpha }_s}$, the source model is helpful when the dimension of the model is higher.

Figure 5.

Root mean squared error (RMSE) of predicted outcome over 200 simulations. In each panel, we vary the correlation ρ between the target and source estimates from 0.3 to 0.95. We vary $\gamma =\frac{p}{n}\in \{\frac{1}{2},1,2\}$ (in rows) and the signal strength ratio $\frac{{\alpha }_t}{{\alpha }_s}\in \{\frac{1}{2},1,2\}$ (in columns).

5.2.2. Multiple source studies

We also consider the situation where we have multiple source studies as introduced in Section 4. The target sample size is set to $n=100$ and each of the five source populations is of size $N=5,000$. We compare the results of the proposed multisource estimators, angleTL-multi1 and angleTL-multi2, with the target-only estimator and the proposed single-source angleTL applying on the best single-source estimates. In angleTL1, we reserve 30% of the target data (30 samples) for aggregation.

Figure 6a shows the case where all source estimates have similar correlations as the target estimates between 0.4 and 0.6, and Figure 6b shows e of the sources are much more helpful than others, with correlations ranging from 0.1 to 0.9. In the former case, the proposed angleTL-multi2 outperforms angleTL which only transfers from the best single source model (${\hat{\mathbf{w}}}_5$ with ${\rho }_5=0.6$) and angleTL-multi1 with supervised aggregation. The latter case shows a similar pattern, where angleTL with the best single source ( ${\hat{\mathbf{w}}}_5$ with ${\rho }_5=0.9$) and angleTL-multi1 show improved performance than target-only and source-only models, slightly worse than angleTL-multi2. In Figure 6c, all five source vectors were very different from the target with $\rho =0.1$. AngleTL shows similar performances as the target-only estimator, without borrowing much information from source models with low similarity to the target. However, this demonstrates the robust performance of angleTL in terms of preventing negative transfer from irrelevant source models. When dealing with a large number of source models, we conduct a simulation study with $K=50$, with ρ ranging from 0.1 to 0.9 (Figure 6d). Specifically, we randomly assign ρ from Uniform $(0.1,0.9)$ for 50 sources. We observe that angleTL-multi1 and angleTL-multi2 perform nearly the same and are better than target and single-source angleTL.

Figure 6.

Root mean squared error (RMSE) of predicted outcome over 1,000 simulations comparing target-only estimates (target-only), proposed angleTL when transferring from the best single model with the smallest ρ (angleTL), and proposed angleTL combined with Algorithm 1 (angleTL-multi1) and proposed angleTL combined with Algorithm 2 (angleTL-multi2). Panels (a), (b), and (c) correspond to configurations with correlation $\rho =({\rho }_1,{\rho }_2,{\rho }_3,{\rho }_4,{\rho }_5)$ being $(0.4,0.45,0.5,0.55,0.6)$  $(0.1,0.3,0.5,0.7,0.9)$, and $(0.1,0.1,0.1,0.1,0.1)$, respectively. Panel (d) corresponds to the setting where $K=50$ and correlations range from 0.1 to 0.9.

6. Application to predict low-density lipoprotein

We apply angleTL, angleTL-multi1, and angleTL-multi2 to predict low-density lipoprotein (LDL) cholesterol, a blood biomarker that plays a crucial role in determining cardiovascular disease (Kwiterovich Jr, 2000), in self-reported White population using data from Mass General Brigham Biobank (MGBB) (Karlson et al., 2016). MGBB is a research database launched in 2021, containing around eighty thousands DNA samples before data quality control, yet still a relatively limited data size compared to other population-based biobanks such as the United Kingdom Biobank (UKB) (Sudlow et al., 2015). In this application, our goal is to build a prediction model for LDL using basic demographic and clinical risk factors, combined with top single nucleotide polymorphisms (SNPs), where our target population is the White population at MGBB. We consider a total of eight source models, including three models trained at UKB (LDL, Apolipoprotein B-100 [ApoB], and triglycerides [Tri]) and five models for LDL trained at the five participating sites at the electronic MEdical Records and GEnomics (eMERGE) Network (Gottesman et al., 2013).

In the MGBB database, we extracted data from 5,600 self-reported White participants. We extract up to the latest five LDL measures and use the average LDL as the outcome. The predictors include demographics such as age and gender, a binary indicator flagging the use of antihyperlipidemic medications, as well as related genetic variants selected from a clumping and thresholding procedure with linkage disequilibrium and p-values of a genome-wide associate study (GWAS) using all the training samples from MGBB data (Choi et al., 2020). A total of 5,306 SNPs are selected with $R^2$ threshold set to 0.6 and p-value threshold set to 0.0005 using PLINK 2.0 (Chang et al., 2015).

We use UKB as one source dataset, from which we identify 409,031 samples with at least one measure of Tri, 408,602 samples with at least one measure of LDL, and 407,366 samples with at least one measure of Apo-B. To reduce the potentially inaccurate reporting ancestry, we selected those whose self-reported ancestry information agrees with the computed PC-based ancestry prediction (D. Zhang et al., 2020).

Among the five source datasets in eMERGE, we identified a total of 16,723 samples with at least one LDL measure. We extract the same set of genetic, medication, and demographic variables from UKB and eMERGE, and fit eight source models using linear regression. A summary of basic information about the target and eight source datasets can be found in Table 1. Additional details regarding the data processing procedures can be found in Section E of the online supplementary material.

Table 1.

Basic information of the target and eight source datasets

	Biobank		Outcome	Sample size	Mean age (SD)	Male (%)
Target	Mass General Brigham Biobank		low-density lipoprotein (LDL)	5,600	64.7 (15.3)	2,959 (52.8)
Source	United Kingdom Biobank		LDL	408,602	69.8 (8.0)	187,607 (45.9)
			Tri	409,031	69.8 (8.0)	187,821 (45.9)
			Apo-B	407,366	69.8 (8.0)	186,763 (45.8)
	eMERGE	Marshfield Clinic (Marsh)	LDL	4,551	70.8 (9.7)	1,496 (39.2)
		University of Washington (UW)	LDL	618	80.9 (0.5)	295 (54.0)
		Mayo Clinic (Mayo)	LDL	3,185	71.3 (10.1)	1,512 (56.7)
		Northwestern University (NWU)	LDL	3,037	53.4 (15.2)	287 (20.0)
		Mount Sinai Hospital (MtSinai)	LDL	5,332	60.4 (13.1)	363 (69.1)

We compare the performance of the target-only estimator, the source-only estimator, distTL, and angleTL. We randomly split the target MGBB data into training and testing with a ratio of 10:1. For angleTL-multi1, which needs validation data for aggregation, we further split 10% from the training data. All methods are trained on the training data, and the performance is measured by the $R^2$ (variance explained by the model) evaluated on the testing data. This process is repeated 100 times to account for the sampling variability. We employed ridge regression to calculate the target-only estimator and the majority of the source estimators, with the exception of those calculated from the UKB. Given that the total sample size in UKB exceeds 400,000, we opted for an ordinary least-square linear regression due to computational efficiency.

In Figure 7, we present the relative performance of each method compared to the benchmark target-only model. With 100 replications, we report the mean, the first and the third quantiles of the percentage improvement of $R^2$. Among all the source-only models shown in the top section (dark blue bars with grey background), the model trained at UKB using triglycerides as the outcome has the worst performance, and the model trained at Marshfield Clinic has the highest performance, where the average improvement is around 64%. When transferring each source model to train the target model using distTL shown in the middle section (light blue bars with light blue background), all the single-source distTL have similar performance, around 6% improvement. When applying single-source angleTL shown in the bottom section (red bars with red background), we observe that the performance is uniformly better than the corresponding source-only models, and angleTL also outperforms distTL. In the last two rows, we see that angleTL-multi1 has comparable performance as the best single source, LDL-Marsh, while angleTL-multi2 has the largest average improvement of 109.3%. To assess the generalizability of the conclusion when the target population changes, we treat each of the sites in eMERGE as the target population, and obtain similar results where angleTL achieves the best performance in nearly all cases. More details can be found in Section 5.2 in Appendix E of the online supplementary material.

Figure 7.

Mean (1st–3rd quantile) percentage change of $R^2$ compared to the target-only model. The dark blue bars with grey background contain source-only estimates from UKB and eMERGE; the light blue bars with light blue background contain distTL; the red bars with red background include the proposed single-source angleTL; and the last two yellow rows show angleTL-multi1 and angleTL-multi2, respectively. Tri-UKB, triglycerides from UKB; ApoB, Apolipoprotein B-100; LDL, low-density lipoprotein; MtSinai, Mount Sinai Hospital; UW, University of Washington; NWU, Northwestern University; Mayo, Mayo Clinic; Marsh, Marshfield Clinic.

In this real application involving real data, the source models are derived using information from diverse study cohorts with varied outcomes. This results in heterogeneity on two levels: the cohort level and the outcome level, when comparing each source model to the target model. For instance, UKB is a population-based biobank, representing a comparatively healthier population than those from hospital-based cohorts collected at the eMERGE sites. Furthermore, the outcomes under consideration extend beyond LDL to include correlated traits such as triglycerides and Apolipoprotein B-100. These traits might possess overlapping genetic architectures with LDL. However, the influence of each genetic variant on the traits may differ. Such heterogeneity requires more adaptive adjustment methods than the $L_2$ constraint offered by distTL. This might explain why angleTL outperforms distTL in this particular application.

7. Discussion

We propose angleTL, a flexible transfer learning framework that leverages the concordance of model parameters across populations. Compared to several benchmark methods, including the target-only model, the source-only models, and the distance-based transfer learning method, angleTL is shown to have improved performance both empirically and numerically. Under the setting with a single source model, we identify factors that influence the predictive performance of angleTL, including (1) the similarity between model parameters measured by the $\sin \mathrm{\Theta }$ distance, (2) the signal strengths of the source and the target model, (3) the ratio between model dimensionality and the target sample size, (4) the estimation error of the source estimator, and (5) the level of the residual noise, which provide useful practical guidance for understanding when a source population can be helpful. With multiple source models, we propose to aggregate the source models first before applying angleTL, which we also provide theoretical justifications of the aggregation algorithms. Like ridge regression, angleTL is easy to implement as it enjoys the advantage of having a closed-form solution. It does not require individual-level data from the source, and can prevent negative transfer. Our simulation and real data application demonstrate the validity and feasibility of angleTL across a wide range of settings.

In this paper, we consider the case where only the point estimates of the source models are available. This is more from a practical standpoint that the variance estimates of the pretrained models are rarely reported. For example, for polygenic risk prediction, over 4,745 models are available in the PGS Catalog (https://www.pgscatalog.org/) (Lambert et al., 2021), but none of them reported variances. However, statistically, it is an interesting future direction to consider whether information about the variances of pretrained model parameters can be helpful in transfer learning. With only a single source model, the variance of each entry of $\hat{\mathbf{w}}$ can certainly provide information about which one is more reliable. We may consider applying angleTL only on the sub-vector with relatively small variances, and those with larger variances can be ignored since they are more likely to be misleading. Another possibility is to use resampling techniques, which are shown to be useful in nonregular inference problems. When multiple source models are available, each ${\hat{\mathbf{w}}}_k$ is estimating its own population parameter ${\mathbf{w}}_k\in {\mathbb{R}}^p$. When a source ${\mathbf{w}}_k$ is totally irrelevant to the target, having a smaller estimation error is not helpful for learning target parameter $\mathit{\beta }$. The two algorithms we proposed in this work only use point estimation. With additional information about the variances, we might consider using a two-step weighting to further adjust the weights we learned using point estimates according to the variances or $\Vert \mathrm{cov}({\mathbf{w}}_k)\Vert$.

One limitation of angleTL is that it requires the source and the target models to include the same covariates, whereas, in practice, source models might include covariates that are not measured at the target data or vice versa. There are existing methods focusing on incorporating source models that use a subset of covariates in the target model. For example, Chatterjee et al. (2016) proposed a constrained maximized likelihood approach where the external data follows the same distribution as the internal data. Accounting for heterogeneous data, methods such as Estes et al. (2018), Kundu et al. (2019), H. Zhang et al. (2020), Gu et al. (2021) and Taylor et al. (2022) are proposed, which all considered low-dimensional models and their ability to handle high-dimensional data requires further investigation. A more detailed discussion on related work, including different empirical approaches, can be found in P. Han (2022). It is an interesting future direction to extend angleTL to the setting where the source study uses a subset of the covariates in the target study. Methods proposed in related work such as Taylor et al. (2022) can be considered to address this, where we can first map the unique variables to a space orthogonal to the shared variables, and then include them in the model while using the angle-based regularization to borrow information.

While we employ a linear model to illustrate the concept of angleTL, it is important to note that the penalty term $\lambda \Vert \mathit{\beta }{\Vert }_2^2-2\eta {\hat{\mathbf{w}}}^{\mathrm{\top }}\mathit{\beta }$ can be seamlessly integrated with the objective functions of various other problems. For instance, in addition to the regression problem discussed in our paper, we can apply this approach to a classification problem. In this scenario, our objective is to classify an outcome variable y based on a covariate vector $\mathbf{x}$, using a classification rule denoted as $f({\mathbf{x}}^{\mathrm{\top }}\mathit{\beta })$. When we aim to learn this classification rule for the target population, we have the flexibility to select an appropriate objective function. Furthermore, we can incorporate source estimators ${\hat{\mathbf{w}}}_k$ derived from different populations using various models, such as logistic regression, support vector machines (Cortes & Vapnik, 1995), and linear or quadratic discrimination analysis (Hastie et al., 2009), into the target model through the angleTL penalty (an additional simulation study to demonstrate the performance of angleTL in a classification problem can be found in Section 4.2 in Appendix D of online supplementary material). While the proposed penalization is motivated by an angle-based similarity measure between the source and target model parameters, in a classification problem, it might be useful to consider an alternative similarity defined by the fitted outcomes. More specifically, we can consider the similarity measure defined as $\sin \mathrm{\Theta }(X\mathit{\beta },X\mathbf{w})$ or a rank-based measure $\sum _{i\ne j}\mathbb{I}(X_i^{\mathrm{\top }}\mathit{\beta }>X_j^{\mathrm{\top }}\mathbf{w},X_i^{\mathrm{\top }}\mathit{\beta }>X_j^{\mathrm{\top }}\mathbf{w})$, which is closely related to the maximum rank correlation problem discussed in A. K. Han (1987), H. Wang (2007), Stephanou and Varughese (2021) and Shin and Todorov (2021).

Funding

This work was supported by National Institutes of Health (R01GM148494, P50MH129699).

Data availability

The UK Biobank data and the MGBB data used in this study are not publicly available. Access to the data can be obtained by qualified researchers through an approved application to the UK Biobank (www.ukbiobank.ac.uk) and via the Mass General Brigham Biobank (https://biobank.massgeneralbrigham.org), respectively. The eMERGE Network Phase II data used in this study can be requested from the NIH dbGaP database (dbGaP Study Accession: phs000888.v1.p1). The code to implement the proposed angleTL method and to replicate the simulation figures in the paper is available on GitHub, https://github.com/biostat-duan-lab/angleTL and https://github.com/biostat-duan-lab/angleTL_code, respectively.

Supplementary material

Supplementary material is available online at Journal of the Royal Statistical Society: Series B.

Author contribution

R.D. conceptualized the method and conducted the theoretical analyses. T.G. developed the method, designed the computational algorithm, and conducted the numerical studies and real-world data analysis. Y.H. assisted with the numerical studies and developed the software package. All authors contributed to the writing of the manuscript.