Transfer Learning via Regularized Random-effects Linear Discriminant Analysis

Abstract

Linear discriminant analysis is a widely used method for classification. However, the high dimensionality of predictors combined with small sample sizes often results in large classification errors. To address this challenge, it is crucial to leverage data from related source models to enhance the classification performance of a target model. This paper proposes a transfer learning approach via regularized random-effects linear discriminant analysis, where the discriminant direction is estimated as a weighted combination of ridge estimates obtained from both the target and source models. Multiple strategies for determining these weights are introduced and evaluated, including one that minimizes the estimation risk of the discriminant vector and another that minimizes the classification error. Utilizing results from random matrix theory, we explicitly derive the asymptotic values of these weights and the associated classification error rates in the high-dimensional setting, where the aspect ratio $\gamma ≔p/n$ as $p,n\to \infty$, with $p$ representing the predictor dimension and $n$ the sample size. Extensive numerical studies, including simulations, the analysis of the proteomics-based cardiovascular disease risk classification and the lipid traits classification problem with genotype data, demonstrate the effectiveness of the proposed approach.

1. Introduction

Large and diverse data sets are ubiquitous in modern applications, including those in genomics and medical decisions. It is of significant interest to integrate different data sets to obtain more accurate parameter estimates or to make a more accurate prediction or classification of an outcome. The success of a supervised statistical learning method relies on the availability of training data. When data is scarce, choosing an appropriately flexible model becomes critical to achieving optimal prediction accuracy. This is a classic illustration of the well-known “bias-variance tradeoff”. When building a prediction model for a target population, many auxiliary source data sets may exist and provide additional information for building the model for the target population. Modern techniques in the field of transfer learning (Pan and Yang, 2009; Weiss et al., 2016) aim to exploit these additional information. Given a target problem to solve, transfer learning (Torrey and Shavlik, 2010) aims at transferring the knowledge from different but related samples or studies to improve the learning performance of the target problem. In biomedical studies, some clinical or biological outcomes are hard to obtain due to ethical or cost issues, in which case transfer learning can be leveraged to boost the prediction and estimation performance by effectively utilizing information from related studies. Other relevant approaches include meta-learning (Peng, 2020; Huisman et al., 2021), domain adaptation (Redko et al., 2020; Sun et al., 2015), and, more recently, continual learning (De Lange et al., 2021).

In the high dimensional setting, Li et al. (2022, 2023) developed transfer learning methods for sparse high dimensional regressions and demonstrated that one can improve predictions of gene expression levels using data across different tissues. Such sparse models work well when the true models are sparse and the sample sizes are large. However, there are settings where sparse model assumption may not be valid. In genetics, estimating polygenetic risk scores (prss) using genome-wide genotype data (Mak et al., 2017; Torkamani et al., 2018) is an active area of research. Such prss can be used in risk stratification, or can be treated as risk factors in population health studies. However, due to very large number of genetic variants but relatively small sample sizes, building a prs model that accurately predicts the prs scores is challenging. Alternatively, ridge regression, which does not require the sparseness assumption, but can handle the linkage disequilibrium among the genetic variants, provides a viable method for prs prediction. Zhang and Li (2023) studied the estimation and prediction of random coefficient ridge regression in the setting of transfer learning, where in addition to observations from the target model, source samples from different but possibly related regression models are available. The informativeness of the source model to the target model can be quantified by the correlation between the regression coefficients.

The method of Zhang and Li (2023) is developed for continuous outcomes. In this paper, we propose a transfer learning framework for regularized linear discriminant analysis (RDA) (Friedman, 1989), which we term TL-RDA. Our model assumes a random classification weights setup, where the means of the covariates between two classes differ by a random quantity, $\delta$, with zero mean and constant variance. The auxiliary and target populations are related through the correlation structure of $\delta$. The TL-RDA framework aims to estimate the Bayes optimal predictor by combining naive RDA estimates from both the auxiliary and target populations through a weighted summation. The weights are designed to minimize the distance between the TL-RDA estimator and the Bayes optimal direction in a high-dimensional setting, where the number of features $p$ grows proportionally to the sample size $n$ in all populations. We derive the explicit asymptotic error rate for TL-RDA and show that it achieves the lowest error rate among all estimators based on weighted summations, including naive RDA using only target population data.

Several other LDA–style transfer learning methods have been proposed, but differ in the way of incorporating the source discriminant information. Yang and Gao (2013) derive the multi-view discriminant vectors through a single generalized-eigenvalue problem that maximizes the inter-view correlation while reducing domain discrepancy. Wang et al. (2020) imposes a group-sparse Bayesian prior on discriminant weights to select and linearly combine the informative source domains. Han et al. (2020); Liu et al. (2023) reformulate the classical LDA to learn explicitly the domain-invariant projections. They also augment Fisher's between-/within-class scatter ratio with a distribution-alignment regularizer that forces the projected source and target means and covariances to match. However, none of these methods has been analyzed in the high-dimensional random effects regime, and none aggregates source information by explicitly minimizing the target-domain prediction risk; which are two aspects that are central to our proposed TL-RDA.

A separate stream of work chooses to fuse or shrink parameters across populations with different objectives. In multi-task learning frameworks, discriminant directions or regression coefficients are encouraged to be close by adding fused penalties (Price and al., 2014; Okazaki and Aoyagi, 2024; Lozano and Swirszcz, 2012). These methods minimize a joint loss over all tasks, seeking a model that performs reasonably well simultaneously across domains. Komárek and Lesaffre (2010) proposes a related random-effects LDA that pools information through hierarchical priors from a Bayesian perspective. In the domain-adaptation literature, hypothesis-weighting techniques combine pre-trained source classifiers by shrinking the weights toward a simplex that balances average source risk (Mansour et al., 2009). Closer in spirit to our work, Gu et al. (2024) moves a high-dimensional ridge estimate toward multiple source coefficients along geodesic paths, with step-sizes chosen to minimize an asymptotic proxy for the target prediction error. Their analysis, however, is confined to linear regression and assumes access only to fitted source coefficients. A complementary direction is to bypass parameter fusion and instead learn latent features whose label–relationship is invariant across environments (Peters et al., 2016; Arjovsky et al., 2019). Such invariance-seeking objectives inevitably trade off target-specific accuracy, while TL-RDA tailors an optimized discriminant vector for the single target population.

The remainder of the paper is organized as follows. We first provide a detailed problem setup and outline the proposed TL-RDA approach in Section 2. We then introduce various types of weights used to integrate auxiliary information, followed by the technical assumptions required for analyzing the estimator in the context of random matrix theory in Section 3. In Section 4, we present the analysis of TL-RDA in the proportional regime, providing explicit asymptotic expressions for the different weighting schemes, along with their corresponding error rates. Section 5 offers interpretations of these weights and guidance on how users can select the most appropriate weights for their applications. In the first five sections, we have assumed all populations share the same population covariance matrix. Section 6 extends the TL-RDA to a heterogeneous population covariance matrix set up. In Section 7, we evaluate the proposed methods on two tasks: (i) cardiovascular-disease classification using proteomics data from the Chronic Renal Insufficiency Cohort (CRIC), and (ii) binary lipid traits prediction using gene-expression data from the Penn Medicine BioBank (PMBB). Section 8 offers a brief discussion.

2. Transfer Learning via Regularized Discriminant Analysis

We consider the setting of two-class LDA in the setting of transfer learning, where we have data observed from both the target model, indexd by $K$, and $K-1$ source models, indexed by $k=1,\cdots ,K-1$. We assume that all models, $k=1,\cdots ,K$, follow the classic two-class Gaussian mixture model. More specifically, for $i=1,\dots ,n_k$ and $k=1,\dots ,K$,

$$y_k\in \left\{-1,+1\right\}\mathbb{P}\left(y_k=\pm 1\right)={\pi }_{\pm 1}{\left(X_k\right)}_i\mid y_k~N\left({\mu }_{y,k},\Sigma \right).$$ (1)

Here ${\left(X_k\right)}_i,i=1,\cdots ,n_k$ is a $p$–dimensional vector, and for ease of notation, we write ${\mathbf{X}}_k={\left({\left(X_k\right)}_1{\left(X_k\right)}_2\dots {\left(X_k\right)}_{n_k}\right)}^{\top }\in {\mathbb{R}}^{n_k\times p}$ as a $n_k\times p$ dimensional matrix. For simplicity of notation, we assume that the mixing proportions ${\pi }_{\pm 1}$ remain the same across populations. In fact, without much loss of generality, we mainly discuss the simpler case ${\pi }_{-1}={\pi }_{+1}=1/2$. The more general case can be managed in a similar manner without significant technical difficulty (Appendix C). Moreover, in most of the paper, we assume the population covariance matrix are the same for all $K$ populations, and are denoted by $\Sigma$. This assumption is relaxed in Section 6 where we allow each population to have their own covariance matrix ${\Sigma }_k$ for $k=1,\dots ,K$.

Under this set up, the Bayes optimal prediction direction for the target population $K$ (Anderson, 1958) is given by,

$$d_{\mathit{\text{Bayes}}}≔{\Sigma }^{-1}{\delta }_K\text{where}{\bar{\mu }}_K=\frac{{\mu }_{+1,K}+{\mu }_{-1,K}}{2}\text{for}{\mu }_{\pm 1,K}={\bar{\mu }}_K\pm {\delta }_K,$$

and ${\delta }_K=\left({\mu }_{+1,K}-{\mu }_{-1,K}\right)/2$. The Bayes prediction for a testing data point $x_0$ from population $K$ is

$${\widehat{y}}_{\mathit{\text{Bayes}}}\left(x_0\right)=\text{sign}\left[d_{\mathit{\text{Bayes}}}^{\top }\left(x_0-{\bar{\mu }}_K\right)\right].$$

The regularized discriminant analysis (RDA) classifier uses an empirical version of the unkown Bayes direction. The RDA classifier for population $k$ is a linear classifier

$${\widehat{y}}_{RDA,k}\left(x_0\right)=\text{sign}\left({\widehat{d}}_k^{\top }{x}_0+{\widehat{b}}_k\right),$$

where we use the plug-in estimates for the population parameters ${\Sigma }_k$ and ${\delta }_k$ as follows:

$${\widehat{d}}_k={\left({\widehat{\Sigma }}_k+{\lambda }_k{\mathbb{I}}_p\right)}^{-1}{\widehat{\delta }}_k$$

$${\widehat{b}}_k=-{\widehat{\delta }}_k^{\top }{\left({\widehat{\Sigma }}_k+{\lambda }_k{\mathbb{I}}_p\right)}^{-1}\left({\widehat{\mu }}_{-1,k}+{\widehat{\mu }}_{+1,k}\right)/2$$

$${\widehat{\mu }}_{\pm 1,k}=\frac{2}{n_k}\sum _{i:{\left(y_k\right)}_i=\pm 1}{\left(X_k\right)}_i{\widehat{\delta }}_k=\frac{{\widehat{\mu }}_{+1,k}-{\widehat{\mu }}_{-1,k}}{2}$$

$${\widehat{\Sigma }}_k=\frac{1}{n_k-2}\sum _{i=1}^{n_k}\left[{\left(X_k\right)}_i-{\widehat{\mu }}_{{\left(y_k\right)}_i,k}\right]{\left[{\left(X_k\right)}_i-{\widehat{\mu }}_{{\left(y_k\right)}_i,k}\right]}^{\top }.$$

Here ${\widehat{\Sigma }}_k$ is the usual sample covariance matrix, while ${\widehat{\mu }}_{\pm 1,k},{\widehat{\delta }}_k$ are simple estimators and the population-level counterpart ${\mu }_{\pm 1,k},{\delta }_k$. The sample covariance matrix is penalized on its diagonal to overcome the overwhelming variances in this estimation when $p$ grows proportionally with $n$. As the penalization parameter ${\lambda }_k$ goes to zero or infinity, this classifier recover the Fisher's discriminant analysis or the naive Bayes method (Bickel and Levina, 2004).

To utilize the underlying relatedness of the RDA problems in $K$ populations, for a vector of weights $\mathbf{w}\in {\mathbb{R}}^K$, we now define a classifier based on a weighted linear combination of the population specific discriminator vectors ${\widehat{d}}_k\text{s}$ as follows:

$$\widehat{d}\left(\mathbf{w}\right)=\sum _{k=1}^K{w}_k{\widehat{d}}_k.$$

Note that taking the co-ordinate basis vectors as weights, i.e., $\mathbf{w}={\mathbf{e}}_k\in {\mathbb{R}}^K$, we end up with the population specific discriminant directions $\widehat{d}\left({\mathbf{e}}_k\right)={\widehat{d}}_k$ for $k=1,\dots ,K$. Now using the weighted combination discriminant vector we define the transfer-learning (TL) classifier

$${\widehat{y}}_{\mathbf{w}}\left(x_0\right)=\text{sign}\left(\widehat{d}(\mathbf{w}{)}^{\top }{x}_0+{\widehat{b}}_K\right)$$ (2)

where $x_0\in {\mathbb{R}}^p$ is a test point in the target population. We call this TL-RDA. We use the regular intercept term ${\widehat{b}}_K$ as all intercept terms are asymptotically zero in the regime considered.

We formulate two criteria to be optimized over $\mathbf{w}$. Let us recall the Bayes classification direction $d_{\mathit{\text{Bayes}}}$. Now for a testing data pair $\left(x_0,y_0\right)$ sampled from population $K$, we consider the two errors:

$${‖d_{\mathit{\text{Bayes}}}-\sum _{k=1}^K{w}_k{\widehat{d}}_k‖}_2^2,$$ (3)

$${\mathbb{E}}_{x_0}{\left[{\left(d_{\mathit{\text{Bayes}}}-\sum _{k=1}^K{w}_k{\widehat{d}}_k\right)}^{\top }{x}_0\right]}^2.$$ (4)

The first criteria compares the transfer-learning classifier (TL-RDA) with the Bayes optimal classifier in terms of the estimation error, while the second criteria compares the two classifiers in terms of their prediction errors. In later sections, we present explicit solutions for optimal $\mathbf{w}$ that optimizes the two criteria above. Though none of (3) or (4) is explicitly related to prediction error rate, we show the optimal weight that minimizes (4) also minimizes the prediction error rate in the target population. We will also show the weights that minimizes these two criteria, are in fact related to one another.

In addition, we assume that the population covariance matrix are the same for all $K$ populations (assumption 1). With this in mind, a natural way to further exploit mutual information across populations is to use a pooled sample covariance for all discriminant directions ${\widehat{d}}_k$. We write ${\widehat{\Sigma }}_P$ to denote the pooled sample covariance matrix

$${\widehat{\Sigma }}_P=\sum _{k=1}^K\sum _{i=1}^{n_k}\left[{\left(X_k\right)}_i-{\widehat{\mu }}_{y_i,k}\right]{\left[{\left(X_k\right)}_i-{\widehat{\mu }}_{y_i,k}\right]}^{\top }/\sum _{k=1}^K\left(n_k-2\right).$$

We then define the pooled classification weights and pooled transfer learning regularized discriminant analysis (TLP-RDA) estimator as

$${\widehat{y}}_{\mathbf{w}}^P\left(x_0\right)=\text{sign}\left({\widehat{d}}^P(\mathbf{w}{)}^{\top }{x}_0+{\widehat{b}}_K\right)$$ (5)

where the direction estimates ${\widehat{d}}_k^P$ are now estimated using the pooled covariance matrix as follows

$${\widehat{d}}_k^P≔{\left({\widehat{\Sigma }}_P+{\lambda }_k{\mathbb{I}}_p\right)}^{-1}{\widehat{\delta }}_k\text{and}{\widehat{d}}^P\left(\mathbf{w}\right)≔\sum _{k=1}^K{w}_k{\widehat{d}}_k^P.$$

Once again using (3) and (4), we can optimize $\mathbf{w}$ in ${\widehat{y}}_{\mathbf{w}}^P\left(x_0\right)$ with respect to the two criteria above as well, and we will discuss how to choose between ${\widehat{y}}_{\mathbf{w}}^P\left(x_0\right)$ and ${\widehat{y}}_{\mathbf{w}}\left(x_0\right)$.

This form of TL estimator (2) has been used in Zhang and Li (2023) and Dobriban and Sheng (2020) for regression methods. Helm et al. (2024) also considers a TL estimator that aggregates auxiliary information by weighted summations of individual discriminant directions. However, their aggregation is less adaptive in the form $\widehat{d}(\mathbf{w}{)}^{*}=a{\widehat{d}}_K+(1-a)\sum _{k=1}^{K-1}{\widehat{d}}_k/(K-1)$ where $a$ is a constant. In addition, the final weight $a$ minimizes only empirical quantities and does not guarantee to minimize population-level criteria such as (3) or (4).

3. Random-effects RDA and the Assumptions on Population Parameters

To find the weights that minimize the estimation or prediction risk defined as 3 and 4, we consider random-effects RDA where the population means are random and are potentially correlated across different models. In addition, we also make the random matrix assumptions on the covariance matrix, which allows us to apply the recent advances in random matrix theory to derive the limiting values of the weights and the corresponding classification errors.

3.1. Classification Weights

A key parameter in the two-class classification problem is the classification weights ${\delta }_k$ that separate the two classes. In this paper, we consider a random classification weights set up formalized below.

Assumption 1 (Random Classification Weights) The following conditions hold for populations $k=1,\cdots ,K$.

1. The class-specific population mean vectors ${\mu }_{-1,k},{\mu }_{+1,k}\in {\mathbb{R}}^p$ are randomly generated as ${\mu }_{\pm 1,k}={\bar{\mu }}_k\pm {\delta }_k$, where each ${\delta }_k$ has i.i.d. coordinates with

   $$\mathbb{E}\left({\left({\delta }_k\right)}_i\right)=0\text{Var}\left({\left({\delta }_k\right)}_i\right)={\alpha }_k^2/p\mathbb{E}\left({\left|{\left({\delta }_k\right)}_i\right|}^{4+\eta }\right)\le \frac{C}{p^{2+\eta }}$$

   for fixed $C,\eta >0$.

2. ${\bar{\mu }}_k$ are either fixed or randomly distributed independent of ${\delta }_k,{\mathbf{X}}_k,y_k$. The second moment of ${\bar{\mu }}_k$ are bounded almost surely such that ${\text{lim} \text{sup}}_{p\to \infty }‖{\bar{\mu }}_k‖/p^{1/2-\xi }\le C$ for some constants $\xi ,C>0$.

The parameter ${\alpha }_k^2$ here plays the role of signal strength. This key assumption asserts that all coordinates of ${\delta }_k$ have the same zero mean and diminishing variance, therefore, all coordinates of an observation play equally important roles on determining the response class. This assumption also used by Dobriban and Wager (2018), is standard in the large $n$, large $p$ regime. In addition, the bounded moment assumptions allow use to circumvent difficulties arise when minimizing criteria (3) and (4) which depends on unknown quantities such as ${\delta }_k,{\Sigma }^{-1}$ by invoking Lemma A.5. The performance of the TL estimators clearly depends on how the $K$ populations are related. We further pose the correlated classification weight assumption below.

Assumption 2 (Correlated Classification Weights) For $k=1,\cdots ,K$ the vectors ${\delta }_k$ are correlated across populations such that $\text{Corr}\left({\delta }_k,{\delta }_{k^{\prime }}\right)={\rho }_{k{k}^{\prime }}{\mathbb{I}}_p,k\ne k^{\prime }$.

For most of the paper, we assume that the parameters ${\alpha }_k^2$ and ${\rho }_{k,k^{\prime }}$ are known constants. We discuss consistent estimators for these parameters in Appendix B.3.

3.2. Random Matrix Assumption and Related Results

The sample covariance matrix is an important part of the transfer learning estimator. We consider the Marchenko-Pastur type sample covariance matrices as in several relevant works (Dobriban and Wager, 2018; Zhao et al., 2023; Zhang and Li, 2023). For a symmetric matrix, we can characterize its spectral distribution by a cumulative distribution function that places equal point mass on its eigenvalues. Our asymptotic analysis is based on the convergences of spectral distributions of sample covariance matrices, for which the following assumptions are required.

Assumption 3 (rmt assumption) For $k=1,\cdots ,K$, the design matrix ${\mathbf{X}}_k\in {\mathbb{R}}^{n_k\times p}$ is generated as

$${\mathbf{X}}_k={\left({\mu }_{{\left(y_k\right)}_1,k}{\mu }_{{\left(y_k\right)}_2,k}\cdots {\mu }_{{\left(y_k\right)}_{n_k},k}\right)}^{\top }+{\mathbf{Z}}_k{\Sigma }^{1/2}$$

for a matrix ${\mathbf{Z}}_k\in {\mathbb{R}}^{n_k\times p}$ with i.i.d. entries coming from an infinite array. The entries ${\left(Z_k\right)}_{ij}$ of ${\mathbf{Z}}_k$ satisfy the moment conditions: $\mathbb{E}\left[{\left(Z_k\right)}_{ij}\right]=0$ and $\mathbb{E}\left[{\left(Z_k\right)}_{ij}^2\right]=1$.

1. The population covariance matrix $\Sigma \in {\mathbb{R}}^{p\times p}$ is deterministic. The observations have unit variance, i.e., ${\Sigma }_{jj}=1$ for $j=1,\cdots ,p$.

2. The eigenvalues of $\Sigma$ are uniformly bounded from above and away from zero with constants independent of the dimension $p$.

3. The sequence of spectral distributions $T≔T_{\Sigma ,p}$ of $\Sigma ≔{\Sigma }_p$ converges weakly to a limiting distribution $H$ supported on $[0,\infty )$, called the population spectral distribution (psd).

For sample covariance $\widehat{\Sigma }≔(\mathbf{X}-\mathbb{E}\mathbf{X}{)}^{\top }(\mathbf{X}-\mathbb{E}\mathbf{X})/n$ with $\mathbf{X}\in {\mathbb{R}}^{n\times p}$ generated following the Assumption 3, its empirical spectral distribution (esd) is the probability measure

$$F^{\widehat{\Sigma }}\left(\omega \right)=\frac{1}{p}\sum _{i=1}^{p}1\left\{{\lambda }_i\left(\widehat{\Sigma }\right)\le \omega \right\},\omega \in \mathbb{R},$$

where ${\lambda }_i\left(\widehat{\Sigma }\right)$ is the $i^{\mathit{\text{th}}}$ eigenvalue of $\widehat{\Sigma }$. The Marchenko-Pastur theorem (Marchenko and Pastur, 1967) claims that $F^{\widehat{\Sigma }}$ converges weakly (in distribution) to a limiting distribution $F_{\gamma }≔F_{\gamma }(H)$ supported on $[0,\infty )$ with probability 1, where $\gamma$ is the aspect ratio defined as the limit of $p/n\to \gamma$ as $p,n\to \infty$.

Furthermore, for any distribution $G$ supported on $[0,\infty )$, we define its Stieltjes transform as

$$m_G\left(z\right)≔{\int }_{l=0}^{\infty }\frac{dG\left(l\right)}{l-z},z\in \mathbb{C}\setminus {\mathbb{R}}^{+}.$$ (6)

The esd of sample covariance matrix is uniquely determined by a fixed-point equation for its Stieltjes transform. The limit of the Stieltjes transform for the ESD is given by:

$$m_{F^{\widehat{\Sigma }}}(z)=\text{tr}\left\{{\left(\widehat{\Sigma }-z{\mathbb{I}}_p\right)}^{-1}/p\right\}{\to }_{a.s.}{m}_{F_{\gamma }}(z).$$ (7)

This can be understood by noting that the Stieltjes transform is a smooth, continuous functional of the esd and the esd converges weakly to $F_{\gamma }$.

We continue to define the Stieltjes transform of the limiting spectral distribution of $\bar{\Sigma }≔(\mathbf{X}-\mathbb{E}\mathbf{X})(\mathbf{X}-\mathbb{E}\mathbf{X}{)}^{\top }/p$ as $v_{F_{\gamma }}\left(z\right)$, called companion Stieltjes transform. For all $z\in \mathbb{C}\setminus {\mathbb{R}}^{+}$, the Stieltjes transform $v_{F_{\gamma }}\left(z\right)$ is related to $m_{F_{\gamma }}\left(z\right)$ by

$$\gamma \left[m_{F_{\gamma }}(z)+\frac{1}{z}\right]=v_{F_{\gamma }}(z)+\frac{1}{z}$$ (8)

where $\gamma$ is the limiting aspect ratio. In addition, we denote by $m_F^{\prime }(-\lambda )$ the derivative of the Stieltjes transform $m_F\left(z\right)$ evaluated at $z=-\lambda$, where

$$m_{F_{\gamma }}^{\prime }(z)={\int }_{l=0}^{\infty }\frac{dG\left(l\right)}{(l-z{)}^2}{v}_{F_{\gamma }}^{\prime }(z)=\gamma \left(m_{F_{\gamma }}^{\prime }(z)-\frac{1}{z^2}\right)+\frac{1}{z^2}.$$ (9)

In terms of the empirical quantities, similarly we have

$$m_{F^{\widehat{\Sigma }}}^{\prime }(z)=\text{tr}\left\{{\left(\widehat{\Sigma }-z{\mathbb{I}}_p\right)}^{-2}/p\right\}{\to }_{a.s.}{m}_{F_{\gamma }}^{\prime }(z).(\text{Bai and Silverstein},2010)$$

These convergences form the bases on which we develop the limiting error rate and the limiting optimal weights according to criteria (3) and (4).

4. Asymptotic Analysis of Weights and Classification Errors

In this section, we present the expressions for the limiting prediction erorrs and the optimal weights. All formulae are compared with simulated data, and in Appendix B, they are demonstrated to be accurate even under small data sizes.

4.1. Classification Error

Under the two-class Gaussian classification model (Assumption 1), the expected test error of the linear classifier TL-RDA under weight $\mathbf{w}$ in target population $K$ can be written as

$$Err(\mathbf{w})={\pi }_{K,-}\Phi \left(\frac{(\widehat{d}(\mathbf{w}){)}^{\top }{\mu }_{-1,K}+{\widehat{b}}_K}{\sqrt{(\widehat{d}(\mathbf{w}){)}^{\top }\Sigma (\widehat{d}(\mathbf{w}))}}\right)+{\pi }_{K,+}\Phi \left(-\frac{(\widehat{d}(\mathbf{w}){)}^{\top }{\mu }_{1,K}+{\widehat{b}}_K}{\sqrt{(\widehat{d}(\mathbf{w}){)}^{\top }\Sigma (\widehat{d}(\mathbf{w}))}}\right)$$

$$\widehat{d}(\mathbf{w})≔\sum _{k=1}^K{w}_k{\widehat{d}}_k$$

where $\Phi (\cdot )$ is the cumulative density function of a standard normal distribution and $\widehat{d}(\mathbf{w})$ is the transfer learning discriminating vector. A simple proof of this formula is given in the Appendix. In this paper, we assume the balanced class such that ${\pi }_{+}={\pi }_{-}$. The proof techniques generalize immediately to unbalanced cases, which is discussed briefly in Appendix C. The limiting form of $Err\left(\mathbf{w}\right)$ is given by Theorem 4.1 below. Two technical assumptions are required so $Z_k$ and the spectrum of $\Sigma$ are well-behaved.

Assumption 4 (Bounded Moment) Assume for each natural number $p$, the entries of $Z$ written by ${\left(Z_k\right)}_{ij}$ has uniformly bounded $p$-th moment. That is, there are constants $C_p$ such that

$$\mathbb{E}{\left|{\left(Z_k\right)}_{ij}\right|}^p\le C_p.$$

Assumption 5 (Anisotropic Local Laws) Define the cumulative distribution function of $H$ as $F\left(x\right)=\sum _{i=1}^{p}I\left({\lambda }_i\le x\right)/p$, where $I(\cdot )$ is the indicator function. Recall ${\lambda }_1\ge {\lambda }_2\ge \cdots \ge {\lambda }_p\ge 0$ are the eigenvalues of $\Sigma$. For a arbitrarily small positive constant $\tau >0$, we assume

$$F\left(\tau \right)\le 1-\tau .$$

We are now in position to state the first main result of this section. The following theorem describes the asymptotic classification error for the regularized transfer learning classifier defined in (2). The result holds under the aforementioned model assumptions.

Theorem 4.1 (Asymptotic Classification Error for TL-RDA) Suppose that assumptions 1–2 as well as 4 and 5 hold. Then for a fixed $K\ge 2$, as $n_k,p\to \infty ,p/n_k\to {\gamma }_k\in (0,\infty ]$ for $1\le k\le K$, we have that the limiting form of $Err\left(\mathbf{w}\right)$ for a given weight vector $\mathbf{w}\in {\mathbb{R}}^K$ is given by:

$$\Phi \left(-\frac{{\mathbf{u}}^{\top }\mathbf{w}}{\sqrt{{\mathbf{w}}^{\top }𝒜\mathbf{w}}}\right)$$ (10)

where the elements of $\mathbf{u}\in {\mathbb{R}}^K$ and $𝒜\in {\mathbb{R}}^{K\times K}$ are as follows:

$$u_k={\rho }_{kK}{\alpha }_k{\alpha }_K{m}_{F_{{\gamma }_k}}\left(-{\lambda }_k\right)\mathit{\text{for}}k=1,\dots ,K,$$

and

$${𝒜}_{k{k}^{\prime }}=\left\{\begin{array}{ll}{\alpha }_k^2\left[\frac{v_{F_{{\gamma }_k}}\left(-{\lambda }_k\right)-{\lambda }_k{v}_{F_{{\gamma }_k}}^{\prime }\left(-{\lambda }_k\right)}{{\gamma }_k{\left[{\lambda }_k{v}_{F_{{\gamma }_k}}\left(-{\lambda }_k\right)\right]}^2}\right]+\frac{v_{F_{{\gamma }_k}}^{\prime }\left(-{\lambda }_k\right)-v_{F_{{\gamma }_k}}^2\left(-{\lambda }_k\right)}{{\lambda }_k^2{v}_{F_{{\gamma }_k}}^4\left(-{\lambda }_k\right)}& \mathit{\text{if}}k=k^{\prime },\\ {\rho }_{k{k}^{\prime }}{\alpha }_k{\alpha }_{k^{\prime }}{ℳ}_{k{k}^{\prime }}& \mathit{\text{otherwise}},\end{array}\right.$$

for $k,k^{\prime }=1,\cdots ,K$. Here $m_{F_{{\gamma }_k}}(\cdot )$, $v_{F_{{\gamma }_k}}(\cdot )$ and $v_{F_{{\gamma }_k}}^{\prime }(\cdot )$ are the Stieltjes transform, the companion Stieltjes transform and its derivative respectively, as defined in (6)–(9). Moreover for $k,k^{\prime }=1,\cdots ,K$ let ${ℳ}_{k{k}^{\prime }}$ be the constants defined as the following limiting quantities:

$$\text{tr}\left[{\left({\widehat{\Sigma }}_k+{\lambda }_k{\mathbb{I}}_p\right)}^{-1}{\left({\widehat{\Sigma }}_{k^{\prime }}+{\lambda }_{k^{\prime }}{\mathbb{I}}_p\right)}^{-1}\Sigma \right]/p{\to }_{a.s.}{ℳ}_{k{k}^{\prime }}.$$

Remark 4.2 The limits of the off-diagonal terms in $𝒜$ depends on the interplay of sample covariances in separated populations. Similar terms also appear later in the paper when finding the optimal weight vector (see Theorems 4.5, 4.10). Depending on the specific scenario, the term ${ℳ}_{k{k}^{\prime }}$ has explicit expressions, some of which are presented separately in Lemma A.8.

Remark 4.3 When $K=1$, writing ${\lambda }_1,{\gamma }_1,{\alpha }_1,m_{F_{{\gamma }_1}}\left(-{\lambda }_1\right),v_{F_{{\gamma }_1}}\left(-{\lambda }_1\right)$, and $v_{F_{{\gamma }_1}}^{\prime }\left(-{\lambda }_1\right)$ as $\lambda ,\gamma ,\alpha ,m,v$ and $v^{\prime }$ respectively yields

$${𝒜}_{11}=\frac{1}{{\lambda }^2{v}^2}\left[\frac{{\alpha }^2\left(v-\lambda v^{\prime }\right)}{\gamma }+\frac{v^{\prime }}{v^2}-1\right],$$

which implies that the limiting misclassification error is given by

$$\text{Err}(w)\to \Phi \left(\frac{{\alpha }^{2}m}{\sqrt{{𝒜}_{11}}}\right)=\Phi \left(-{\alpha }^2\lambda mv{\left\{\frac{{\alpha }^2\left(v-\lambda v^{\prime }\right)}{\gamma }+\frac{v^{\prime }}{v^2}-1\right\}}^{-1/2}\right).$$

This matches exactly with Theorem 3.1 of Dobriban and Wager (2018).

We now consider the classification error of the transfer learning classifier, when using the pooled covariance matrix. Let us replace $\widehat{d}(\mathbf{w})$ with ${\widehat{d}}_P\left(\mathbf{w}\right)$ in $Err\left(\mathbf{w}\right)$ and define this new function of $\mathbf{w}$ as ${Err}_P\left(\mathbf{w}\right)$. This gives us the classification error for TLP-RDA given a weight vector $\mathbf{w}$.

Corollary 4.4 (Asymptotic Classification Error for TLP-RDA) Under the same set up as Theorem 4.1, we further define $p/\sum _{k=1}^K{n}_k\to \bar{\gamma }$. Then assuming ${\lambda }_1=\cdots ={\lambda }_K=\lambda$, we have

$${Err}_P(\mathbf{w})=\Phi \left(-\frac{{\mathbf{u}}_P^{\top }\mathbf{w}}{\sqrt{{\mathbf{w}}^{\top }{𝒜}_P\mathbf{w}}}\right)$$

where the elements of ${\mathbf{u}}_P\in {\mathbb{R}}^K$ and ${𝒜}_P\in {\mathbb{R}}^{K\times K}$ are as follows:

$${\left(u_P\right)}_k={\rho }_{kK}{\alpha }_k{\alpha }_K{m}_{F_{\bar{\gamma }}}\left(-\lambda \right)\mathit{\text{for}}k=1,\dots ,K,$$

and

$${\left({𝒜}_P\right)}_{k{k}^{\prime }}=\left\{\begin{array}{ll}{\alpha }_k^2\left[\frac{v_{F_{\bar{\gamma }}}(-\lambda )-\lambda v_{F_{\bar{\gamma }}}^{\prime }(-\lambda )}{\bar{\gamma }{\left[\lambda v_{F_{\bar{\gamma }}}(-\lambda )\right]}^2}\right]+{\gamma }_k\left[\frac{v_{F_{\bar{\gamma }}}^{\prime }\left(-{\lambda }_k\right)-v_{F_{\bar{\gamma }}}^2(-\lambda )}{\bar{\gamma }{\lambda }^2{v}_{F_{\bar{\gamma }}}^4(-\lambda )}\right]& \mathit{\text{if}}k=k^{\prime },\\ {\rho }_{k{k}^{\prime }}{\alpha }_k{\alpha }_{k^{\prime }}\left[\frac{v_{F_{\bar{\gamma }}}(-\lambda )-\lambda v_{F_{\bar{\gamma }}}^{\prime }(-\lambda )}{\bar{\gamma }{\left[\lambda v_{F_{\bar{\gamma }}}(-\lambda )\right]}^2}\right]& \mathit{\text{otherwise}},\end{array}\right.$$

for $k,k^{\prime }=1,\cdots ,K$. Here $m_{F_{\bar{\gamma }}}(\cdot ),v_{F_{\bar{\gamma }}}(\cdot )$ and $v_{F_{\bar{\gamma }}}^{\prime }(\cdot )$ are the Stieltjes transform, the companion Stieltjes transform and its derivative respectively, as defined in (6)–(9).

In this corollary, we make the assumption that all studies have the same degree of penalization, which brings us the simplified cross terms in ${𝒜}_P$ and a more amenable expression for ${𝒜}_P$ as a whole. This assumption might not be reasonable when the signal strengths ${\alpha }_k^2$ are vastly different, as lighter penalization may be given to population with stronger signal strength. In this case, one can use the general formula in Theorem 4.1.

4.2. Minimum Estimation Risk Weight

We firstly present the way to minimize the coordinate-wise estimation error of $\widehat{d}(W)$ with respect to $d_{\mathit{\text{Bayes}}}$.

Theorem 4.5 (Asymptotic Estimation Error Minimization for TL-RDA) Suppose that assumptions 1–2 as well as 4 and 5 hold. Then for a fixed $K\ge 2$, as $n_k,p\to \infty ,p/n_k\to {\gamma }_k\in (0,\infty ]$ for $1\le k\le K$, the weight for minimizing the error in estimating the Bayes optimal discriminator $d_{\mathit{\text{Bayes}}}$ is given by:

$${\mathbf{w}}^E≔\text{arg}\underset{\mathbf{w}}{\text{min}}{‖d_{\mathit{\text{Bayes}}}-\sum _{k=1}^K{w}_k{\widehat{d}}_k‖}_2^2={\left({𝒜}^E+{ℛ}^E\right)}^{-1}{\mathbf{u}}^E,$$

where the elements of ${\mathbf{u}}^E\in {\mathbb{R}}^K,{𝒜}^E\in {\mathbb{R}}^{K\times K}$, and ${ℛ}^E\in {\mathbb{R}}^{K\times K}$ are:

$${\left(u^E\right)}_k={\rho }_{kK}{\alpha }_k{\alpha }_K\left[\frac{1}{{\lambda }_k}\mathbb{E}\left(T^{-1}\right)-m_{F_{{\gamma }_k}}{\left(-{\lambda }_k\right)}^2\right]\mathit{\text{for}}k=1,\dots ,K,$$

$${𝒜}_{k{k}^{\prime }}^E=\left\{\begin{array}{ll}{\alpha }_k^2{m}_{F_{{\gamma }_k}}^{\prime }\left(-{\lambda }_k\right)& \mathit{\text{if}}k=k^{\prime },\\ {\rho }_{k{k}^{\prime }}{\alpha }_k{\alpha }_{k^{\prime }}{ℰ}_{k{k}^{\prime }}& \mathit{\text{otherwise}},\end{array}\right.$$

and

$${ℛ}_{k{k}^{\prime }}^E=\left\{\begin{array}{ll}\frac{v_{F_{{\gamma }_k}}\left(-{\lambda }_k\right)-{\lambda }_k{v}_{F_{{\gamma }_k}}^{\prime }\left(-{\lambda }_k\right)}{{\lambda }_k{v}_{F_{{\gamma }_k}}{\left(-{\lambda }_k\right)}^2}& \mathit{\text{if}}k=k^{\prime },\\ 0& \mathit{\text{otherwise}},\end{array}\right.$$

for $k,k^{\prime }=1,\cdots ,K$. Here $m_{F_{{\gamma }_k}}(\cdot )$, $v_{F_{{\gamma }_k}}(\cdot )$ and $v_{F_{{\gamma }_k}}^{\prime }(\cdot )$ are the Stieltjes transform, the companion Stieltjes transform and its derivative respectively, as defined in (6)–(9), while $T$ is the limiting spectral distribution of the population covariance matrix $\Sigma$. Moreover for $k,k^{\prime }=1,\cdots ,K$, let ${ℰ}_{k{k}^{\prime }}$ be the constants defined as the following limiting quantities:

$$\text{tr}\left[{\left({\widehat{\Sigma }}_k+{\lambda }_k{\mathbb{I}}_p\right)}^{-1}{\left({\widehat{\Sigma }}_{k^{\prime }}+{\lambda }_{k^{\prime }}{\mathbb{I}}_p\right)}^{-1}\right]/p{\to }_{a.s.}{ℰ}_{k{k}^{\prime }}.$$

Just like the expressions for the asymptotic classification rate in Theorem 4.1, more explicit forms of ${ℰ}_{k{k}^{\prime }}$ are available if ${\gamma }_k$ and ${\lambda }_k$ are the same across populations. We call ${\mathbf{w}}^E$ the minimum estimation weight, as it minimizes the ${\ell }_2$ error in estimating the Bayes optimal discriminating vector $d_{\mathit{\text{Bayes}}}$. In order to better illustrate the effect of transfer learning, we consider next a simpler situation where all the sources are homogeneous, in that ${\gamma }_1={\gamma }_2=\cdots ={\gamma }_K$ and the corresponding regularization parameters are also the same, i.e., ${\lambda }_1={\lambda }_2=\cdots ={\lambda }_K$. We further assume equal correlation among all sources, i.e., ${\rho }_{k{k}^{\prime }}=\rho$ whenever $k\ne k^{\prime }$.

Corollary 4.6 (Estimation Error for Homogeneous Sources) Under the setup of Theorem 4.5, suppose ${\gamma }_1={\gamma }_2=\cdots ={\gamma }_K≕\gamma ,{\lambda }_1={\lambda }_2=\cdots ={\lambda }_K≕\lambda$, and ${\rho }_{k{k}^{\prime }}=\rho$ whenever $k\ne k^{\prime },1\le k,k^{\prime }\le K$. Then the weight vector ${\mathbf{w}}^E$ that minimizes the asymptotic error in estimating $d^{\text{Bayes}}$ is given by:

$${\mathbf{w}}^E={\alpha }_K\left[\frac{1}{\lambda }\mathbb{E}\left(T^{-1}\right)-m_{F_{\gamma }}(-\lambda {)}^2\right]\text{vec}\left(\frac{\{\rho -\xi +(1-\rho \left)I\right(k=K\left)\right\}{\alpha }_k}{t_{m,\epsilon ,\lambda }{\alpha }_k^2+t_{v,\lambda }}:1\le k\le K\right)$$

where $\epsilon ={ℰ}_{12},t_{\rho ,\epsilon ,\lambda }≔-\rho \epsilon +m_{F_{\gamma }}^{\prime }(-\lambda ),t_{v,\lambda }≔\frac{v_{F_{\gamma }}(-\lambda )-\lambda v_{F_{\gamma }}^{\prime }(-\lambda )}{\lambda v_{F_{\gamma }}(-\lambda {)}^2}$ and

$$\xi =\frac{\rho \epsilon \sum _{k=1}^K\{\rho +(1-\rho )I(k=K){\}}_{\frac{{\alpha }_k}{t_{m,\epsilon ,\lambda }{\alpha }_k^2+t_{v,\lambda }}}}{1+\rho \epsilon \sum _{k=1}^K\frac{{\alpha }_k^2}{t_{m,\epsilon ,\lambda }{\alpha }_k^2+t_{v,\lambda }}}.$$

The above corollary shows that when all sources have equal sample sizes, and equal correlation, the optimal weight is proportional to ${\alpha }_k{\alpha }_K/\left(t_{m,\epsilon ,\lambda }{\alpha }_k^2+t_{v,\lambda }\right)$. In particular, if ${\alpha }_k\to 0$ for some source population, then the corresponding weight for that source increases at the rate of $1/{\alpha }_k$. This is intuitive since in the homogeneous setting of equal sample sizes, the signal strength is completely determined by the inverse of the variance of ${\delta }_k$, which is precisely $1/{\alpha }_k$. For practical usage of the optimal transfer weights, we now turn to the estimation of the above quantities.

Remark 4.7 (Estimating optimal weights) The Marchenko-Pastur law, along with the equations (6), (7), (8), and (9) imply that the sample Stieltjes transform $\text{tr}\left[{\left({\widehat{\Sigma }}_k-z{\mathbb{I}}_p\right)}^{-1}\right]$ can be suitably used to estimate the functions $m_{F_{{\gamma }_k}}(\cdot )$, $v_{F_{{\gamma }_k}}(\cdot )$, and $m_{F_{{\gamma }_k}}^{\prime }(\cdot )$ reliably. We remind the reader that the estimation of ${\alpha }_k^2$ and ${\rho }_{k{k}^{\prime }}$ is tackled in Appendix B.3. Finally, we note that the expression of ${\mathbf{u}}^E$ in all the above cases involves the expectation of $T^{-1}$, and we discuss a consistent estimator for $\mathbb{E}\left(T^{-1}\right)$ in the following proposition, the proof of which is immediate from Corollary 4.2 of Dobriban and Sheng (2021).

The following proposition holds under Assumptions 3.

Proposition 4.8 When ${\gamma }_k<1$, we have $\text{tr}\left({\widehat{\Sigma }}_k^{-1}\right)/p{\to }_{\mathit{\text{a.s.}}}\mathbb{E}\left(T^{-1}\right)/\left(1-{\gamma }_k\right)$.

Since in our developments so far we have assumed all populations to share the same covariance matrix $\Sigma$, one can always use the pooled sample covariance in the estimation scheme above. That means we can reliably estimate $\mathbb{E}\left(T^{-1}\right)$ as long as $\bar{\gamma }<1$. We then present the optimal estimation for TLP-RDA, i.e., the transfer learning discriminant analysis done using the pooled covariance matrix.

Corollary 4.9 (Asymptotic Estimation Error Minimization for TLP-RDA) Under the same set up as Theorem 4.5, assuming ${\lambda }_1=\cdots ={\lambda }_K=\lambda$, we have

$${\mathbf{w}}_P^E≔\text{arg}\underset{\mathbf{w}}{\text{min}}{‖d_{\mathit{\text{Bayes}}}-\sum _{k=1}^K{w}_k{\hat{d}}_k^P‖}_2^2={\left({𝒜}_P^E+{ℛ}_P^E\right)}^{-1}{\mathbf{u}}_P^E,$$

where the elements of ${\mathbf{u}}_P^E\in {\mathbb{R}}^K,{𝒜}_P^E\in {\mathbb{R}}^{K\times K}$, and ${ℛ}_P^E\in {\mathbb{R}}^{K\times K}$ are:

$${\left(u_P^E\right)}_k={\rho }_{kK}{\alpha }_k{\alpha }_K\left[\frac{1}{\lambda }\mathbb{E}\left(T^{-1}\right)-m_{F_{\bar{\gamma }}}(-\lambda {)}^2\right]\mathit{\text{for}}k=1,\dots ,K,$$

while

$${\left({𝒜}_P^E\right)}_{k{k}^{\prime }}={\rho }_{k{k}^{\prime }}{\alpha }_k{\alpha }_{k^{\prime }}{m}_{F_{\bar{\gamma }}}^{\prime }(-\lambda )$$

and

$${\left({ℛ}_P^E\right)}_{k{k}^{\prime }}=\left\{\begin{array}{ll}{\gamma }_k\frac{v_{F_{\bar{\gamma }}}(-\lambda )-\lambda v_{F_{\bar{\gamma }}}^{\prime }(-\lambda )}{\bar{\gamma }\lambda v_{F_{\bar{\gamma }}}(-\lambda {)}^2}& \mathit{\text{if}}k=k^{\prime },\\ 0& \mathit{\text{otherwise}},\end{array}\right.$$

for $k,k^{\prime }=1,\cdots ,K$.

Once again we can follow the remark and proposition above for estimating the weight that minimizes the estimation error when using the pooled covariance matrix.

4.3. Minimum Prediction Risk Weight

In the high dimensional regime considered in this paper, a weight minimizing the estimation error does not translate into maximizing the classification score $\widehat{d}(\mathbf{w}{)}^{\top }{x}_0$. We can as well compute the optimal prediction weight by directly minimizing the difference between the TL-RDA classification score and the Bayes classification score. This is precisely the objective of the next theorem.

Theorem 4.10 (Asymptotic Prediction Error Minimization for TL-RDA) Suppose that Assumptions 1–2 as well as 4 and 5 hold. Then for a fixed $K\ge 2$, as $n_k,p\to \infty ,p/n_k\to {\gamma }_k\in (0,\infty ]$ for $1\le k\le K$, the weight for minimizing the excess risk, i.e., the error in predicting the class at a random test point $x_0$, when compared to the Bayes optimal discriminator $d_{\mathit{\text{Bayes}}}$, is given by:

$${\mathbf{w}}^P≔\text{arg}\underset{\mathbf{w}}{\text{min}}{\mathbb{E}}_{x_0}{\left[{\left(d_{\mathit{\text{Bayes}}}-\sum _{k=1}^K{w}_k{\widehat{d}}_k\right)}^{\top }{x}_0\right]}^2={\left({𝒜}^P+{ℛ}^P\right)}^{-1}{\mathbf{u}}^P,$$

where the elements of ${\mathbf{u}}^P\in {\mathbb{R}}^K,{𝒜}^P\in {\mathbb{R}}^{K\times K}$, and ${ℛ}^P\in {\mathbb{R}}^{K\times K}$ are:

$${\left(u^P\right)}_k={\rho }_{kK}{\alpha }_k{\alpha }_K{m}_{F_{{\gamma }_k}}\left(-{\lambda }_k\right)\mathit{\text{for}}k=1,\dots ,K,$$

$${𝒜}_{k{k}^{\prime }}^P=\left\{\begin{array}{ll}{\alpha }_k^2\left[\frac{v_{F_{{\gamma }_k}}\left(-{\lambda }_k\right)-{\lambda }_k{v}_{F_{{\gamma }_k}}^{\prime }\left(-{\lambda }_k\right)}{{\gamma }_k{\left[{\lambda }_k{v}_{F_{{\gamma }_k}}\left(-{\lambda }_k\right)\right]}^2}\right]& \mathit{\text{if}}k=k^{\prime },\\ {\rho }_{k{k}^{\prime }}{\alpha }_k{\alpha }_{k^{\prime }}{ℳ}_{k{k}^{\prime }}& \mathit{\text{otherwise}},\end{array}\right.$$

and

$${ℛ}_{k{k}^{\prime }}^E=\left\{\begin{array}{ll}\frac{v_k^{\prime }\left(-{\lambda }_k\right)-v_k^2\left(-{\lambda }_k\right)}{{\lambda }_k^2{v}_k^4\left(-{\lambda }_k\right)}& \mathit{\text{if}}k=k^{\prime },\\ 0& \mathit{\text{otherwise}},\end{array}\right.$$

for $k,k^{\prime }=1,\cdots ,K$. Here $m_{F_{{\gamma }_k}}(\cdot ),v_{F_{{\gamma }_k}}(\cdot )$ and $v_{F_{{\gamma }_k}}^{\prime }(\cdot )$ are the Stieltjes transform, the companion Stieltjes transform and its derivative respectively, as defined in (6)–(9). Moreover for $k,k^{\prime }=1,\cdots ,K$, let ${ℳ}_{k{k}^{\prime }}$ be the constants defined as the following limiting quantities:

$$\text{tr}\left[{\left({\widehat{\Sigma }}_k+{\lambda }_k{\mathbb{I}}_p\right)}^{-1}{\left({\widehat{\Sigma }}_{k^{\prime }}+{\lambda }_{k^{\prime }}{\mathbb{I}}_p\right)}^{-1}\Sigma \right]/p{\to }_{a.s.}{ℳ}_{k{k}^{\prime }}.$$

As in the case of estimation error weights, we discuss the issue of estimating the optimal weight that minimizes prediction error in this setting. Note that unlike the estimation error minimizing weight ${𝒲}^E$, the prediction risk minimizing weight ${𝒲}^P$, involves no population spectral distribution. It is thus estimable in all cases, including the case when $\gamma >1$. On the other hand, estimating ${ℳ}_{k{k}^{\prime }}$ requires more care, especially in the case $k=K$ or $k^{\prime }=K$. The following proposition describes the estimation in this case.

Proposition 4.11 We have the following consistent estimators ${\widehat{ℳ}}_{k{k}^{\prime }}$ for ${ℳ}_{k{k}^{\prime }}$ separately for three cases:

$${\widehat{ℳ}}_{k{k}^{\prime }}=\left\{\begin{array}{ll}\text{tr}\left[{\left({\widehat{\Sigma }}_k+{\lambda }_k{\mathbb{I}}_p\right)}^{-1}{\left({\widehat{\Sigma }}_{k^{\prime }}+{\lambda }_{k^{\prime }}{\mathbb{I}}_p\right)}^{-1}{\widehat{\Sigma }}_K\right]/p& \mathit{\text{if}}k\ne k^{\prime },k\ne K,k^{\prime }\ne K\\ \text{tr}\left[{\left({\widehat{\Sigma }}_k+{\lambda }_k{\mathbb{I}}_p\right)}^{-2}{\widehat{\Sigma }}_K\right]/p& \mathit{\text{if}}k=k^{\prime }\ne K\\ \frac{1}{p{x}_p}\text{tr}\left[{\left({\widehat{\Sigma }}_{k^{\prime }}+{\lambda }_{k^{\prime }}{\mathbb{I}}_p\right)}^{-1}\right]-\frac{{\lambda }_K}{p{x}_p}\text{tr}\left[{\left({\widehat{\Sigma }}_K+{\lambda }_K{\mathbb{I}}_p\right)}^{-1}{\left({\widehat{\Sigma }}_{k^{\prime }}+{\lambda }_{k^{\prime }}{\mathbb{I}}_p\right)}^{-1}\right]& \mathit{\text{if}}k=K,k^{\prime }\ne K\end{array}\right.$$

where $x_p=x\left({\gamma }_K,{\lambda }_K\right)$ is the solution to the equation:

$$1-x_p={\gamma }_K\left[1-{\lambda }_K\int {\left(x_{p}t+{\lambda }_K\right)}^{-1}d{H}_K(t)\right]$$

Then

$${ℳ}_{k{k}^{\prime }}-{\widehat{ℳ}}_{k{k}^{\prime }}{\to }_{a.s.}0.$$

In order to better illustrate the effect of transfer learning, we consider next a simpler situation where all the sources are homogeneous, in that ${\gamma }_1={\gamma }_2=\cdots ={\gamma }_K$ and the corresponding regularization parameters are also the same, i.e., ${\lambda }_1={\lambda }_2=\cdots ={\lambda }_K$. We further assume equal correlation among all sources, i.e., ${\rho }_{k{k}^{\prime }}=\rho$ whenever $k\ne k^{\prime }$.

Corollary 4.12 (Prediction Error for Homogeneous Sources) Under the earlier setup of Theorem 4.5, suppose ${\gamma }_1={\gamma }_2=\cdots ={\gamma }_K≕\gamma ,{\lambda }_1={\lambda }_2=\cdots ={\lambda }_K≕\lambda$, and ${\rho }_{k{k}^{\prime }}=\rho$ whenever $k\ne k^{\prime },1\le k,k^{\prime }\le K$. Then the weight vector ${\mathbf{w}}^P$ that minimizes the asymptotic excess risk is given by:

$${\mathbf{w}}^P={\alpha }_K{m}_{F_{\gamma }}(-\lambda )\text{vec}\left(\frac{\left\{\rho -{\xi }^P+(1-\rho )I(k=K)\right\}{\alpha }_k}{t_{v,\rho ,\lambda }^P{\alpha }_k^2+t_{v,\lambda }^P}:1\le k\le K\right)$$

where $m={ℳ}_{12},t_{v,\rho ,\lambda }^P≔-m\rho +\frac{v_{F_{\gamma }}(-\lambda )-\lambda v_{F_{\gamma }}^{\prime }(-\lambda )}{\gamma {\left[\lambda v_{F_{\gamma }}(-\lambda )\right]}^2},t_{v,\lambda }^P≔\frac{v^{\prime }(-\lambda )-v^2(-\lambda )}{{\lambda }^2{v}^4(-\lambda )}$ and

$${\xi }^P=\frac{m\rho {\sum }_{k=1}^K\{\rho +(1-\rho )I(k=K)\}\frac{{\alpha }_k^2}{t_{v,\rho ,\lambda }^P{\alpha }_k^2+t_{v,\lambda }^P}}{1+m\rho \sum _{k=1}^K\frac{{\alpha }_k^2}{t_{v,\rho ,\lambda }^P{\alpha }_k^2+t_{v,\lambda }^P}}.$$

The above corollary shows that when all sources have equal sample sizes, and equal correlation, the optimal weight is proportional to ${\alpha }_k{\alpha }_K/\left(t_{v,\rho ,\lambda }^P{\alpha }_k^2+t_{v,\lambda }^P\right)$. Thus similar to Corollary 4.6, if ${\alpha }_k\to 0$ for some source population, then the corresponding weight for that source increases at the rate of $1/{\alpha }_k$. This is intuitive since in the homogeneous setting of equal sample sizes, the signal strength is completely determined by the inverse of the variance of ${\delta }_k$, which is precisely $1/{\alpha }_k$. Finally, since our assumption so far posits the same variance $\Sigma$ for all populations, we also describe the estimation based on the pooled sample covariance matrix. This is given in the following corollary for the pooled covariance based classifier TLP-RDA.

Corollary 4.13 (Asymptotic Prediction Error Minimization for TLP-RDA) Under the same set up as theorem 4.10, assume ${\lambda }_1=\cdots ={\lambda }_K=\lambda$, we have

$${\mathbf{w}}_P^P≔\text{arg}\underset{\mathbf{w}}{\text{min}}{\mathbb{E}}_{x_0}{\left[{\left(d_{\mathit{\text{Bayes}}}-\sum _{k=1}^K{w}_k{\widehat{d}}_k^P\right)}^{\top }{x}_0\right]}^2={\left({𝒜}_P^P+{ℛ}_P^P\right)}^{-1}{\mathbf{u}}_P^P$$

where the elements of ${\mathbf{u}}_P^P\in {\mathbb{R}}^K,{𝒜}_P^P\in {\mathbb{R}}^{K\times K}$, and ${ℛ}_P^P\in {\mathbb{R}}^{K\times K}$ are:

$${\left(u_P^P\right)}_k={\rho }_{kK}{\alpha }_k{\alpha }_K{m}_{F_{\bar{\gamma }}}\left(-\lambda \right)\mathit{\text{for}}k=1,\dots ,K,$$

while

$${\left({𝒜}_P^P\right)}_{k{k}^{\prime }}={\rho }_{k{k}^{\prime }}{\alpha }_k{\alpha }_{k^{\prime }}\left[\frac{v_{F_{\bar{\gamma }}}(-\lambda )-\lambda v_{F_{\bar{\gamma }}}^{\prime }(-\lambda )}{\bar{\gamma }{\left[\lambda v_{F_{\bar{\gamma }}}(-\lambda )\right]}^2}\right]$$

and

$${\left({ℛ}_P^P\right)}_{k{k}^{\prime }}=\left\{\begin{array}{ll}{\gamma }_k\frac{v_{F_{\bar{\gamma }}}^{\prime }\left(-{\lambda }_k\right)-v_{F_{\bar{\gamma }}}^2(-\lambda )}{\bar{\gamma }{\lambda }^2{v}_{F_{\bar{\gamma }}}^4(-\lambda )}& \mathit{\text{if}}k=k^{\prime },\\ 0& \mathit{\text{otherwise}},\end{array}\right.$$

for $k,k^{\prime }=1,\cdots ,K$.

The optimal prediction weights ${\mathbf{w}}_P^E,{\mathbf{w}}_P^P$ dominate ${\mathbf{w}}^E,{\mathbf{w}}^P$ respectively in classification error, meaning

$$Err\left({\mathbf{w}}^E\right)\ge Err\left({\mathbf{w}}^P\right),{Err}_P\left({\mathbf{w}}_P^E\right)\ge {Err}_P\left({\mathbf{w}}_P^P\right).$$

Note that a priori it is not immediate that the weight which minimizes the difference in prediction error from the Bayes optimal direction, also minimizes the classification error. In particular, the accuracy of classification depends on the sign of $\widehat{d}(\mathbf{w}{)}^{\top }{x}_0$, and not the actual value. However, we now show that in addition to optimizing the score $\widehat{d}(\mathbf{w}{)}^{\top }{x}_0$, the prediction weights ${\mathbf{w}}^P$ and ${\mathbf{w}}_P^P$ also minimize the misclassification error. This is formalized by the following proposition.

Proposition 4.14 The optimal prediction weight minimizes the testing data classification error:

$${\mathbf{w}}^P=\text{arg}\underset{\mathbf{w}}{\text{min}}{\mathbb{E}}_{x_0,y_0}\left[I\left(\text{sign}\left[\left(\widehat{d}\right(\mathbf{w}){)}^{\top }{x}_0\right]\ne y_0\right)\right],$$

$${\mathbf{w}}_P^P=\text{arg}\underset{\mathbf{w}}{\text{min}}{\mathbb{E}}_{x_0,y_0}\left[I\left(\text{sign}\left[{\left({\widehat{d}}_P\left(\mathbf{w}\right)\right)}^{\top }{x}_0\right]\ne y_0\right)\right].$$

We conclude this section by validating the existence of the optimal weights. Indeed, the solution to all four types of weights implicitly assume that the matrices ${𝒜}^E+{ℛ}^E,{𝒜}^P+{ℛ}^P,{𝒜}_P^E+{ℛ}_P^E,{𝒜}_P^P+{ℛ}_P^P$ are invertible. We prove this is always the case in the following proposition.

Proposition 4.15 (Existence of Optimal Weights) The matrices ${𝒜}^E+{ℛ}^E,{𝒜}^P+{ℛ}^P,{𝒜}_P^E+{ℛ}_P^E,{𝒜}_P^P+{ℛ}_P^P$ are invertible, and hence the limiting optimal weights ${\mathbf{w}}^E,{\mathbf{w}}^P,{\mathbf{w}}_P^E,{\mathbf{w}}_P^P$ exist.

We compare the error rates of the four transfer learning estimators under several different scenarios in Appendix B.

4.4. Geometric Interpretation

We now provide some geometric interpretations on the optimal weights derived in the previous subsections. Let us define the following discriminant directions:

$$d_{est}≔\widehat{d}\left({\mathbf{w}}_E\right),d_{err}≔\widehat{d}\left({\mathbf{w}}_P\right).$$

Also recall the Bayes discriminant direction $d_{\mathit{\text{Bayes}}}$. These discriminant directions are visualized in Figure 1. The blue plane is the space spanned by linear combinations of the local discriminant directions $\left\{{\widehat{d}}_k\right\}$, and the top black line stands for the Bayes direction, which is not necessarily in the linear span of $\left\{{\widehat{d}}_k\right\}$. Since both the TL-RDA directions $d_{\mathit{\text{est}}}$ and $d_{\mathit{\text{err}}}$ are linear combinations of ${\widehat{d}}_k$, they both lie on the blue plane and are denoted by colored lines. The direction obtained by minimizing the error in estimating $d_{\mathit{\text{Bayes}}}$ is given by $d_{\mathit{\text{est}}}$, which is the projection of $d_{\mathit{\text{Bayes}}}$ onto the blue plane, denoting $\text{span}\left\{{\widehat{d}}_k\right\}$. The criterion 3 suggests that $d_{\mathit{\text{est}}}$ is the minimizer of the OLS loss when fitting $d_{\mathit{\text{Bayes}}}$ with linear combinations of $d_k$.

Figure 1:

Geometric interpretations of optimal weights

Denoting the angle between $d_{\mathit{\text{Bayes}}}$ and $\widehat{d}(\mathbf{w})$ as $\theta$, we now show that the cosine of $\theta$ is directly related to $Err\left(\mathbf{w}\right)$ accounting for the scaling $\Sigma$. Let us define the scaled inner product $⟨a,b{⟩}_{\Sigma }=a^{\top }\Sigma b$ and the scaled cos angle ${\text{cos}}_{\Sigma }(a,b)=⟨a,b{⟩}_{\Sigma }/\sqrt{⟨a,a{⟩}_{\Sigma }⟨b,b{⟩}_{\Sigma }}$. Then we have

$$\text{cos}\theta ≔{\text{cos}}_{\Sigma }\angle \left(\widehat{d}(\mathbf{w}),d_{\mathit{\text{Bayes}}}\right)=\widehat{d}(\mathbf{w}{)}^{\top }{\delta }_K/\sqrt{\widehat{d}(\mathbf{w}{)}^{\top }\Sigma \widehat{d}(\mathbf{w}){\delta }_K^{\top }{\Sigma }^{-1}{\delta }_K}=\frac{\Theta (\mathbf{w})}{{\Theta }_{\mathit{\text{Bayes}}}}$$

$$\Theta (\mathbf{w})≔\frac{\widehat{d}(\mathbf{w}{)}^{\top }{\delta }_K}{\sqrt{\widehat{d}(\mathbf{w}{)}^{\top }\Sigma \widehat{d}(\mathbf{w})}},{\Theta }_{\mathit{\text{Bayes}}}≔\sqrt{{\delta }_K^{\top }{\Sigma }^{-1}{\delta }_K}$$

Recall that $\Phi (-\Theta (\mathbf{w}\left)\right)$ is the classification error rate of TL-RDA (Theorem 4.1) and $\Phi \left(-{\Theta }_{\mathit{\text{Bayes}}}\right)$ is the Bayes error rate. This implies that

$$\text{cos}\theta =\frac{\Theta \left(\mathbf{w}\right)}{{\Theta }_{\mathit{\text{Bayes}}}}=\frac{{\Phi }^{-1}(Err(\mathbf{w}))}{{\Phi }^{-1}\left({Err}_{\mathit{\text{Bayes}}}\right)}.$$

Since ${\Phi }^{-1}(\cdot )$ is a monotonically increasing function, it is clear that $\text{cos}\left(\theta \right)$ is close to one, i.e., $\theta$ is close to zero, if and only if $Err\left(\mathbf{w}\right)$ is close to the Bayes error ${Err}_{\mathit{\text{Bayes}}}$. Therefore, the size of $\theta$ directly quantifies the inefficiency of TL-RDA relative to $d_{\mathit{\text{Bayes}}}$ in terms of classification error, and ${\mathbf{w}}^P$ simultaneously minimizes the classification error and $\text{cos}\theta$ (Proposition 4.14).

Finally, let us also consider an observation $x_0$ pointing in a random direction, and denote the angle between $d_{\mathit{\text{err}}}$ and $x_0$ as $\beta$ and the angle between $d_{\mathit{\text{Bayes}}}$ and $x_0$ as $\alpha$. Straightforwardly, $d_{\mathit{\text{err}}}$ minimizes the difference between $\beta$ and $\alpha$ as ${\mathbf{w}}^P$ minimizes the difference between the inner products $d_{\mathit{\text{err}}}^{\top }{x}_0$ and $d_{\mathit{\text{Bayes}}}^{\top }{x}_0$.

5. Robustness and Weight Selection

In this section, we present some guidance on how one can choose between different weighting schemes. As we have repeatedly illustrated in the previous section, the $d_{\mathit{\text{est}}}$ is dominated by $d_{\mathit{\text{err}}}$ in terms of classification error on unseen data. This claim, however, holds only when the test data distribution is as given by Assumption 1. The first goal of this section is to demonstrate that $d_{\mathit{\text{est}}}$ outperforms $d_{\mathit{\text{err}}}$ considerably in terms of classification error, when there is a distribution shift in the covariates. The second goal is to compare between the individual vs. pooled covariance estimators: i.e., TL-RDA and TLP-RDA. We will prove, at least in special cases, that the optimal TLP-RDA outperforms the optimal TL-RDA when the aspect ratio is large enough.

5.1. Robustness of Optimal Estimation Weight

In this section we demonstrate the robustness of $d_{\mathit{\text{est}}}$ to covariate shifts in test data. This robustness is intuitive as we know the optimal estimation weight only attempts to minimize the difference in TL discriminant direction $\left(\widehat{d}\right(\mathbf{w}\left)\right)$ and the Bayes discriminant direction $d_{\mathit{\text{Bayes}}}$. In fact, we can show the weights obtained by minimizing criteria (3) are equivalent to a conservative solution to the problem of minimizing criteria (4) when test data distribution is unknown. A similar argument is in Zhang and Li (2023), we summarize it in the following proposition.

Proposition 5.1 (Robustness of Estimation Weight) For the class of covariate distributions given by $𝒫≔\left\{P:x~P,{\mathbb{E}}_P\left(\Vert x{\Vert }_2\right)\le c\right\}$, we have:

$$\underset{\mathbf{w}}{\text{arg}\text{min}}{‖d_{\mathit{\text{Bayes}}}-\sum _{k=1}^K{w}_k{\widehat{d}}_k‖}^2=\underset{\mathbf{w}}{\text{arg}\text{min}}\underset{x_0\in 𝒫}{\text{max}}{\mathbb{E}}_{x_0}{\left[{\left(d_{\mathit{\text{Bayes}}}-\sum _{k=1}^K{w}_k{\widehat{d}}_k\right)}^{\top }{x}_0\right]}^2.$$

Proposition 5.1 claims that, when we only know the target data comes from a distribution with bounded expected norm, the safest option to minimize prediction error criteria (4) is to focus only on the estimation error criteria (3). We demonstrate the usefulness of this in Figure 2. Here $p=150$ and $n_k=250,\cdots ,160$ for study 1 to study $K$. The pairwise correlations across studies are fixed at 0.5. We use the same Toeplitz covariance matrix for all training data, however, the eigenvalues of the test data $x_0$ covariance matrix are modified to decay much faster. The testing accuracy of the three methods are shown on the $y$ axis as $\lambda$ changes. One can see the optimal estimation weight consistently outperforms others, suggesting the robustness and conservative nature of the optimal estimation weight boost the performance of TL-RDA when there is a change in testing data distribution.

Figure 2:

TL-RDA with the optimal estimation weight outperforms regular RDA and TL-RDA with the optimal prediction weight when the target distribution changes.

5.2. Pooled Sample Covariance and Individual Sample Covariance Matrix

For TLP-RDA, all discriminant directions uses the same covariance estimate ${\widehat{\Sigma }}_P$. Although ${\widehat{\Sigma }}_P$ is a better covariance estimate than all ${\widehat{\Sigma }}_k,{\widehat{d}}_k^P$ are inevitably similar and the columns space of ${\widehat{d}}_k^P$ would be less informational. This essentially becomes a bias-variance trade off. When ${\gamma }_k$ is small, the estimates ${\widehat{d}}_k$ are reliable already. In this case, individual covariance matrices bring more variances to this column space, therefore, increase the quality of final TL-RDA estimator. When ${\gamma }_k$ are large, direction estimate based on ${\widehat{\Sigma }}_k$ are no longer reliable and one should consider the more stable ${\widehat{\Sigma }}_P$. We will formalize this statement in this section also.

Proposition 5.2 (TLP-RDA out performs TL-RDA when $\mathit{\gamma }$ is large) Assume $\Sigma ={\mathbb{I}}_p,{\gamma }_1=\cdots ={\gamma }_K=\gamma ,{\lambda }_1,\cdots ,{\lambda }_K=\lambda =r\left(\gamma -\frac{1}{r+1}\right)$ for some fixed $r>(1-\gamma {)}_{+}/\gamma$. For the pooled covariance matrix, we choose ${\lambda }^{\prime }=r^{\prime }\left(\gamma /K-\frac{1}{r^{\prime }+1}\right)$ for some $r^{\prime }>(K-\gamma {)}_{+}/\gamma$.

1. When $\rho =1,Err\left({\mathbf{w}}^P\right)\ge {Err}_P\left({\mathbf{w}}_P^P\right)$, if and only if

   $${\gamma }^2\left[{\left(1+r^{\prime }\right)}^2-K(1+r{)}^2\right]\ge K\left(\left[\gamma (1+r{)}^2-1\right]\sum _k{\alpha }_k^2\right).$$
2. When $\rho =0,Err\left({\mathbf{w}}^P\right)\ge {Err}_P\left({\mathbf{w}}_P^P\right)$, if and only if

   $$\gamma \left[{\left(1+r^{\prime }\right)}^2-(1+r{)}^2\right]\ge K-1.$$

We also numerically demonstrate this phenomenon by plugging values into the limiting expressions under a more general set up (Figure 3). We again use $\rho =0.5$ while changing the other parameters, including the number of auxiliary studies $K$, the decay rate of the eigenvalues of $\Sigma$ and the signal strength $\alpha$. We can see that TLP-RDA can outperform TL-RDA in all cases as ${\gamma }_K$ grows, as the error rates of TLP-RDA decrease at slower rates. The transition point ${\gamma }^{*}$, defined as the ${\gamma }_K$ when TLP-RDA outperforms TL-RDA, differs in different scenarios. One can see that it decreases when the eigenvalues of $\Sigma$ decrease slower, and interestingly, also when the number of auxiliary population decreases.

Figure 3:

TLP-RDA outperforms TL-RDA when $\gamma$ is large under general setups.

6. Heterogeneous Population Covariance Matrix

The previous discussions of TL-RDA have been restricted to the case where observed covariates $X_k$ in all populations $k=1,\cdots ,K$ share the same covariance matrix $\Sigma$. In this section we extend TL-RDA to accommodate the scenario when covariance matrices are different. We call this generalization beyond identical matrices as transfer-learning-heterogeneous (TLH)-RDA. We firstly formalize this set up.

Assumption 6 (Heterogeneous Two-class Gaussian) We assume all populations $k=1,\cdots ,K$ follow the classic two-class Gaussian mixture model. More specifically, for $i=1,\dots ,n_k$ and $k=1,\dots ,K$,

$${\left(y_k\right)}_i\in \{-1,+1\}\mathbb{P}\left({\left(y_k\right)}_i=\pm 1\right)={\pi }_{\pm 1}{\left(X_k\right)}_i\mid {\left(y_k\right)}_i~N\left({\mu }_{{\left(y_k\right)}_i},{\Sigma }_k\right)$$ (11)

Note that this is identical to the original set up except each population has a different population covariance matrix ${\Sigma }_k$. In addition, we assume the assumption 3 holds for all covariance matrices ${\Sigma }_k$.

Assumption 7 (Heterogeneous rmt assumption) For $k=1,\cdots ,K$, the design matrix ${\mathbf{X}}_k\in {\mathbb{R}}^{n_k\times p}$ is generated as

$${\mathbf{X}}_k={\left({\mu }_{{\left(y_k\right)}_1,k}{\mu }_{{\left(y_k\right)}_2,k}\dots {\mu }_{{\left(y_k\right)}_{n_k},k}\right)}^{\top }+{\mathbf{Z}}_k{\Sigma }_k^{1/2}$$

for a matrix ${\mathbf{Z}}_k\in {\mathbb{R}}^{n_k\times p}$ with i.i.d. entries coming from an infinite array. The entries ${\left(Z_k\right)}_{ij}$ of ${\mathbf{Z}}_k$ satisfy the moment conditions:

$$\mathbb{E}\left[{\left(Z_k\right)}_{ij}\right]=0,\mathbb{E}\left[{\left(Z_k\right)}_{ij}^2\right]=1\mathit{\text{and}}\mathbb{E}\left[{\left(Z_k\right)}_{ij}^4\right]\le C.$$

1. The population covariance matrix ${\Sigma }_k\in {\mathbb{R}}^{p\times p}$ is deterministic. The observations have unit variance, i.e., ${\left({\Sigma }_k\right)}_{jj}=1$ for $j=1,\cdots ,p$.

2. The eigenvalues of ${\Sigma }_k$ are uniformly bounded from above and away from zero with constants independent of the dimension $p$.

3. The sequence of spectral distributions $T_k≔{\left(T_k\right)}_{\Sigma ,p}$ of ${\Sigma }_k≔{\left({\Sigma }_k\right)}_p$ converges weakly to a limiting distribution $H_k$ supported on $[0,\infty )$, called the population spectral distribution (psd).

We can then derive the optimal prediction and estimation weights in a manner identical to Theorems 4.5 and 4.10. Since the Bayes optimal discriminant direction is given by ${\Sigma }_K^{-1}{\delta }_K$, and as before we aim to leverage the related observations in each source through a weighted linear combination of their discriminant directions ${\widehat{d}}_k$, for $k=1,\dots ,K$.

Theorem 6.1 (Asymptotic Estimation Error Minimization for TLH-RDA) Suppose that assumptions 1, 2, 4, 5 as well as 6 and 7 hold. Then for a fixed $K\ge 2$, as $n_k,p\to \infty ,p/n_k\to {\gamma }_k\in (0,\infty ]$ for $1\le k\le K$, the weight for minimizing the error in estimating the Bayes optimal discriminator $d_{\mathit{\text{Bayes}}}$ is given by:

$${\mathbf{w}}_H^E≔\text{arg}\underset{\mathbf{w}}{\text{min}}{‖d_{\mathit{\text{Bayes}}}-\sum _{k=1}^K{w}_k{\widehat{d}}_k‖}_2^2={\left({𝒜}_H^E+{ℛ}_H^E\right)}^{-1}{\mathbf{u}}_H^E,$$

where the elements of ${\mathbf{u}}_H^E\in {\mathbb{R}}^K,{𝒜}_H^E\in {\mathbb{R}}^{K\times K}$, and ${ℛ}_H^E\in {\mathbb{R}}^{K\times K}$ are:

$${\left(u_H^E\right)}_k=\left\{\begin{array}{ll}{\rho }_{kK}{\alpha }_k{\alpha }_K\text{tr}\left({\Sigma }_K^{-1}{\left({\widehat{\Sigma }}_k+{\lambda }_k{\mathbb{I}}_p\right)}^{-1}\right)& \mathit{\text{if}}k=1,\dots ,K-1,\\ {\alpha }_K^2\left[\frac{1}{{\lambda }_k}\mathbb{E}\left(T_K^{-1}\right)-m_{F_{{\gamma }_k}}{\left(-{\lambda }_k\right)}^2\right]& \mathit{\text{if}}k=K.\end{array}\right.$$

$${\left({𝒜}_H^E\right)}_{k{k}^{\prime }}=\left\{\begin{array}{ll}{\alpha }_k^2{m}_{F_{{\gamma }_k}}^{\prime }\left(-{\lambda }_k\right)& \mathit{\text{if}}k=k^{\prime },\\ {\rho }_{k{k}^{\prime }}{\alpha }_k{\alpha }_{k^{\prime }}{𝒰}_{k{k}^{\prime }}& \mathit{\text{otherwise}},\end{array}\right.$$

and

$${\left({ℛ}_H^E\right)}_{k{k}^{\prime }}=\left\{\begin{array}{ll}\frac{v_{F_{{\gamma }_k}}\left(-{\lambda }_k\right)-{\lambda }_k{v}_{F_{{\gamma }_k}}^{\prime }\left(-{\lambda }_k\right)}{{\lambda }_k{v}_{F_{{\gamma }_k}}{\left(-{\lambda }_k\right)}^2}& \mathit{\text{if}}k=k^{\prime },\\ 0& \mathit{\text{otherwise}},\end{array}\right.$$

for $k,k^{\prime }=1,\cdots ,K$. Moreover for $k,k^{\prime }=1,\cdots ,K$, let ${𝒰}_{k{k}^{\prime }}$ be the constants defined as the following limiting quantities:

$$\text{tr}\left[{\left({\widehat{\Sigma }}_k+{\lambda }_k{\mathbb{I}}_p\right)}^{-1}{\left({\widehat{\Sigma }}_{k^{\prime }}+{\lambda }_{k^{\prime }}{\mathbb{I}}_p\right)}^{-1}\right]/p{\to }_{a.s.}{𝒰}_{k{k}^{\prime }}.$$ (12)

Since the quantities on the left hand side of (12) are exactly known in terms of sample quantities, we do not seek the exact limits of the traces of the cross sample covariance terms ${𝒰}_{k{k}^{\prime }}$. Instead, we advocate directly using the known quantities

$$\text{tr}\left[{\left({\widehat{\Sigma }}_k+{\lambda }_k{\mathbb{I}}_p\right)}^{-1}{\left({\widehat{\Sigma }}_{k^{\prime }}+{\lambda }_{k^{\prime }}{\mathbb{I}}_p\right)}^{-1}\right]/p.$$

The situation changes however for estimating ${\left(u_H^E\right)}_k$ when $k\ne K$, since they depend on the unknown target population covariance ${\Sigma }_K$. This is guaranteed by the almost sure convergence of a suitable sample based quantity, as recorded in the following proposition.

Proposition 6.2

$$\text{tr}\left({\Sigma }_K^{-1}{\left({\widehat{\Sigma }}_k+{\lambda }_k{\mathbb{I}}_p\right)}^{-1}\right)-\left(1-{\gamma }_K\right)\text{tr}\left({\widehat{\Sigma }}_K^{-1}{\left({\widehat{\Sigma }}_k+{\lambda }_k{\mathbb{I}}_p\right)}^{-1}\right){\to }_{a.s.}0$$

as $n_k,p\to 0;p/n_k\to {\gamma }_k$ for $k=1,\cdots ,K-1$.

The proof of this proposition is immediate from Corollary 4.2 of Dobriban and Sheng (2021). We next move on to the optimal prediction weights for TLH-RDA.

Theorem 6.3 (Asymptotic Prediction Error Minimization for TLH-RDA) Suppose that assumptions 1, 2 as well as 6 and 7 hold. Then for a fixed $K\ge 2$, as $n_k,p\to \infty ,p/n_k\to {\gamma }_k\in (0,\infty ]$ for $1\le k\le K$, the weight for minimizing the excess risk, i.e., the error in predicting the class at a random test point $x_0$, when compared to the Bayes optimal discriminator $d_{\mathit{\text{Bayes}}}$, is given by:

$${\mathbf{w}}_H^P≔\text{arg}\underset{\mathbf{w}}{\text{min}}{\mathbb{E}}_{x_0}{\left[{\left(d_{\mathit{\text{Bayes}}}-\sum _{k=1}^K{w}_k{\widehat{d}}_k\right)}^{\top }{x}_0\right]}^2={\left({𝒜}_H^P+{ℛ}_H^P\right)}^{-1}{\mathbf{u}}_H^P,$$

where the elements of ${\mathbf{u}}^P\in {\mathbb{R}}^K,{𝒜}^P\in {\mathbb{R}}^{K\times K}$, and ${ℛ}^P\in {\mathbb{R}}^{K\times K}$ are:

$${\left(u_H^P\right)}_k={\rho }_{kK}{\alpha }_k{\alpha }_K{m}_{F_{{\gamma }_k}}\left(-{\lambda }_k\right)\mathit{\text{for}}k=1,\dots ,K,$$

$${\left({𝒜}_H^P\right)}_{k{k}^{\prime }}=\left\{\begin{array}{ll}{\alpha }_k^2\left[\frac{v_{F_{{\gamma }_k}}\left(-{\lambda }_k\right)-{\lambda }_k{v}_{F_{{\gamma }_k}}^{\prime }\left(-{\lambda }_k\right)}{{\gamma }_k{\left[{\lambda }_k{v}_{F_{{\gamma }_k}}\left(-{\lambda }_k\right)\right]}^2}\right]& \mathit{\text{if}}k=k^{\prime }=K,\\ {\rho }_{k{k}^{\prime }}{\alpha }_k{\alpha }_{k^{\prime }}{𝒴}_{k{k}^{\prime }}& \mathit{\text{otherwise}},\end{array}\right.$$

and

$${\left({ℛ}_H^E\right)}_{k{k}^{\prime }}=\left\{\begin{array}{ll}\frac{v_k^{\prime }\left(-{\lambda }_k\right)-v_k^2\left(-{\lambda }_k\right)}{{\lambda }_k^2{v}_k^4\left(-{\lambda }_k\right)}& \mathit{\text{if}}k=k^{\prime },\\ 0& \mathit{\text{otherwise}},\end{array}\right.$$

for $k,k^{\prime }=1,\cdots ,K$. Moreover for $k,k^{\prime }=1,\cdots ,K$, let ${𝒴}_{k{k}^{\prime }}$ be the constants defined as the following limiting quantities:

$$\text{tr}\left[{\left({\widehat{\Sigma }}_k+{\lambda }_k{\mathbb{I}}_p\right)}^{-1}{\left({\widehat{\Sigma }}_{k^{\prime }}+{\lambda }_{k^{\prime }}{\mathbb{I}}_p\right)}^{-1}{\Sigma }_K\right]/p{\to }_{a.s.}{𝒴}_{k{k}^{\prime }}.$$

In order to utilize the above weights in a practical scenario, we need to replace all unknown quantities by their estimates. As before, the Marcenko Pastur law is crucial in estimating the parameters related to the spectral distributions of ${\Sigma }_k$, for $k=1,\dots ,K$. Most of the details are straightforward and identical in estimation procedure as the rest of the paper. It remains to provide consistent estimators for ${𝒴}_{k{k}^{\prime }}$, which we describe in the following proposition.

Proposition 6.4 We have the following consistent estimators ${\widehat{𝒴}}_{k{k}^{\prime }}$ for ${𝒴}_{k{k}^{\prime }}$ separately for three cases:

$${\widehat{𝒴}}_{k{k}^{\prime }}=\left\{\begin{array}{ll}\text{tr}\left[{\left({\widehat{\Sigma }}_k+{\lambda }_k{\mathbb{I}}_p\right)}^{-1}{\left({\widehat{\Sigma }}_{k^{\prime }}+{\lambda }_{k^{\prime }}{\mathbb{I}}_p\right)}^{-1}{\widehat{\Sigma }}_K\right]/p& \mathit{\text{if}}k\ne k^{\prime },k\ne K,k^{\prime }\ne K\\ \text{tr}\left[{\left({\widehat{\Sigma }}_k+{\lambda }_k{\mathbb{I}}_p\right)}^{-2}{\widehat{\Sigma }}_K\right]/p& \mathit{\text{if}}k=k^{\prime }\ne K\\ \frac{1}{p{x}_p}\text{tr}\left[{\left({\widehat{\Sigma }}_{k^{\prime }}+{\lambda }_{k^{\prime }}{\mathbb{I}}_p\right)}^{-1}\right]-\frac{{\lambda }_K}{p{x}_p}\text{tr}\left[{\left({\widehat{\Sigma }}_K+{\lambda }_K{\mathbb{I}}_p\right)}^{-1}{\left({\widehat{\Sigma }}_{k^{\prime }}+{\lambda }_{k^{\prime }}{\mathbb{I}}_p\right)}^{-1}\right]& \mathit{\text{if}}k=K,k^{\prime }\ne K\end{array}\right.$$

where $x_p=x\left({\gamma }_K,{\lambda }_K\right)$ is the solution to the equation:

$$1-x_p={\gamma }_K\left[1-{\lambda }_K\int {\left(x_{p}t+{\lambda }_K\right)}^{-1}d{H}_K(t)\right]$$

Then

$${𝒴}_{k{k}^{\prime }}-{\widehat{𝒴}}_{k{k}^{\prime }}{\to }_{a.s.}0.$$

7. Real Data

7.1. Proteomics-based Prediction of 10-year Cardiovascular Disease Risk

We utilize the data set from the Chronic Renal Insufficiency Cohort (CRIC) study to evaluate the performance of the proposed transfer learning methods. The dataset comprises observations from 2, 182 subjects who had not had any cardiovascular disease at the baseline, with 4, 830 protein measurements collected. The number of subjects are 272, 280, 333, 326, 409, 180, 382 respectively for seven university sites. Our goal is to build a classification model for 10-year cardiovascular disease risk based on the baseline plasma proteomic data. We hypothesize that while the mechanisms by which proteins predict events are related across these sites, they are not identical. Consequently, we sequentially designate each site as the target population, with the remaining sites serving as auxiliary populations, aiming to enhance prediction accuracy using TL-RDA or TLP-RDA.

For each target population, approximately 20% of the patients were set aside as a testing dataset, while the remaining 80% were used for training. We ensured that the proportion of events was consistent between the training and testing datasets (~7%). A univariate filtering procedure was applied to the training dataset using two-sample t-tests, and we evaluated the benefits of transfer learning using the top 500, 1000, and 1500 proteins. All predictors are centered and scaled within each sites, using training dataset information only. We considered all four variants of transfer learning RDA: TL-RDA with optimal estimation or prediction weights and TLP-RDA with optimal estimation/prediction weights. The “optimal weights” were estimated as instructions given by remark 4.7. The tuning parameters $\lambda$ for each model were selected independently by four-fold cross-validations over a length 9 logarithmically spaced grid ranges from 0.01 to 100. The domain specific penalization is obtained by ${\lambda }_k=\lambda {\gamma }_k/{\alpha }_k^2$. We refit each model on full training data with chosen $\lambda$ and test them on the held-out testing dataset. All transfer learning weights are estimated based on empirical estimators provided in the theorems.

The testing AUCs were then compared with those obtained from competing methods, namely naive RDA and pool-RDA. As implied by their names, naive RDA is fitted using only the target population data, while pool-RDA is fitted on data pooled from all populations. These procedures were repeated 25 times, and we report the average AUCs and standard deviations of the AUCs in Figure 4.

Figure 4:

AUC ± SD on testing data for 10-year cardiovascular disease risk for each of the 7 clinical sites in the CRIC cohort using 500, 1000 and 1500 proteins, respectively. TL-RDA-E, TL-RDA-P, TLP-RDA-E and TLP-RDA-P denote TL-RDA with optimal estimation/prediction weights and TLP-RDA with optimal estimation/prediction weights, respectively.

It is evident that the transfer learning RDA methods outperform the others in the vast majority of cases. This demonstrates that not only are TL-RDAs capable of borrowing information to improve prediction accuracy, but they also do so more efficiently than simply pooling the data. Notably, TLP-RDA with optimal prediction weights outperforms the other methods most frequently. The amount of improvement brought by transfer learning also seemingly increase as we include more proteins in the model. Additionally, we observe that methods based on pooled sample covariance tend to perform better when the number of features $p$ is larger, consistent with Proposition 5.2.

7.2. Lipid traits classification using genotype data

We also evaluate the proposed transfer-learning estimators using a data set from Penn Medicine BioBank, which contains both genome-wide genotype data and the electronic health record–derived phenotypes. We focus on three lipid traits with substantial genetic signal, high-density lipoprotein cholesterol (HDL), low-density lipoprotein cholesterol (LDL), and triglycerides (TRI), and dichotomize each outcome using guideline-based clinical cutoffs (Grundy et al., 2019, 2005; NCEP, 2002). To emulate settings where related auxiliary traits are available, we analyze each trait in turn as the target and use the remaining two traits as sources. The analytic cohort contains approximately 12000 individuals, partitioned into three trait-specific datasets. For each target analysis, feature screening is performed solely on the target training split via univariate association tests, and the top 1000, 2000, and 3000 nucleotide polymorphism (SNPs) (ranked by significance) are retained as the predictors.

All regularization hyperparameters were tuned exactly as in the proteomics experiment, using the same grid of candidate values. Figure 5 reports averaged AUCs and their standard deviations on held-out test sets for the proposed methods and the baselines. The statistics are based on 25 independent train / test splits and cross-validations. Across most configurations, Trans-Ridge improves upon target-only ridge regression. In contrast to the proteomics based prediction across different clinical sites in previous Section, the TL-RDA methods outperform the TLP-RDA variants based on pooled sample-covariance estimators. This ordering is again consistent with Proposition 5.2, which predicts an advantage for TL-RDA in the regime of small ${\gamma }_k$, the regime relevant to the present data.

Figure 5:

AUC ± SD on testing data for HDL/LDL/Trig using 1000, 2000 and 3000 SNPs, respectively. TL-RDA-E, TL-RDA-P, TLP-RDA-E and TLP-RDA-P denote TL-RDA with optimal estimation/prediction weights and TLP-RDA with optimal estimation/prediction weights, respectively.

8. Discussion

We have developed methods of transfer learning for classification using regularized random effects linear discriminant analysis. In this approach, the discriminant direction is estimated through a weighted combination of the regularized estimates of discriminant directions derived from both the target and source models. By leveraging results from random matrix theory, we have demonstrated that the weights, as well as the classification error rate, can be accurately estimated using the data. Our findings highlight that the optimal choice of weights depends critically on the underlying true models and the presence or absence of distributional shifts between the testing and training data in the target model.

Through comprehensive empirical evaluations, we have illustrated the practical utility of the proposed methods. Specifically, we applied the transfer learning approach to predict 10-year cardiovascular disease risk using high-dimensional protein expression data and to the lipid trait prediction problem with genotype data. The results demonstrate that incorporating information from related source datasets substantially improved classification performance compared to target-only models or models based on pooled data. This improvement underscores the importance of accounting for shared information across datasets while accommodating potential distributional differences. In addition, the theoretical guidance for selecting among the four candidate weighting schemes is supported by the empirical evidence. In the high-dimensional proteomics application, pooled-covariance weighting performs the best, consistent with its variance-reduction advantage when signals are diffuse. For the genotype data, where target and source data includes correlated but different outcomes, weights optimized for estimation typically outperform those optimized for prediction. When the requisite assumptions are uncertain, we recommend selecting among the four schemes using cross-validation.

In terms of computational scalability, TL-RDA has the same leading complexity as running the $K$ source RDA models. To compute RDA directions with naive algorithms, forming the covariances costs $O\left(K{n}_k{p}^2\right)$ and the $K+1$ matrix inversions cost $O\left(K{p}^3\right)$. Solving and estimating the optimal weights are equivalent to at most two linear systems and one eigen-problem of size $K\times K$. This $O\left(K^3\right)$ overhead is negligible because $K\ll p$ in our applications. In addition, all heavy computations are performed locally within each domain. Each domain returns only its RDA direction and a few scalar trace statistics required by Theorem 4.5 and Theorem 4.10. The central node only needs to combine these $O\left(Kp\right)$ and $O\left(K\right)$ summaries to obtain the weight vector. Moreover, because each inversion has the ridge form ${\left({\widehat{\Sigma }}_k+\lambda I_p\right)}^{-1}$, we can apply Woodbury and perform a Cholesky factorization on an $n_k\times n_k$ system, thereby eliminating the cubic dependence on $p$ and reducing per-domain cost to $O\left(n_k^3+n_{k}p\right)$ when $n_k\ll p$.

This paper focuses on two-class LDA models. A promising direction for future research is extending these methods to multi-class classification within the framework of transfer learning. Other interesting avenues include exploring this problem under privacy or communication constraints, as has been recently studied for other classification methods, or analyzing LDA trained with mixed in-distribution and out-of-distribution samples (see, e.g., Auddy et al., 2024; De Silva et al., 2023). Also, we have assumed the number of source domains $K$ is small compared to number of observations and predictors ${(n}_k,p)$. If instead $K$ is very large, source selection becomes necessary. As noted in Section 4.4, the optimal weight vectors solves a least-squares problem in which the Bayes discriminant direction is regressed on the collection of domain-specific discriminant directions. When the number of sources $K$ grows to the same order as $n$ and $p$, this becomes a high-dimensional regression and variable-selection problem. A simple heuristic is to retain only those sources whose discriminant directions have the largest absolute correlations with the target estimate; however, this ignores cross-source dependencies and may discard complementary information. A natural extension is to add a ridge-type weight penalization $\Vert \mathbf{w}{\Vert }_2^2$ on criterion 3 or criterion 4. The resulting estimator can still be computed in closed form via the machinery of Theorems 4.5 and Theorems 4.10. Developing principled, sparsity-inducing penalties or screening rules that account for inter-source correlations and analyzing their theoretical guarantees, remains an interesting direction for future research.