Partial least squares regression and principal component analysis: similarity and differences between two popular variable reduction approaches

Abstract

In many statistical applications, composite variables are constructed to reduce the number of variables and improve the performances of statistical analyses of these variables, especially when some of the variables are highly correlated. Principal component analysis (PCA) and factor analysis (FA) are generally used for such purposes. If the variables are used as explanatory or independent variables in linear regression analysis, partial least squares (PLS) regression is a better alternative. Unlike PCA and FA, PLS creates composite variables by also taking into account the response, or dependent variable, so that they have higher correlations with the response than composites from their PCA and FA counterparts. In this report, we provide an introduction to this useful approach and illustrate it with data from a real study.

Introduction

Composite variables are widely used to summarise information from a set of outcomes in statistical analysis. In some studies, composite variables are used to create domain scales or subscales, such as the SF-36 (the MOS item short-form health survey) instrument, while in some other studies, composite variables are used to deal with limitations of data. For example, in regression analysis, we may need to create composite variables if the number of explanatory or independent variables is larger than the sample size. This statistical issue arises when modelling high-throughput data such as in fitting regression models to determine associations of brain functions with behavioural and health outcomes of interest due to large numbers of braining imaging variables and limited study sample sizes in most studies. Principal component analysis (PCA) and factor analysis (FA) are generally used for creating composite variables. In this report, we describe another less-known approach called partial least squares (PLS) regression, to create composite variables and discuss scenarios where this approach is more effective than PCA and FA. We illustrate this approach with a real-life application to research data.

Partial least squares regression

As noted earlier, PCA and FA are two popular approaches for creating composite variables. Under PCA, a set of ordered composite variables are created to represent the original set of outcomes. Each composite variable is a linear combination, or a weighted sum, of the original outcomes. The coefficients, or weights, of the linear combination in each compositive variable, are called loadings, and their signs and magnitudes indicate the directions and contributions of the corresponding variables. Unlike the original outcomes, the composite variables are orthogonal to each other. Moreover, the first composite variable has the largest variance, followed by the second and so on. FA also creates a set of composite variables. However, unlike PCA, FA composite variables are not ordered in the sense of PCA composites and are not orthogonal to each other. Instead, loadings of the FA composites can be used to group the original variables to create subscale, or domain scales, for different constructs such as the domains of Physical Functioning and Emotional Well-being in the SF-36.¹

PCA and FA create composite variables for general purposes. When composite variables are used as explanatory or independent variables in regression analysis involving a response or dependent variable, a more effective approach is PLS. Like PCA, PLS composite variables are also ordered. However, unlike PCA, PLS composite variables are ordered by their correlations with the response in the regression model; the first composite variable has the maximum correlation with the response, followed by the second and so on. If interest lies in finding a subset of the original explanatory variables in the linear model that explains the most variability in the response, PLS composite variables are more effective than PCA.

To describe in detail how to compute PLS composite variables, consider a linear regression with a continuous response of interest, $Y$ , and a set of $p$ explanatory variables, $X_1,X_2,\dots ,X_p$ . We are interested in modelling the relationship of $Y$ with $X_1,X_2,\dots ,X_p$ . Given a sample of $n$ subjects, the classic linear regression relating $Y$ to $X_1,X_2,\dots ,X_p$ is given by:

$${\displaystyle \begin{array}{ll}{Y}_i& ={\beta }_0+{\beta }_1{X}_{i1}+{\beta }_2{X}_{i2}+\cdots +{\beta }_p{X}_{ip}+{\epsilon }_i,{\epsilon }_i\sim N\left(0,{\sigma }^2\right),\\ & 1\le i\le n,\end{array}}$$ (1)

where $i$ indexes the subjects, ${\beta }_0,{\beta }_1,{\beta }_2,\dots ,{\beta }_p$ denote the regression parameters, ${\epsilon }_i$ is the error term, and $N(\mu ,{\sigma }^2)$ denotes a normal distribution with mean $\mu$ and variance ${\sigma }^2$ . The first part of the linear regression,

$${\displaystyle {\beta }_0+{\beta }_1{X}_{i1}+{\beta }_2{X}_{i2}+\cdots +{\beta }_p{X}_{ip},1\le i\le n,}$$ (2)

is called the conditional (population) mean of $Y_i$ given the explanatory variables $X_{i1},X_{i2},\dots ,X_{ip}$ . On estimating the regression parameters ${\beta }_0,{\beta }_1,{\beta }_2,\cdots ,{\beta }_p$ , this conditional mean describes the association of $Y_i$ with each of the explanatory variables.

When the explanatory variables $X_{i1},X_{i2},\dots ,X_{ip}$ are highly correlated, estimates of ${\beta }_0,{\beta }_1,{\beta }_2,\cdots ,{\beta }_p$ may not be reliable due to multicollinearity using the standard least squares (LS) or maximum likelihood (ML) method. In studies of high-throughput data, the number of explanatory variables exceeds sample sizes, in which case LS method will not apply. In both cases, we need to reduce the number of the variables, $X_{i1},X_{i2},\dots ,X_{ip}$ , or the dimension $p$ . There are different approaches to address this issue. One may use the least absolute shrinkage and selection operator (LASSO) to determine a subset of $X_{i1},X_{i2},\dots ,X_{ip}$ that provides reliable associations with $Y_i$ . Alternatively, one may create composite variables $Z_{i1},Z_{i2},\dots ,Z_{ip}$ from $X_{i1},X_{i2},\dots ,X_{ip}$ and use a subset of the composite variables to predict $Y_i$ . The latter composite variable approach is preferred if some or all $X_{i1},X_{i2},\dots ,X_{ip}$ need to work together to explain the variability in $Y_i$ . For example, if one wants to predict areas of rectangles, lengths or widths alone will not provide reliable predictions since a rectangle with a very large length can still have a small area if it has a small width. LASSO is most effective to deal with high-throughput data as dimension is the primary problem in this case. In the presence of multicollinearity, it is likely more meaningful to aggregate information in correlated variables using a subset of composite variables, rather than to select a subset of the original variables. In this case, correlated variables may all contribute to explaining the variability in the response $Y_i$ and composite variables will account for all such contributions.

The composite variables $Z_{i1},Z_{i2},\dots ,Z_{ip}$ for PLS are obtained by solving an optimisation problem.² Unlike PCA composite variables, PLS finds directions of the composite variables $Z_{i1},Z_{i2},\dots ,Z_{ip}$ that have both high variance and high correlation with the response $Y_i$ . Specifically, let $Z_{il}$ denote the $l$ th composite variable:

$${\displaystyle Z_{il}=X_{i1}{\alpha }_{l1}+X_{i2}{\alpha }_{l2}+\cdots +X_{ip}{\alpha }_{lp},1\le i\le n,1\le l\le p,}$$ (3)

where ${\alpha }_1,{\alpha }_2,\dots ,{\alpha }_p$ denote weights, or loadings, of the composite $Z_l$ . We can also express (3) equivalently in a vector form

$${\displaystyle {\stackrel{\sim }{Z}}_l={\stackrel{\sim }{X}}_1{\alpha }_{l1}+{\stackrel{\sim }{X}}_2{\alpha }_{l2}+\cdots +{\stackrel{\sim }{X}}_p{\alpha }_{lp},1\le i\le n,1\le l\le p,}$$ (4)

or in a matrix form:

$${\displaystyle {\stackrel{\sim }{Z}}_l=\mathit{X}{\stackrel{\sim }{\alpha }}_l,1\le l\le p,}$$

where ${\tilde{Z}}_l={\left(Z_{1l},Z_{2l},\cdots ,Z_{nl}\right)}^{{\rm T}}$ , ${\tilde{X}}_l={\left(X_{1l},X_{2l},\cdots ,X_{nl}\right)}^{{\rm T}}$ and $\tilde{\alpha }={\left({\alpha }_1,{\alpha }_2,\cdots ,{\alpha }_p\right)}^{{\rm T}}$ denote $p\times 1$ column vectors, and $\mathit{X}=\left({\tilde{X}}_1,{\tilde{X}}_2,\cdots ,{\tilde{X}}_n\right)$ denotes an $n\times p$ matrix. The loadings are determined by the following optimisation procedure:

$${\mathit{max}}_{{\alpha }_l}\hspace{0.33em}Cor{r}^2\left(\tilde{Y},\mathit{X}{\tilde{\alpha }}_l\right)Var\left(\mathit{X}{\tilde{\alpha }}_l\right)$$

$${\displaystyle \mathrm{s}\mathrm{u}\mathrm{b}\mathrm{j}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{t}\mathrm{o}:||{\tilde{\alpha }}_l||=\sqrt{{\alpha }_{l1}^2+{\alpha }_{l2}^2+\cdots +{\alpha }_{lp}^2}=1,{\tilde{\alpha }}^T{}_{l}S{\tilde{\alpha }}_m=0,m=1,\cdots ,l-1.}$$

where $\tilde{Y}={\left(Y_1,Y_2,\cdots ,Y_n\right)}^{{\rm T}}$ is a $n\times 1$ column vector, $S$ is the sample covariance matrix of $X_{i1},X_{i2},\dots ,X_{ip}$ , $Cor{r}^2\left(\tilde{Y},X{\tilde{\alpha }}_l\right)$ denotes the squared (Pearson) correlation matrix between $\tilde{Y}$ and $X{\tilde{\alpha }}_l$ , and $Var\left(\mathit{X}{\tilde{\alpha }}_l\right)$ denotes the sample variance of $\mathit{X}{\tilde{\alpha }}_l$ . The condition ${{\tilde{\alpha }}^{{\rm T}}}_{l}S{\tilde{\alpha }}_m=0$ ensures that the lth composite ${\tilde{Z}}_l=\mathit{X}{\tilde{\alpha }}_l$ is uncorrelated with all previous composite variables ${\tilde{Z}}_m=\mathit{X}{\tilde{\alpha }}_m$ ( $1\le m\S amp;lt;l\le p$ ).

In practice, we can use the following procedure to find the PLS composite variables.³ We start by standardising each of the original explanatory variables $X_{i1},X_{i2},\dots ,X_{ip}$ to have mean 0 and variance 1. Set ${\tilde{Y}}^{\left(0\right)}={\overline{Y}}_{·}{1}_n$ and ${\tilde{X}}_l^{\left(0\right)}={\tilde{X}}_l$ ( $1\le l\le p$ ), where ${\overline{Y}}_{·}=\frac{1}{n}{\sum }_{i=1}^n\left(Y_1+Y_2+\cdots +Y_n\right)$ denotes the sample mean of $Y_i$ ( $1\le i\le n$ ) and ${\displaystyle {\stackrel{\sim }{\text{1}}}_n={\left(1,1,\cdots ,1\right)}^T}$ denotes a $n\times 1$ volume vector of 1. For $1\le l\le p$ , we perform the following steps:

(a) ${\tilde{Z}}_l={\tilde{X}}_1^{\left(l-1\right)}{\alpha }_{l1}+{\tilde{X}}_2^{\left(l-1\right)}{\alpha }_{l2}+\cdots +{\tilde{X}}_p^{\left(l-1\right)}{\alpha }_{lp}$ , where ${\alpha }_{lj}=⟨{\tilde{X}}_j^{(l-1)},\tilde{Y}⟩$ , where $\langle a,b\rangle$ denotes the inner product between two vectors $a$ and $b$ ;

(b) ${\displaystyle {\theta }_l=\frac{⟨{\stackrel{\sim }{Z}}_l,\stackrel{\sim }{Y}⟩}{⟨{\stackrel{\sim }{Z}}_l,{\stackrel{\sim }{Z}}_l⟩}}$ ;

(c) ${\displaystyle {\stackrel{\sim }{Y}}^{\left(l\right)}={\stackrel{\sim }{Y}}^{\left(l-1\right)}+{\theta }_l{\stackrel{\sim }{Z}}_l}$ ;

(d) Orthogonalise each ${\tilde{X}}_j^{\left(l-1\right)}$ with respect to ${\tilde{Z}}_l$ :

$${\displaystyle {\stackrel{\sim }{X}}_j^{\left(l\right)}={\stackrel{\sim }{X}}_j^{\left(l-1\right)}-\frac{⟨{\stackrel{\sim }{Z}}_l,{\stackrel{\sim }{X}}_j^{\left(l-1\right)}⟩}{⟨{\stackrel{\sim }{Z}}_l,{\stackrel{\sim }{Z}}_l⟩}{\stackrel{\sim }{Z}}_l,1\le j\le p.}$$

To illustrate the difference between PSL and PCA, here is the procedure to compute composite variables under PCA:

We start with the $n\times p$ data matrix $X$ , which is formed by the column vectors $X_1$ , $X_2$ ,…, $X_p$ , that is, $X=[X_1,X_2,\dots ,X_k,\dots ,X_p]$ . Then we perform the following steps:

1. Average over all the columns of $X$ : $\overline{X}=\frac{1}{p}{\displaystyle \sum _{k=1}^p{X}_k}$ ;

2. Centre the matrix $X$ at this average $\overline{X}$ by subtracting $\overline{X}$ from each column vector $X_k$ of $X$ , denote as: $Z=\left[X_1-\overline{X},X_2-\overline{X},\dots ,X_p-\overline{X}\right]$ ;

3. Compute the sample variance–covariance matrix: $\mathrm{\Sigma }=Z^{T}Z$ ;

4. Compute the eigenvalues ${\left\{{\lambda }_k\right\}}_{k=1,2\dots ,p}$ and corresponding eigenvectors ${\left\{U_k\right\}}_{k=1,2\dots ,p}$ of $\Sigma$ with ${\lambda }_1\S amp;gt;{\lambda }_2\S amp;gt;\dots \S amp;gt;{\lambda }_p$ ;

5. The top $m$ ( $\le p$ ) principal components, or composite variables, ${\left\{U_k\right\}}_{k=1,2\dots ,m}$ , are then used as independent variables in linear regression models with $Y$ as the dependent variable, where $m$ is generally determined by the magnitude of the sum of the top $m$ eigenvalues relative to the sum of all $p$ eigen values, $\frac{{\displaystyle \sum _{k=1}^m{\lambda }_k}}{{\displaystyle \sum _{l=1}^p{\lambda }_l}}$ , which has the interpretation of being the percent of the variability of $X$ explained by the top $m$ eigenvectors ${\left\{U_k\right\}}_{k=1,2\dots ,m}$ .

By comparing the two procedures, we can see that PCA creates composite variables without using any information in the dependent variable $Y$ as PLS does in creating its composite variables. If the goal is to find composite variables of $X$ that are most predictive of $Y$ , PLS is more preferrable than PCA. On the other hand, if the goal is to find composite variables that maximally explain the variability of the data matrix $X$ , then PCA is more preferrable.

Real study example

We illustrate PLS with data from a recent study on the association of loneliness and wisdom with gut microbial diversity and composition.⁴ Loneliness and wisdom have opposite effects on health and well-being. Loneliness is a serious public health problem associated with increased morbidity and mortality. Wisdom is associated with better health and well-being. Nguyen et al ⁴ successfully applied PLS to demonstrate relationships between the association of loneliness and wisdom with alpha-diversity. We use this study to illustrate the advantages of PLS over standard linear regression. More details about the study population, measures of loneliness, wisdom, gut microbial diversity and other outcomes, and additional findings can be found in the paper.

The study included 184 community-dwelling adults (28–97 years). Participants completed validated scales of loneliness (UCLA Loneliness Scale),⁵ wisdom (including cognitive, affective and reflective dimensions; Three-Dimensional Wisdom Scale),⁶ compassion (Santa Clara Brief Compassion Scale),⁷ social support (Emotional Support Scale)⁸ and social engagement (Cognitively Stimulating Questionnaire).⁹ These variables are interrelated; loneliness and wisdom have strong inverse correlations; social support, social engagement and loneliness are often inversely correlated, but they are distinct concepts. Faecal samples were obtained from participants using at-home self-collection kits and returned via mail. Alpha- diversity is the ecological diversity (ie, richness, evenness, compositional complexity) of a single sample and was quantified using Faith’s Phylogenetic Diversity (PD) based on the DNAs extracted from the faecal samples. It measures the total length of branches in a reference phylogenetic tree for all species in a given sample.¹⁰

We first fit a standard linear regression to model the association of alpha-diversity with individual loneliness, wisdom, compassion, social support and social engagement outcomes as predictors, controlling for age and body mass index (BMI). Shown in table 1 were estimated regression coefficients (β) for the predictors and covariates, along with associated t-statistics (t) and p values. As seen, none of the predictors were significant.

Table 1.

Results from linear regression for association of alpha-diversity (Faith’s Phylogenetic Diversity) with loneliness and wisdom outcomes, controlling for covariates

Predictors/covariates	β	t	P value
Intercept	11.492	1.52	0.131
Loneliness	0.022	0.40	0.686
Social support	1.314	1.37	0.172
Wisdom components
Affective	0.869	0.73	0.473
Cognitive	−1.164	−1.10	0.266
Reflective	1.350	1.03	0.302
Compassion	0.270	0.67	0.499
Social engagement	0.312	0.43	0.671
Age	−0.013	−0.45	0.652
BMI	−0.111	−1.37	0.168

BMI, body mass index.

We then applied PLS to construct composite variables from all the predictors and included the extracted composite variables and the covariates to build the linear regression to predict alpha-diversity by examining the contribution of each composite component added in terms of the amount of explained variability in the outcome of alpha-diversity.¹¹ We settled on the first two composite variables because adding component 3 led to a decreased adjusted R squared. Shown in table 2 are estimated regression coefficients (β) for the predictors and covariates, along with t-statistics (t) and p values. The model revealed that the effect of component 1 was significantly positively associated with alpha-diversity (p=0.008), whereas component 2 was not (p=0.217).

Table 2.

Coefficients from linear regression model of partial least squares (PLS) composite variables predicting alpha-diversity (Faith’s Phylogenetic Diversity), controlling for age and BMI

	β	t	P value
Intercept	21.911	8.52	<0.001
Component 1	0.717	2.71	0.008
Component 2	0.545	1.24	0.217
Age	−0.015	−0.57	0.569
BMI	−0.103	−1.32	0.188

BMI, body mass index.

When applying PLS, it is important to determine directions of effects for the original predictors of interest (loneliness, wisdom, compassion, social support and social engagement) when a composite variable is used as a predictor in the final model. Shown in table 3 were loadings of individual predictors on the first two composite variables. The sign of the loading of a predictor on the composite variable indicates the direction of association of the predictor with the composite variable. Except for loneliness, all the predictors had positive loadings on the first composite variable, indicating that wisdom, compassion, social support and social engagement had positive associations with alpha-diversity. Loneliness had a negative association with alpha-diversity because of its negative sign. The first composite variable also accounted for 40% of the total variability of the psychosocial variables.

Table 3.

Loadings for PLS composite variables

	Composite variable 1	Composite variable 2
Loneliness	−0.419	0.272
Wisdom-cognitive	0.233	−0.836
Wisdom-reflective	0.462	−0.370
Wisdom-affective	0.454	−0.140
Compassion	0.417	0.421
Social support	0.316	0.187
Social engagement	0.358	0.233

To illustrate the differences between PLS and PCA, we also applied PCA to construct composite variables and use them as explanatory variables in modelling the association of alpha-diversity with the psychosocial variables. To be consistent with the PLS, we used the first two eigenvectors as the composite variables and controlled for age and BMI. Shown in table 4 were estimated regression coefficients (β) for the predictors and covariates, along with t-statistics (t) and p values (component 1: p=0.015; component 2: p=0.190). The results were similar to their PLS counterparts. A notable difference is the slightly weaker association between the first composite variable and alpha-diversity. Both PCA and PLS yielded the same conclusion regarding the association of composite variables with alpha-diversity.

Table 4.

Coefficients from linear regression model of principal component analysis (PCA) composite variables predicting alpha-diversity (Faith’s Phylogenetic Diversity), controlling for age and BMI

	β	t	P value
Intercept	21.940	8.48	<0.001
Component 1	0.639	2.47	0.015
Component 2	0.500	1.32	0.190
Age	−0.021	−0.78	0.439
BMI	−0.092	−1.18	0.240

BMI, body mass index.

Shown in table 5 were loadings of individual predictors on the first two PCA composite variables. The signs of the loadings are consistent with their PLS counterparts. The wisdom-cognitive subscore had less loading under PLS than PCA, while compassion, social support and social engagement had higher loadings under PLS than PCA.

Table 5.

Loadings for the first PCA composite variables

	Composite variable 1	Composite variable 2
Loneliness	−0.423	0.071
Wisdom-cognitive	0.317	−0.601
Wisdom-reflective	0.483	−0.253
Wisdom-affective	0.442	−0.037
Compassion	0.342	0.528
Social support	0.280	0.058
Social engagement	0.303	0.536

Discussion

In this report, we described the partial least squares (PLS) regression, discussed its relationship with a closely related alternative, the principal component analysis (PCA), and illustrated the PLS with a real study example. Although both aim to reduce explanatory variables (predictors), PLS and PCA work quite differently in developing composite variables. While PCA constructs the composite variables to explain the maximum variability in all the original predictors, or the explanatory variables of interest, PLS creates its composite variables to explain the maximum variability in the response within the context of linear regression.

In practice, if the goal is to develop a set of composite variables for use as explanatory variables in regression models for multiple responses, PCA may be preferred since, unlike PLS, it will create a common set of composite variables for regression across all the responses. On the other hand, if the objective is to develop a set of composite variables to explain the maximum variability for a given response, then PLS should be used. When applying PLS to develop composite variables for regression analysis for multiple responses, multiple sets of composite variables will be created with one set for each response and consequently, regression results from composite variables must be interpreted with respect to factor loadings within each set of composite variables.

In the illustrative example, the two approaches yield similar results. In general, results from the two approaches may differ and yield different conclusions. For example, PLS may yield significant associations of its composite variables while PCA does not. If interest lies in finding associations of a response with a set of explanatory variables, PLS should be used.

Biography

Chenyu Liu is a PhD student in Division of Biostatistics and Bioinformatics, Herbert Wertheim School of Public Health and Human Longevity Science, UC San Diego in USA. She is currently working as a Graduate Student Researcher at UC San Diego. She got her master’s degree in statistics from University of Minnesota, Twin Cities in USA. Her main research interests include statistical learning, statistical methods, clinical trial and causal inference.

Ethics statements

Patient consent for publication

Not applicable.

Ethics approval

The studies involving human participants were reviewed and approved by UCSD Human Research Protections Program. The patients/participants provided their written informed consent to participate in this study. Participants gave informed consent to participate in the study before taking part.