Residualization: justification, properties and application

Abstract

Although it is usual to find collinearity in econometric models, it is commonly disregarded. An extended solution is to eliminate the variable causing the problem but, in some cases, this decision can affect the goal of the research. Alternatively, residualization not only allows mitigation of collinearity, but it also provides an alternative interpretation of the coefficients isolating the effect of the residualized variable. This paper fully develops the residualization procedure and justifies its application not only for dealing with multicollinearity but also for separating the individual effects of the regressor variables. This contribution is illustrated by two econometric models with financial and ecological data, although it can also be extended to many different fields.

1. Introduction

Explanatory variables of an econometric model can present strong collinearity and consequently the variance of the ordinary least squares (OLS) estimators may be large compared to the values of the estimated parameters that can be insignificant or have the wrong sign. Even when collinearity diagnostic measures consider that the collinearity is not of concern, it is possible that the individual effects of the variables may not be separated or displayed clearly. This idea resembles the objective of the Shapley value regression [57], which presents an entirely different strategy for assessing the contribution of regressor variables to the dependent variable. It owes its origin to the theory of cooperative games. The value of $R^2$ obtained by fitting a linear regression model is regarded as the value of a cooperative game played by the independent variables (each variable is a member) against the dependent variable (explaining it). The analyst does not have sufficient information to disentangle the contributions made by the individual members, only their joint contribution $R^2$ is known. The Shapley value decomposition imputes the most likely contribution of each individual member. On the other hand, [3] proposed an alternative methodology to OLS based on ordered variable regression (OVR), originally presented by [64], which entirely resolves the issue of related predictors by creating and using predictors that are perfectly unrelated.

These antecedents lead to residualization, which is a procedure applied in numerous research articles published in relevant social science journals in many different fields, such as linguistics [1,8,26,34–36], environmental issues [28–30] or economic development and policies [4,6,32,38,62], for example. This method has been also applied in previous research under the name of regression with orthogonal variables (see [44,53]). However, this method has not been fully developed in prior works and we consider that this lack of specification can lead to different criticisms such as the one in [66]. [65] also concluded that ‘residualization of predictor variables is not the hoped-for panacea [to collinearity]’. We consider that the key point not taken into consideration until now is that this methodology provides an alternative interpretation for the estimated parameters, apart from the mitigation of collinearity. This could be seen as a limitation since the methodology is not always applicable but it can be also seen as an opportunity to obtain new interpretations which are not possible from the initial model. To briefly explain the general concept of the method, it might be said that by residualizing one of the explanatory variables, its effect is being isolated from the rest of the variables of the model. Thus the part of this variable that has no relationship with the rest of independent variables is being included in the model, leading to a new interpretation of the residualized variable.

This paper fully develops the residualization procedure and justifies its application not only for dealing with multicollinearity but also for separating the individual effects of the regressor variables. Main properties and inference are also presented together with the variance inflation factor (VIF) and the condition number (CN), which allow us to check whether the collinearity has been mitigated after the application of residualization.

The structure of this paper is as follows: Section 2 presents the estimation and main properties of residualization procedure showing that the estimation of the variance of the random disturbance, the global significance test, the individual significance test of the residualized variable and the goodness of fit will be similar to that of the original model. Section 3 analyzes how residualization mitigates collinearity. Section 4 compares residualization procedure with OLS and other well-known techniques, such as ridge regression, principal component regression (PCR) or partial least squares regression (PLSR). Section 5 presents the successive residualization. Finally, Section 6 illustrates the contribution of this paper with two econometric models: the first one shows the application of the method when the main goal of the researcher is to mitigate collinearity, and the second shows the application of the method when the purpose of the study is to obtain new interpretations of the variables. The two empirical examples belong to different fields: the first is a financial model, while the second is usually applied in ecological studies. The main conclusions are summarized in Section 7.

2. Estimation and properties

Consider the following general linear regression model for p exogenous variables and n observations:

$$\mathbf{Y}=\mathbf{X}\mathrm{\text{β}}+\mathbf{u},$$ (1)

where the first column of $\mathbf{X}$ corresponds to the independent term and the random disturbance, $\mathbf{u}$, is spherical.

The first step is to define the following auxiliary regression:

$${\mathbf{X}}_i={\mathbf{X}}_{-i}\mathit{\alpha }+\mathbf{v},i=2,\dots ,p,$$ (2)

where ${\mathbf{X}}_{-i}$ is the result obtained after eliminating column (variable) i from matrix $\mathbf{X}$ and ${\mathbf{X}}_i$ represent the variable i. That is, $\mathbf{X}=\left({\mathbf{X}}_{-i}{\mathbf{X}}_i\right)$.

By OLS estimation of (2), it will be obtained the correspondent estimated residuals, ${\mathbf{e}}_i$. They will represent the part of variable i that has no relation with any other exogenous variable of model (1) since the residuals ${\mathbf{e}}_i$ are orthogonal to ${\mathbf{X}}_{-i}$ (that is, ${\mathbf{e}}_i^t{\mathbf{X}}_{-i}=\mathbf{0}$, with $\mathbf{0}$ being a vector of zeros with appropriate dimensions).

Taking into account the previous, residualization procedure involves replacing the variable ${\mathbf{X}}_i$ by the estimated residuals of model (2), ${\mathbf{e}}_i$, in the original model (1). Thus residualization procedure is obtained from the following regression:

$$\mathbf{Y}={\mathbf{X}}_O\mathit{\gamma }+\mathbf{w},$$ (3)

where ${\mathbf{X}}_O=\left({\mathbf{X}}_{-i}{\mathbf{e}}_i\right)$.

Once the basic procedure is explained, the results of model (1) and model (3) are compared.

2.1. Estimation

From $\mathbf{X}=\left({\mathbf{X}}_{-i}{\mathbf{X}}_i\right)$, the OLS estimator for model (1), $\widehat{\mathrm{\text{β}}}$, will be

$$\begin{array}{rl}\widehat{\mathrm{\text{β}}}& ={\left({\mathbf{X}}^t\mathbf{X}\right)}^{-1}{\mathbf{X}}^t\mathbf{Y}={\left(\begin{array}{cc}f{\mathbf{X}}_{-i}^t{\mathbf{X}}_{-i}& {\mathbf{X}}_{-i}^t{\mathbf{X}}_i\\ {\mathbf{X}}_i^t{\mathbf{X}}_{-i}& {\mathbf{X}}_i^t{\mathbf{X}}_i\end{array}\right)}^{-1}\cdot \left(\begin{array}{c}{\mathbf{X}}_{-i}^t\mathbf{Y}\\ {\mathbf{X}}_i^t\mathbf{Y}\end{array}\right)=\left(\begin{array}{cc}\mathbf{A}& \mathbf{B}\\ {\mathbf{B}}^t& C\end{array}\right)\cdot \left(\begin{array}{c}{\mathbf{X}}_{-i}^t\mathbf{Y}\\ {\mathbf{X}}_i^t\mathbf{Y}\end{array}\right)\\ & =\left(\begin{array}{c}{\left({\mathbf{X}}_{-i}^t{\mathbf{X}}_{-i}\right)}^{-1}{\mathbf{X}}_{-i}^t\mathbf{Y}-\hat{\mathit{\alpha }}\cdot \frac{{\mathbf{e}}_i^t\mathbf{Y}}{{\mathbf{e}}_i^t{\mathbf{e}}_i}\\ \frac{{\mathbf{e}}_i^t\mathbf{Y}}{{\mathbf{e}}_i^t{\mathbf{e}}_i}\end{array}\right)=\left(\begin{array}{c}{\widehat{\mathrm{\text{β}}}}_{-i}\\ {\widehat{\mathrm{\text{β}}}}_i\end{array}\right),\end{array}$$ (4)

taking into account that

$$\begin{array}{rl}C& ={\left({\mathbf{X}}_i^t{\mathbf{X}}_i-{\mathbf{X}}_i^t{\mathbf{X}}_{-i}{\left({\mathbf{X}}_{-i}^t{\mathbf{X}}_{-i}\right)}^{-1}{\mathbf{X}}_{-i}^t{\mathbf{X}}_i\right)}^{-1}\\ & =\left({\mathbf{X}}_i^t\left(\mathbf{I}-{\mathbf{X}}_{-i}{\left({\mathbf{X}}_{-i}^t{\mathbf{X}}_{-i}\right)}^{-1}{\mathbf{X}}_{-i}^t\right){\mathbf{X}}_i\right)={\left({\mathbf{e}}_i^t{\mathbf{e}}_i\right)}^{-1},\\ \mathbf{B}& =-{\left({\mathbf{X}}_{-i}^t{\mathbf{X}}_{-i}\right)}^{-1}{\mathbf{X}}_{-i}^t{\mathbf{X}}_i\cdot {\left({\mathbf{e}}_i^t{\mathbf{e}}_i\right)}^{-1}=-\hat{\mathit{\alpha }}\cdot {\left({\mathbf{e}}_i^t{\mathbf{e}}_i\right)}^{-1},\\ \mathbf{A}& ={\left({\mathbf{X}}_{-i}^t{\mathbf{X}}_{-i}\right)}^{-1}+{\left({\mathbf{X}}_{-i}^t{\mathbf{X}}_{-i}\right)}^{-1}{\mathbf{X}}_{-i}^t{\mathbf{X}}_i\cdot {\left({\mathbf{e}}_i^t{\mathbf{e}}_i\right)}^{-1}{\mathbf{X}}_i^t{\mathbf{X}}_{-i}{\left({\mathbf{X}}_{-i}^t{\mathbf{X}}_{-i}\right)}^{-1}\\ & ={\left({\mathbf{X}}_{-i}^t{\mathbf{X}}_{-i}\right)}^{-1}+{\left({\mathbf{e}}_i^t{\mathbf{e}}_i\right)}^{-1}\cdot \hat{\mathit{\alpha }}{\hat{\mathit{\alpha }}}^t,\end{array}$$

where $\hat{\mathit{\alpha }}$ and ${\mathbf{e}}_i^t{\mathbf{e}}_i$ are, respectively, the OLS estimator and the sum of the square of the residuals of the auxiliary regression (2).

Likewise, since ${\mathbf{e}}_i^t{\mathbf{X}}_{-i}=\mathbf{0}$, the OLS estimation of model (3), $\hat{\mathit{\gamma }}$, is given by

$$\begin{array}{rl}\hat{\mathit{\gamma }}& ={\left({\mathbf{X}}_O^t{\mathbf{X}}_O\right)}^{-1}{\mathbf{X}}_O^t\mathbf{Y}={\left(\begin{array}{cc}{\mathbf{X}}_{-i}^t{\mathbf{X}}_{-i}& {\mathbf{X}}_{-i}^t{\mathbf{e}}_i\\ {\mathbf{e}}_i^t{\mathbf{X}}_{-i}& {\mathbf{e}}_i^t{\mathbf{e}}_i\end{array}\right)}^{-1}\cdot \left(\begin{array}{c}{\mathbf{X}}_{-i}^t\mathbf{Y}\\ {\mathbf{e}}_i^t\mathbf{Y}\end{array}\right)\\ & =\left(\begin{array}{c}{\left({\mathbf{X}}_{-i}^t{\mathbf{X}}_{-i}\right)}^{-1}{\mathbf{X}}_{-i}^t\mathbf{Y}\\ \frac{{\mathbf{e}}_i^t\mathbf{Y}}{{\mathbf{e}}_i^t{\mathbf{e}}_i}\end{array}\right)=\left(\begin{array}{c}{\hat{\mathit{\gamma }}}_{-i}\\ {\hat{\gamma }}_i\end{array}\right).\end{array}$$ (5)

Thus it is possible to compare the OLS estimators, expression (5), of the residualized model (3) with the OLS estimators of model (1), expression (4), obtaining the following conclusions:

• The estimate of the coefficient of the residualized variable does not change in model (3), that is, ${\widehat{\mathrm{\text{β}}}}_i={\hat{\gamma }}_i$. However, the interpretation is different: the variation produced in dependent variable, $\mathbf{Y}$, given an increase in ${\mathbf{e}}_i$, that is to say, the part of the independent variable ${\mathbf{X}}_i$ that is not related to the rest of independent variables, ${\mathbf{X}}_{-i}$. Hence, due to the new interpretation of the residualized variable, the residualization could be applied to obtain conclusions that would not otherwise be possible.

• The orthogonality of ${\mathbf{e}}_i$ with ${\mathbf{X}}_{-i}$ verifies the ceteris paribus assumption, that is to say, when this variable increases, the other variables remain constant.

• The estimation of the non-residualized variables in model (3) differs:

  $${\widehat{\mathrm{\text{β}}}}_{-i}={\hat{\mathit{\gamma }}}_{-i}-\hat{\mathit{\alpha }}\cdot \frac{{\mathbf{e}}_i^t\mathbf{Y}}{{\mathbf{e}}_i^t{\mathbf{e}}_i}.$$ (6)

  However, the interpretation will be the same as in model (1).

In addition, it is interesting to take into consideration that:

• For convenience purposes, all of the independent variables in model (1) have been included in the auxiliary regression (2). However, it will be possible to include only some of the independent variables, depending on the interest of the researcher (e.g. trying to obtain interpretable residuals). In this case, the estimations of explanatory variables which are not included in the auxiliary regression will not change their value. Furthermore, if the constant is included in the auxiliary regression, nonessential collinearity will be mitigated because the residuals will be orthogonal to the constant. See Section 3 for details on distinguishing among the different types of collinearity.

• The estimate of the non-residualized variables in model (3) coincides with the estimate obtained from model $\mathbf{Y}={\mathbf{X}}_{-i}\mathit{\delta }+\mathit{\vartheta }$. That is, the estimation and interpretation of the non-residualized variables will be the same as that obtained in a regression in which the residualized variable is eliminated. Nevertheless, this coincidence only occurs when we introduce in the auxiliary regression all the rest of explanatory variables of the original model. Furthermore, since the two models have different residuals, the inference associated with these coefficients will be different.

Remark 2.1

Another interesting issue is how to select the variable to residualize. This paper presents different criteria that can be applied, or a combination thereof, depending on the goal of the research.

If the goal is to look for new interpretations, the variable to residualize will be the one that leads to the new interpretation desired by the researcher since the only interpretation that changes is that of the residualized variable.

It may be also interesting to rank the independent variables in model (1) according to their relevance to avoid residualizing variables considered to be relevant maintaining the original interpretation of these coefficients. This fact was already been proposed in [3] with the use of OVR models.

2.2. Goodness of fit, estimation of the variance of the random disturbance and joint significance

The estimated residuals of the original model (1) will be given by

$$\begin{array}{rl}\mathbf{e}& =\mathbf{Y}-\hat{\mathbf{Y}}=\mathbf{Y}-\mathbf{X}\cdot \widehat{\mathrm{\text{β}}}=\mathbf{Y}-\left({\mathbf{X}}_{-i}{\mathbf{X}}_i\right)\cdot \left(\begin{array}{c}{\left({\mathbf{X}}_{-i}^t{\mathbf{X}}_{-i}\right)}^{-1}{\mathbf{X}}_{-i}^t\mathbf{Y}-\hat{\mathit{\alpha }}\cdot \frac{{\mathbf{e}}_i^t\mathbf{Y}}{{\mathbf{e}}_i^t{\mathbf{e}}_i}\\ \frac{{\mathbf{e}}_i^t\mathbf{Y}}{{\mathbf{e}}_i^t{\mathbf{e}}_i}\end{array}\right)\\ & =\mathbf{Y}-{\mathbf{X}}_{-i}{\left({\mathbf{X}}_{-i}^t{\mathbf{X}}_{-i}\right)}^{-1}{\mathbf{X}}_{-i}^t\mathbf{Y}+{\mathbf{X}}_{-i}\hat{\mathit{\alpha }}\cdot \frac{{\mathbf{e}}_i^t\mathbf{Y}}{{\mathbf{e}}_i^t{\mathbf{e}}_i}-{\mathbf{X}}_i\frac{{\mathbf{e}}_i^t\mathbf{Y}}{{\mathbf{e}}_i^t{\mathbf{e}}_i}\\ & =\mathbf{Y}-{\mathbf{X}}_{-i}{\left({\mathbf{X}}_{-i}^t{\mathbf{X}}_{-i}\right)}^{-1}{\mathbf{X}}_{-i}^t\mathbf{Y}-{\mathbf{e}}_i\frac{{\mathbf{e}}_i^t\mathbf{Y}}{{\mathbf{e}}_i^t{\mathbf{e}}_i}.\end{array}$$ (7)

Since ${\mathbf{e}}_i$ are the residuals of the auxiliary regression (2), it is verified that ${\mathbf{e}}_i={\mathbf{X}}_i-{\mathbf{X}}_{-i}\hat{\mathit{\alpha }}$.

And the residuals of the residualized model (3) are

$$\mathbf{e}=\mathbf{Y}-{\hat{\mathbf{Y}}}_O=\mathbf{Y}-{\mathbf{X}}_O\cdot \hat{\mathit{\gamma }}=\mathbf{Y}-{\mathbf{X}}_{-i}{\left({\mathbf{X}}_{-i}^t{\mathbf{X}}_{-i}\right)}^{-1}{\mathbf{X}}_{-i}^t\mathbf{Y}-{\mathbf{e}}_i\frac{{\mathbf{e}}_i^t\mathbf{Y}}{{\mathbf{e}}_i^t{\mathbf{e}}_i}.$$ (8)

It is evident that expression (8) coincides with (7), that is to say, the residuals of the original (1) and residualized (3) models coincide. Therefore, it is possible to conclude the following:

• It is evident that the squared sums of the residuals of the two models coincide and, consequently, both models yield the same estimate of the variance of the random disturbance.

• Since the two models employ the same dependent variable, the total sum of squares will be also the same, and consequently, the coefficients of determination will also coincide.

• Since the F statistic of the global significance test can be expressed as a function of the coefficient of determination, it is evident that the global significance tests of both models will also be the same.

• It is clear that $\hat{\mathbf{Y}}={\hat{\mathbf{Y}}}_O$, it is to say that the original model and the residualized one provide the same estimation.

2.3. Individual inference

Since the random disturbances are spherical, the individual inference will be given by the main diagonal of matrix $({\mathbf{X}}^t\mathbf{X}{)}^{-1}$, that is to say (see expression (4)), by

$$\left(\begin{array}{cc}{\left({\mathbf{X}}_{-i}^t{\mathbf{X}}_{-i}\right)}^{-1}+{\left({\mathbf{e}}_i^t{\mathbf{e}}_i\right)}^{-1}\cdot \hat{\mathit{\alpha }}{\hat{\mathit{\alpha }}}^t& -\hat{\mathit{\alpha }}\cdot {\left({\mathbf{e}}_i^t{\mathbf{e}}_i\right)}^{-1}\\ -{\hat{\mathit{\alpha }}}^t\cdot {\left({\mathbf{e}}_i^t{\mathbf{e}}_i\right)}^{-1}& {\left({\mathbf{e}}_i^t{\mathbf{e}}_i\right)}^{-1}\end{array}\right).$$ (9)

Taking into account the following expression:

$${\left({\mathbf{X}}_O^t{\mathbf{X}}_O\right)}^{-1}=\left(\begin{array}{cc}{\left({\mathbf{X}}_{-i}^t{\mathbf{X}}_{-i}\right)}^{-1}& \mathbf{0}\\ \mathbf{0}& {\left({\mathbf{e}}_i^t{\mathbf{e}}_i\right)}^{-1}\end{array}\right),$$ (10)

it is evident that the main diagonals of both matrices are different, except for the i element. Since the estimation of the variance of the random disturbance is the same, considering the estimation of the coefficients, it is possible to conclude the following:

• The inference related to the individual significance (Student's t-test) of the unchanged variables differs between models (1) and (3).

• The inference related to the individual significance (Student's t-test) of the residualized variable coincides in models (1) and (3).

Consequently, the residualization of the initial model does not affect the estimation of the variance of the random disturbance, the coefficient of determination, the global significance test or the individual significance test of the residualized variable. It only changes the individual significance of unaltered variables.

Remark 2.2

Another option to select the variable to be residualized is to choose a variable with a coefficient that is significantly different from zero in the original model since the individual significance test of the residualized variable is maintained in the residualized model.

3. Collinearity

Multicollinearity consists of the presence of interdependency between explanatory variables [19], distinguishing between two principal types of multicollinearity: perfect collinearity and near-collinearity [43,47,58]. The first type occurs when the interdependency between variables is exact, and the second occurs when it is approximate. Near-collinearity, also known as imperfect or approximate collinearity, may be divided into essential and nonessential collinearity [39,40,59]. The former concerns the relationship between explanatory variables, excluding the intercept, while the latter involves the relationship between the intercept and at least one of the remaining independent variables of the model.

In addition to the new interpretation of the coefficient of the residualized variable, another result of interest in the residualized model is the effect on the linear relationship between the independent variables of the initial model. To verify that collinearity is mitigated after the residualization of the initial model, the estimated variances of the estimated coefficients, the VIF and the CN are analyzed in the residualized model.

3.1. Decrease in estimated standard variance

Considering that the estimation of the random disturbance variance is the same in the original and residualized models, the estimated variances of the coefficients will be determined by the main diagonal of the matrices $({\mathbf{X}}^t\mathbf{X}{)}^{-1}$ and $({\mathbf{X}}_O^t{\mathbf{X}}_O{)}^{-1}$. As noted above, the element corresponding to the residualized variable is the same in both matrices, and thus, the estimated variance will be also the same. That is, $\widehat{Var}\left({\widehat{\mathrm{\text{β}}}}_i\right)=\widehat{Var}\left({\hat{\gamma }}_i\right)$.

For the rest of the variables, given expressions (9) and (10), it is possible to obtain that

$$\widehat{Var}\left({\widehat{\mathrm{\text{β}}}}_j\right)={\hat{\sigma }}^2\cdot \left(w_{jj}+({\mathbf{e}}_i^t{\mathbf{e}}_i{)}^{-1}{\alpha }_{jj}\right),\widehat{Var}\left({\hat{\gamma }}_j\right)={\hat{\sigma }}^2\cdot w_{jj},j=1,\dots ,p,j\ne i,$$

where $w_{jj}$ and ${\alpha }_{jj}={\alpha }_j^2$ are the elements $(j,j)$ of the matrices $({\mathbf{X}}_{-i}^t{\mathbf{X}}_{-i}{)}^{-1}$ and $\hat{\mathit{\alpha }}{\hat{\mathit{\alpha }}}^t$, respectively. Since $({\mathbf{e}}_i^t{\mathbf{e}}_i{)}^{-1}{\alpha }_j^2\ge 0$, it is verified that $\widehat{Var}\left({\widehat{\mathrm{\text{β}}}}_j\right)\ge \widehat{Var}\left({\hat{\gamma }}_j\right)$ for $j=1,\dots ,p,$ with $j\ne i$. In consequence, the estimated variances of the residualized model will be always less than or equal to those in the original model.

This result is relevant since it demonstrates that the residualization implies a decrease in the estimated variances of the estimated coefficients (which are assumed to be inflated due to the presence of collinearity). Note that this result is contrary to the conclusions presented in [7].

Remark 3.1

The linear relationship on the coefficients of the model (1) given in (6) can also be used to reduce the variance of the estimated coefficients only estimating the model by the restricted least-squares estimator. In this case, the residualization could be used to mitigate this particular consequence of the existence of severe collinearity in the multiple linear regression model.

3.2. Variance inflation factor

Each explanatory variable of model (1) has an associated VIF given by

$${\mathrm{V}\mathrm{I}\mathrm{F}}_i=\frac{1}{1-R_i^2},i=2,\dots ,p,$$ (11)

where $R_i^2$ is the coefficient of determination of model (2). It is generally accepted that values of VIF higher than 10 indicate severe collinearity [31].

Applying this definition in the residualized model (3) and being ${\mathbf{e}}_i$ the dependent variable of the auxiliary regression, its coefficient of determination will be zero and the associated VIF will be one (the minimum value possible). In other case, $R_j^2$ will be obtained from the following auxiliary regression:

$${\mathbf{X}}_j={\mathbf{X}}_{O_{-j}}\mathit{\xi }+\mathit{ϵ},j=2,\dots ,p,j\ne i,$$ (12)

where ${\mathbf{X}}_{O_{-j}}$ is the result obtained after eliminating column (variable) j from matrix ${\mathbf{X}}_O$.

Due to the orthogonality of ${\mathbf{e}}_i$ with ${\mathbf{X}}_{-i,-j}$ (matrix $\mathbf{X}$ without columns (variables) i and j), the residuals of (12) coincide¹ with the residuals of the following model:

$${\mathbf{X}}_j={\mathbf{X}}_{-i,-j}\mathit{\eta }+\mathit{\epsilon },i,j=2,\dots ,p,i\ne j.$$ (13)

Then, models (12) and (13) have the same coefficient of determination since the dependent variable is the same in both models.

However, the coefficient of determination of model (13) will be lower than that of the following model:

$${\mathbf{X}}_j={\mathbf{X}}_{-j}\mathit{\theta }+\mathit{\omega },i,j=2,\dots ,p,i\ne j,$$ (14)

since this latter model contains an additional independent variable, ${\mathbf{X}}_i$. Then, the coefficient of determination of model (12) is lower than that of the model (14).

Thus since the VIF associated with variable j in the original model (1) is obtained from the coefficient of determination of the auxiliary regression given by (14) and in the residualized model (3) from the coefficient of determination of model (12), it is clear that the VIF is decreased after residualizing the model. That is, the collinearity present in the model has been diminished.

Remark 3.2

If the goal is to mitigate the collinearity in the model, one suggestion may be to residualize the variable with the highest VIF because, after the residualization, the VIF will be equal to 1. In this case, all independent variables have to be included in the auxiliary regression (2) to mitigate the essential and nonessential collinearities in the most efficient way.

3.3. Condition number

Given the model (1), the CN is given by

$$K\left({\tilde{\mathbf{X}}}^t\tilde{\mathbf{X}}\right)=\sqrt{\frac{{\mu }_{\mathrm{m}\mathrm{a}\mathrm{x}}}{{\mu }_{\mathrm{m}\mathrm{i}\mathrm{n}}}},$$

where ${\mu }_{\mathrm{m}\mathrm{i}\mathrm{n}}$ and ${\mu }_{\mathrm{m}\mathrm{a}\mathrm{x}}$ are, respectively, the minimum and maximum eigenvalues of ${\mathbf{X}}^t\mathbf{X}$. Note that, previously, the matrix $\mathbf{X}$ should be transformed to be unit length by columns, it is to say, the data should be divided by the square root of the sum of its squared elements (see [5]). This author stated that values of CN between 20 and 30 indicate moderated collinearity and values higher than 30 indicate high collinearity.

Then, the CN associated to model (3) is obtained as follows:

$$K\left({\tilde{\mathbf{X}}}_O^t{\tilde{\mathbf{X}}}_O\right)=\sqrt{\frac{{\lambda }_{\mathrm{m}\mathrm{a}\mathrm{x}}}{{\lambda }_{\mathrm{m}\mathrm{i}\mathrm{n}}}},$$

where ${\lambda }_{\mathrm{m}\mathrm{i}\mathrm{n}}$ and ${\lambda }_{\mathrm{m}\mathrm{a}\mathrm{x}}$ are, respectively, the minimum and maximum eigenvalues of ${\tilde{\mathbf{X}}}_O^t{\tilde{\mathbf{X}}}_O$, where

$${\tilde{\mathbf{X}}}_O=\left(\frac{{\mathbf{X}}_1}{||{\mathbf{X}}_1||}\cdots \frac{{\mathbf{X}}_{i-1}}{||{\mathbf{X}}_{i-1}||}\frac{{\mathbf{X}}_{i+1}}{||{\mathbf{X}}_{i+1}||}\cdots \frac{{\mathbf{X}}_p}{||{\mathbf{X}}_p||}\frac{{\mathbf{e}}_i}{||{\mathbf{e}}_i||}\right)=\left({\tilde{\mathbf{X}}}_{-i}\frac{{\mathbf{e}}_i}{||{\mathbf{e}}_i||}\right),$$

being $||{\mathbf{X}}_k||=\sqrt{\sum _{j=1}^n{X}_{kj}^2}$ for $k=1,\dots ,i-1,i+1,\dots ,p$ and $||{\mathbf{e}}_i||=\sqrt{\sum _{j=1}^n{e}_{ij}^2}$. Then

$${\tilde{\mathbf{X}}}_O^t{\tilde{\mathbf{X}}}_O=\left(\begin{array}{cc}{\tilde{\mathbf{X}}}_{-i}^t{\tilde{\mathbf{X}}}_{-i}& \mathbf{0}\\ \mathbf{0}& 1\end{array}\right).$$

Thus one of the p eigenvalues of ${\tilde{\mathbf{X}}}_O^t{\tilde{\mathbf{X}}}_O$ will be equal to one and the rest will coincide with the eigenvalues of matrix ${\tilde{\mathbf{X}}}_{-i}^t{\tilde{\mathbf{X}}}_{-i}$. Supposing that the eigenvalue equal to one is the first one, ${\lambda }_1=1$, it is verified that

• If this is the minimum eigenvalue of ${\tilde{\mathbf{X}}}_O^t{\tilde{\mathbf{X}}}_O$, the rest of eigenvalues will be equal or higher than one (${\lambda }_i\ge 1,i=2,\dots ,p$), and, consequently, its sum will be equal or higher than p−1 $(\sum _{i=2}^p{\lambda }_i\ge p-1)$. However, this sum will be equal to p−1 (since the trace of ${\tilde{\mathbf{X}}}_{-i}^t{\tilde{\mathbf{X}}}_{-i}$ is equal to p−1). Then, all the eigenvalues will be equal to one (${\lambda }_i=1$ with $i=1,2,\dots ,p$), it is to say, ${\tilde{\mathbf{X}}}_O^t{\tilde{\mathbf{X}}}_O$ will be the identity matrix and, then, all the variables will be considered orthogonal to each other.

• If this is the maximum eigenvalue of ${\tilde{\mathbf{X}}}_O^t{\tilde{\mathbf{X}}}_O$, the rest of eigenvalues will be equal or lesser than one (${\lambda }_i\le 1,i=2,\dots ,p$), and, consequently, its sum will be equal or lesser than p−1 $(\sum _{i=2}^p{\lambda }_i\le p-1)$. However, it was justified that this sum is equal to p−1. Then, all the eigenvalues will be equal to one (${\lambda }_i=1$ with $i=1,2,\dots ,p$). Thus, as before, all the variables will be considered orthogonal to each other.

If the eigenvalue equal to one cannot be the minimum or maximum eigenvalue of ${\tilde{\mathbf{X}}}_O^t{\tilde{\mathbf{X}}}_O$, they will have to be found in the rest of the eigenvalues of ${\tilde{\mathbf{X}}}_{-i}^t{\tilde{\mathbf{X}}}_{-i}$. Thus the CN of model (3) coincides with that of the auxiliary regression (2):

$$K\left({\tilde{\mathbf{X}}}_O^t{\tilde{\mathbf{X}}}_O\right)=K\left({\tilde{\mathbf{X}}}_{-i}^t{\tilde{\mathbf{X}}}_{-i}\right).$$

On the other hand, according to the Cauchy's Interlace Theorem for Eigenvalues of Hermitian Matrices,² since ${\tilde{\mathbf{X}}}_{-i}^t{\tilde{\mathbf{X}}}_{-i}$ is a submatrix of order p−1 of ${\tilde{\mathbf{X}}}^t\tilde{\mathbf{X}}$, it is evident that it has to be verified that

$$K\left({\tilde{\mathbf{X}}}_O^t{\tilde{\mathbf{X}}}_O\right)=K\left({\tilde{\mathbf{X}}}_{-i}^t{\tilde{\mathbf{X}}}_{-i}\right)\le K\left({\tilde{\mathbf{X}}}^t\tilde{\mathbf{X}}\right).$$

Thus the CN of the residualization (3) has to be equal or lesser than the CN of the original model (1).

Remark 3.3

If the goal is to mitigate the collinearity in the model, one suggestion could be to residualize the variable i whose auxiliary regression (where the variable i is the dependent one) presents the lowest CN since it coincides with the CN of the residualized model that will always be equal to or lesser than the CN of the original model.

4. Comparison of the residualization method with other existing methods

This section presents a Monte Carlo simulation to compare the residualization methodology with other existing methods such as ridge regression, PCR and PLSR in relation to the mean square error (MSE) and prediction error. First, the obtention of the MSE of the residualization method is presented, as well as the way to compare it with the MSE obtained by OLS. Second, the metrics used to measure the prediction capability of each methodology are also presented.

4.1. Mean square error

Note that the original model is different from the residualized model, and for this reason, both models should be analyzed separately and the comparison may not be convenient. However, some publications have not considered this divergence (see, e.g. [66]). For this reason, given that $\hat{\mathit{\gamma }}$ is a biased estimator of $\mathrm{\text{β}}$:

$$\begin{array}{rl}\hat{\mathit{\gamma }}& ={\left({\mathbf{X}}_O{\mathbf{X}}_O\right)}^{-1}\cdot {\mathbf{X}}_O^t\mathbf{Y}={\left({\mathbf{X}}_O{\mathbf{X}}_O\right)}^{-1}\cdot {\mathbf{X}}_O^t\mathbf{X}\cdot \mathrm{\text{β}}+{\left({\mathbf{X}}_O{\mathbf{X}}_O\right)}^{-1}\cdot {\mathbf{X}}_O^t\cdot \mathbf{u}E\left[\hat{\mathit{\gamma }}\right]\\ & ={\left({\mathbf{X}}_O{\mathbf{X}}_O\right)}^{-1}\cdot {\mathbf{X}}_O^t\mathbf{X}\cdot \mathrm{\text{β}}\ne \mathrm{\text{β}}\text{ since }{\mathbf{X}}_O^t\ne \mathbf{X},\end{array}$$

it could be interesting to calculate the MSE of residualization and to compare it with the MSE of the OLS estimator.

Given an estimator $\widetilde{\mathrm{\text{β}}}$ of $\mathrm{\text{β}}$, its MSE is expressed as

$$\mathrm{M}\mathrm{S}\mathrm{E}\left(\widetilde{\mathrm{\text{β}}}\right)=trace\left(var\left(\widetilde{\mathrm{\text{β}}}\right)\right)+{\left(E\left[\widetilde{\mathrm{\text{β}}}\right]-\mathrm{\text{β}}\right)}^t\left(E\left[\widetilde{\mathrm{\text{β}}}\right]-\mathrm{\text{β}}\right).$$

In the case of the OLS estimator, $\widehat{\mathrm{\text{β}}}$ is an unbiased estimator, $E\left[\widehat{\mathrm{\text{β}}}\right]=\mathrm{\text{β}}$, and, taking into account expression (9), the following is verified:

$$\begin{array}{rl}\mathrm{M}\mathrm{S}\mathrm{E}\left(\widehat{\mathrm{\text{β}}}\right)& =trace\left(var\left(\widehat{\mathrm{\text{β}}}\right)\right)\\ & ={\sigma }^2\cdot \left[trace{\left({\mathbf{X}}_{-i}^t{\mathbf{X}}_{-i}\right)}^{-1}+({\mathbf{e}}_i^t{\mathbf{e}}_i{)}^{-1}\cdot trace\left(\stackrel{\mathbf{ˆ}}{\mathit{\alpha }}{\stackrel{\mathbf{ˆ}}{\mathit{\alpha }}}^t\right)+({\mathbf{e}}_i^t{\mathbf{e}}_i{)}^{-1}\right].\end{array}$$ (15)

For the estimator $\hat{\mathit{\gamma }}$, starting from (10), it is verified³ that

$$\begin{array}{rl}\mathrm{M}\mathrm{S}\mathrm{E}\left(\hat{\mathit{\gamma }}\right)& =trace\left(var\left(\hat{\mathit{\gamma }}\right)\right)+{\left(E\left[\hat{\mathit{\gamma }}\right]-\mathrm{\text{β}}\right)}^t\left(E\left[\hat{\mathit{\gamma }}\right]-\mathrm{\text{β}}\right)\\ & ={\sigma }^2\cdot \left[trace{\left({\mathbf{X}}_{-i}^t{\mathbf{X}}_{-i}\right)}^{-1}+({\mathbf{e}}_i^t{\mathbf{e}}_i{)}^{-1}\right]+{\mathrm{\text{β}}}_i\cdot {\stackrel{\mathbf{ˆ}}{\mathit{\alpha }}}^t\stackrel{\mathbf{ˆ}}{\mathit{\alpha }}\cdot {\mathrm{\text{β}}}_i.\end{array}$$ (16)

From expressions (15) and (16), it is clear that

$$\mathrm{M}\mathrm{S}\mathrm{E}\left(\hat{\mathit{\gamma }}\right)=\mathrm{M}\mathrm{S}\mathrm{E}\left(\widehat{\mathrm{\text{β}}}\right)-{\sigma }^2\cdot ({\mathbf{e}}_i^t{\mathbf{e}}_i{)}^{-1}\cdot trace\left(\stackrel{\mathbf{ˆ}}{\mathit{\alpha }}{\stackrel{\mathbf{ˆ}}{\mathit{\alpha }}}^t\right)+{\mathrm{\text{β}}}_i^2\cdot {\hat{\mathit{\alpha }}}^t\hat{\mathit{\alpha }},$$

so $\hat{\mathit{\gamma }}$ has a lower MSE than $\widehat{\mathrm{\text{β}}}$ if

$${\mathrm{\text{β}}}_i^2\cdot {\hat{\mathit{\alpha }}}^t\hat{\mathit{\alpha }}<{\sigma }^2\cdot ({\mathbf{e}}_i^t{\mathbf{e}}_i{)}^{-1}\cdot trace\left(\stackrel{\mathbf{ˆ}}{\mathit{\alpha }}{\stackrel{\mathbf{ˆ}}{\mathit{\alpha }}}^t\right).$$ (17)

4.2. Metrics

The root mean squared error (RMSE) and the mean absolute error (MAE) will be applied to measure the fit capability of each model while the prediction capability will be measured by the root mean squared prediction error (RMSPE) and the mean absolute prediction error (MAPE).

Given a sample with n observations and assuming it is divided into two subsamples: the first with m observations and the second with h observations, verifying that m + h = n. Then, the first subsample is applied to measure the fit capability calculating the RMSE and MAE. The second subsample is applied to evaluate the prediction capability obtaining the RMSPE and MAPE. Then, the following expressions are obtained:

$$\begin{array}{rl}\mathrm{R}\mathrm{M}\mathrm{S}\mathrm{E}& =\sqrt{\frac{1}{m}\cdot \sum _{i=1}^m(Y_i-{\hat{Y}}_i{)}^2},\mathrm{M}\mathrm{A}\mathrm{E}=\frac{1}{m}\sum _{i=1}^m|Y_i-{\hat{Y}}_i|,\\ \mathrm{R}\mathrm{M}\mathrm{S}\mathrm{P}\mathrm{E}& =\sqrt{\frac{1}{h}\cdot \sum _{i=m+1}^n(Y_i-{\hat{Y}}_i{)}^2},\mathrm{M}\mathrm{A}\mathrm{P}\mathrm{E}=\frac{1}{h}\cdot \sum _{i=m+1}^n|Y_i-{\hat{Y}}_i|.\end{array}$$

4.3. Simulation

The simulation performed to compare the residualization with other existing methods is described below.

Given the model $\mathbf{Y}={\mathrm{\text{β}}}_1+{\mathrm{\text{β}}}_2\cdot {\mathbf{X}}_2+{\mathrm{\text{β}}}_3\cdot {\mathbf{X}}_3+\mathbf{u}$, the following simulation is performed in order to establish the behavior of condition (17):

1. It is considered that ${\mathit{\mu }}_{2\times 1}=({\mu }_1,{\mu }_2{)}^t$ with ${\mu }_1,{\mu }_2\in \{-10,-9,-8,\dots ,8,9,10\}$.

2. It is also considered that $a_1,a_2\in \{0,1,2,3,4\}$ and $b_1,b_2\in \{0.1,0.2,0.3,\dots ,1.9,2\}$, so $c_{5\times 1}^i\sim N(a_i,b_i^2)$ is generated. Thus, given matrix $\mathbf{C}=\left[c^1{c}^2\right]$, a symmetric positive-definite matrix, ${\mathrm{\Sigma }}_{2\times 2}={\mathbf{C}}^t\mathbf{C}$, is built.

3. ${\mathbf{X}}_2$ and ${\mathbf{X}}_3$ are generated from $N_2({\mathit{\mu }}_{2\times 1},{\mathrm{\Sigma }}_{2\times 2})$.

4. The random perturbance, $\mathbf{u}$, is generated as $\mathbf{u}\sim N(0,d^2)$, where $d\in \{1,2,3,4\}$, from which it is calculated that $\mathbf{Y}={\mathrm{\text{β}}}_1+{\mathrm{\text{β}}}_2\cdot {\mathbf{X}}_2+{\mathrm{\text{β}}}_3\cdot {\mathbf{X}}_3+\mathbf{u}$, where ${\mathrm{\text{β}}}_i\in \{-5,-4,\dots ,4,5\}$.

5. A comparison of both models (OLS and residualization) is conducted with different sample sizes, $n\in \{25,50,75,100,125,150\}$, such that 60,000 simulations are performed in this experiment.

First, once the previous model and the corresponding auxiliary regressions are estimated, condition (17) is calculated from the obtained estimations of ${\mathrm{\text{β}}}_i$, ${\sigma }^2$ and $\mathit{\alpha }$. In Table 1, it can be observed that there are two types of situations: one where essential collinearity does not imply strong collinearity problems (the mean correlation is equal to 0.4877, which leads to a VIF value of 1.31208) and another where essential collinearity implies strong collinearity problems (the maximum and minimum correlations lead to VIF values of approximately 50.2512). It can also be observed that there are two types of situations in relation to nonessential collinearity: one where it is not worrisome (the mean value of the coefficient of variation (CV) is approximately 6, which implies the data have enough variability) and another where nonessential collinearity is worrisome (the minimum values of CV for each variable are close to zero, which implies slight variability of the data and indicates that the data may be considered almost constant and hence related to the intercept).

Table 1. Simulation results for MSE.

n		25	50	75	100	125	150	Mean
Cond. (17)	Resid. var. ${\mathbf{X}}_2$	8.13%	6.96%	7.15%	6.69%	6.96%	7.00%	7.159%
	Resid. var. ${\mathbf{X}}_3$	8.42%	7.15%	7.12%	6.71%	6.77%	6.84%
	$mincor({\mathbf{X}}_2,{\mathbf{X}}_3)$	−0.9862	−0.9747	−0.9883	−0.9973	−0.9934	−0.9915	0.4877
	$maxcor({\mathbf{X}}_2,{\mathbf{X}}_3)$	0.9999	0.9999	0.9999	0.9999	0.9998	0.9999
	$min\mathrm{C}\mathrm{V}\left({\mathbf{X}}_2\right)$	0.00958	0.00935	0.00911	0.0111296	0.00628	0.01072	6.6665
	$max\mathrm{C}\mathrm{V}\left({\mathbf{X}}_2\right)$	30222.48	12359.03	3249.84	3426.69	31001.38	1498.77
	$min\mathrm{C}\mathrm{V}\left({\mathbf{X}}_3\right)$	0.0107	0.0116	0.00791	0.00682	0.0108	0.0114	5.7096
	$max\mathrm{C}\mathrm{V}\left({\mathbf{X}}_3\right)$	9763.29	37893.3	5199.45	2718.18	2655.24	3586.79

The first and second rows of Table 1 show the percentage of cases in which $\mathrm{M}\mathrm{S}\mathrm{E}\left(\hat{\mathit{\gamma }}\right)<\mathrm{M}\mathrm{S}\mathrm{E}\left(\widehat{\mathrm{\text{β}}}\right)$ (condition (17)) is verified, considering that variables ${\mathbf{X}}_2$ and ${\mathbf{X}}_3$ are residualized. Note that both results are similar and that there are no material differences for different sample sizes. The results show that in only 7.159% of the cases, the condition $\mathrm{M}\mathrm{S}\mathrm{E}\left(\hat{\mathit{\gamma }}\right)<\mathrm{M}\mathrm{S}\mathrm{E}\left(\widehat{\mathrm{\text{β}}}\right)$ is verified.

Second, Table 2 is obtained from 60,000 more simulations performed dividing the sample as was described in Section 4.2 and considering $h=0.15\cdot n$. R's package pls was applied to obtain values of PCR and PLSR considering one and two main components. For ridge regression, the value of k was selected to mitigate the collinearity considering that it is not worrying for values of CN lesser than 20, as shown by [54]. This idea was also applied by [22] but using the VIF instead of the CN, but in this case it was considered more appropriate to use the CN since the VIF ignores the nonessential collinearity [52].

Table 2. Simulation results for RMSE, MAE, RMSPE and MAPE.

Metric	OLS	Resid. var. ${\mathbf{X}}_2$	Resid. var. ${\mathbf{X}}_3$	Ridge	PCR (1c)	PLSR (1c)	PCR (2c)	PLSR (2c)
RMSE	2.635103	2.635103	2.635103	2.637032	8.289639	5.435111	8.810389	6.182757
MAE	1.960508	1.960508	1.960508	1.961239	6.611481	4.300871	8.507369	6.266333
RMSPE	2.725054	19.73536	19.64029	2.721985	8.49428	5.591137	2.725054	2.725054
MAPE	2.087758	17.56701	17.48688	2.084665	6.92049	4.524651	2.087758	2.087758

From the results of the first sample, it is obtained that the residualization method and OLS lead to the same results for RMSE and MAE as was commented in Section 2.2 due to $\hat{\mathbf{Y}}={\hat{\mathbf{Y}}}_O$ being verified. These values are slightly lower than those of other techniques.

From values of RMSPE and MAPE obtained from the second sample, it is possible to conclude that the residualization method presents the lowest prediction capability. However, the fact that the rest of methods do not improve the results obtained by OLS, could indicate that, when the purpose is prediction, the best way to proceed is to do nothing. These results support the idea provided by [25]: if the goal is simply to predict, then multicollinearity is not a problem because the predictions will still be accurate.

5. Successive residualization

It is possible that the goal of the researcher (to mitigate collinearity or obtain a new interpretation for the estimated coefficients) has not been achieved after residualizing the first variable. In that case, it is necessary to residualize a second variable. In this case, the doubly residualized model will be given by

$$\mathbf{Y}={\mathbf{X}}_{OO}\mathit{\lambda }+\mathit{\varsigma },$$ (18)

where ${\mathbf{X}}_{OO}=\left({\mathbf{X}}_{-i,-j}{\mathbf{e}}_i{\mathbf{e}}_j\right)$ with ${\mathbf{e}}_j$ being the residuals of the auxiliary regression (13).

The goal of this section is not to obtain the estimated inference of this model (18) but to highlight that successive residualization may be interesting, and in this case, the residuals ${\mathbf{e}}_i$ y ${\mathbf{e}}_j$ will be orthogonal since it is verified that

$${\mathbf{e}}_i^t{\mathbf{e}}_j={\mathbf{e}}_i^t\left({\mathbf{X}}_j-{\mathbf{X}}_{-i,-j}\hat{\mathit{\eta }}\right)={\mathbf{e}}_i^t{\mathbf{X}}_j-{\mathbf{e}}_i^t{\mathbf{X}}_{-i,-j}\hat{\mathit{\eta }}=\mathbf{0}.$$

This relationship between the residuals will still hold if more variables are residualized; that is, the degree of multicollinearity will continue decreasing. Note that if the process is repeated p−1 times, all the variables of the model will be orthogonal to each other. Interested readers should consult [53] for further information on successive residualization.

6. Empirical application

The methodology proposed herein is applied to two different models in relation to economic-financial and ecological data sets. The first example is focused on the application of residualization when the main goal of the researcher is the mitigation of collinearity and also presents a new interpretation of the modified variable. The second example shows the application of residualization when the major purpose of the study is to obtain new interpretations of the variables. The use of different models from different fields is motivated by the challenge of demonstrating the relevance and application of the method in real-world examples across a wide range of disciplines.

6.1. The economic-financial model

The first empirical application is based on a model developed by Wooldridge [63] with interest rate data from the market yields published by Salomon Brothers in An Analytical Record of Yields and Yield Spreads in the 1990s but modified in this paper for the period June 2008 to April 2019 (end-of-month data) by using a dataset from the U.S. Department of the Treasury:

$${\mathbf{c}}_{52}={\mathrm{\text{β}}}_1+{\mathrm{\text{β}}}_2{\mathbf{c}}_{13}+{\mathrm{\text{β}}}_3{\mathbf{c}}_{26}+\mathbf{u},$$ (19)

where $\mathbf{u}$ is the random disturbance, which is spherical, and ${\mathbf{c}}_{13}$, ${\mathbf{c}}_{26}$ and ${\mathbf{c}}_{52}$ represent the coupon equivalents for different maturity tranches: 13 weeks, 26 weeks and 52 weeks, respectively. The coupon equivalent, also called the bound equivalent or the investment yield, is the bill's yield based on the purchase price, discount and a 365/366-day year. The coupon equivalent can be used to compare the yield on a discount bill to the yield on a nominal coupon bond that pays semiannual interest.

The following matrix represents the correlation between the variables:

$$\begin{array}{c}\\ {\mathbf{c}}_{52}\\ {\mathbf{c}}_{13}\\ {\mathbf{c}}_{26}\end{array}\begin{array}{l}\begin{array}{rcc}{\mathbf{c}}_{52}& {\mathbf{c}}_{13}& {\mathbf{c}}_{26}\end{array}\\ \left(\begin{array}{ccc}1.000& & \\ 0.981& 1.000& \\ 0.995& 0.993& 1.000\end{array}\right).\end{array}$$

From this matrix, it is observed that all explanatory variables are positively related to the dependent variable. In addition, the coefficient of correlation between explanatory variables ${\mathbf{c}}_{13}$ and ${\mathbf{c}}_{26}$ is equal to 0.993, which can serve as a first indication of the dependence between them. From this coefficient of correlation, it is determined that the VIF is equal to 71.516, while the CN is equal to 23.233. Both measures confirm the existence of worrisome near-collinearity in this model. In relation to nonessential collinearity, the coefficients of variation ($\mathrm{C}\mathrm{V}\left({\mathbf{c}}_{13}\right)=1.486997$ and $\mathrm{C}\mathrm{V}\left({\mathbf{c}}_{26}\right)=1.280229$) are higher than the threshold established by [56], and consequently, it is possible to conclude that there is no relation between the intercept and any of the independent variables.

Table 3 presents the results of the estimation by OLS, ridge regression and residualization. As seen from the OLS results, the negative values of the estimated parameter ${\widehat{\mathrm{\text{β}}}}_2$ do not make economic sense, particularly when taking into account the matrix of correlation previously presented that suggests the existence of a direct relation between the explanatory and dependent variables.

Table 3. Results of Wooldridge model.

		OLS	Ridge	Residualization
Intercept	${\widehat{\mathrm{\text{β}}}}_1$	0.050 **	0.058	0.183 **
	(s.d.)	(0.008)	n/a	(0.007)
${\mathbf{c}}_{13}$	${\widehat{\mathrm{\text{β}}}}_2$	−0.557 **	−0.464	1.067 **
	(s.d.)	(0.065)	n/a	(0.008)
${\mathbf{c}}_{26}$	${\widehat{\mathrm{\text{β}}}}_3$	1.562 **	1.473
	(s.d.)	(0.063)	n/a
${\mathbf{e}}_{{\mathbf{c}}_{26}}$	${\widehat{\mathrm{\text{β}}}}_{3_0}$			1.562 **
	(s.d.)			(0.063)
$R^2$		0.9935	n/a	0.9935
F statistic		9794	n/a	9794
p-value (of F)		$<2.2\cdot {10}^{-16}$	n/a	$<2.2\cdot {10}^{-16}$

**, * mean the coefficient is statistically significant at 0.01 (99% level of confidence) and at 0.05 (95% level of confidence), respectively.

This nonexpected sign can be a consequence of the presence of collinearity.

To apply residualization, the first step will be to select the variable to be residualized. In this case, it is possible to consider that the medium term, ${\mathbf{c}}_{26}$, includes in some way the short term, ${\mathbf{c}}_{13}$. From the Shapley values (Table 4), it can also be seen that it contributes more to the $R^2$ of the model. Additionally, the residuals obtained from the auxiliary regression between ${\mathbf{c}}_{26}$ and ${\mathbf{c}}_{13}$ will be the part of ${\mathbf{c}}_{26}$ that is not explained by ${\mathbf{c}}_{13}$. That is, the residuals will represent the second 13-week period. This fact could indicate that the variable selected to be residualized is ${\mathbf{c}}_{26}$. Note that residualization provides a new interpretation that it is not possible to obtain from the initial model.

Table 4. Results of Wooldridge model: Shapley values.

	${\mathbf{c}}_{13}$	${\mathbf{c}}_{26}$	$R^2$
Shapley value (OLS)	0.4828	0.5107	0.9935
Share (% of $R^2$)	48.594%	51.406%	100%

When residualization is applied, the value of the estimated parameter for ${\mathbf{c}}_{13}$ becomes positive. Additionally, all the estimated coefficients are individually significant with a level of confidence of 99%, the coefficient of determination of the original model is maintained, and the essential near-collinearity problem is mitigated (the lowest possible value for the VIF, 1.000, is obtained). In this case, the CN is equal to 1.878, which indicates the relationship between ${\mathbf{c}}_{13}$ and the intercept, reflecting that this relationship is not worrisome. Finally, it is important to remark that the estimated variances of the estimators have been diminished, except in the case of the residualized variable, which remains constant.

It may be interesting to compare these results with the results obtained from ridge regression, which is widely applied to estimate models with collinearity. For this, a value of k was selected that results in a VIF value of less than 10 (VIF $=9.948$, calculated by following [55]), which is k = 0.047, see Table 3. Note that in this case, the signs obtained are not the expected ones. Furthermore, ridge regression does not allow any conclusion to be drawn about the global characteristics of the model, the individual significance, the inference or the analysis of the isolated effect of both variables, ${\mathbf{c}}_{13}$ and ${\mathbf{c}}_{26}$.

In addition, Figure 1 compares the estimations in terms of essential near-collinearity (the nonessential collinearity is not analyzed since it is not worrisome in this model), showing that the VIF value for ridge regression for different values of k is always lower than that of the OLS estimation but higher than the VIF obtained by residualization.

Figure 1.

VIF values obtained by ridge regression of the Wooldridge model (monthly data). k in steps of 0.001.

6.2. The ecological model

The second example is based on the STIRPAT model that is usually applied in environmental economics to observe the influence of some social and economic variables on the atmospheric impact of a country or a group of countries. It can be defined as the stochastic version of the IPAT identity [11,12], which identifies the impact on the atmosphere as a function of the population, the affluence measured from the gross domestic product (GDP), and one or more technology variables (usually variables related to industries, see e.g. [41] or [42]. Ehrlich and Holdren, who were the first researchers to study the IPAT identity [15–17], were aware of the problem of the relationship between variables in this identity, and although collinearity is likely to appear, most STIRPAT applications have disregarded it (see, e.g. [2,9,10,13,24,33,42,45,46,48–50,60]). However, there have been some efforts to address collinearity in STIRPAT models (see [14,18,20,27,37,41,51,61]).

This paper applies the STIRPAT model to data from China ( 1990–2014), the most polluting country in the world, as revealed by the World Bank, with a CO${}_2$ emissions value of 10291926.878 kilotonnes (kt) in 2014. The traditional specification of the STIRPAT model is

$$\mathbf{I}={\mathrm{\text{β}}}_1+{\mathrm{\text{β}}}_2\mathbf{P}+{\mathrm{\text{β}}}_3{\mathbf{A}}_{pc}+{\mathrm{\text{β}}}_4\mathbf{T}+\mathbf{u},$$ (20)

where $\mathbf{u}$ is the random disturbance, which is spherical; $\mathbf{I}$ represents CO${}_2$ emissions; $\mathbf{P}$ the total of population (billions); variable ${\mathbf{A}}_{pc}$ is the per capita GDP (expressed in trillions of constant 2010 US$); and finally, $\mathbf{T}$ is industrialization (% of GDP). The dataset has been extracted from the World Bank. However, in this paper, the following specification is proposed:

$$\mathbf{I}={\mathrm{\text{β}}}_1+{\mathrm{\text{β}}}_2\mathbf{P}+{\mathrm{\text{β}}}_3{\mathbf{e}}_{\mathbf{A}}+{\mathrm{\text{β}}}_4\mathbf{T}+\mathbf{u},$$ (21)

where ${\mathbf{e}}_{\mathbf{A}}$ are the residuals of the following auxiliary regression:

$$\mathbf{A}={\alpha }_1+{\alpha }_2\mathbf{P}+{\alpha }_3\mathbf{T}+\mathbf{v},$$ (22)

where $\mathbf{A}$ is the GDP (expressed in trillions of constant 2010 US$).

The use of model (21) instead of model (20) intends to overcome the following disadvantages:

• Traditionally, per capita GDP has been used to avoid the existing dependency between the GDP and the population. However, as the reader will see in the following correlation matrix, the linear relationship between per capita GDP and population is higher than the relationship between GDP and population. This means that the linear relationship is not mitigated but increased.

  $$\begin{array}{c}\\ \mathbf{I}\\ \mathbf{P}\\ {\mathbf{A}}_{pc}\\ \mathbf{A}\\ \mathbf{T}\end{array}\begin{array}{l}\begin{array}{rccrc}\mathbf{I}& \mathbf{P}& {\mathbf{A}}_{pc}& \mathbf{A}& \mathbf{T}\end{array}\\ \left(\begin{array}{ccccc}1.0000& & & & \\ 0.8896& 1.0000& & & \\ 0.9910& 0.9111& 1.0000& & \\ 0.9905& 0.9081& 0.9999& 1.0000& \\ -0.5464& -0.6194& -0.6296& -0.6320& 1.0000\end{array}\right)\end{array}$$

• In STIRPAT studies, ${\mathbf{A}}_{pc}$ (per capita GDP) is usually taken as the variable that represents affluence. However, the use of ${\mathbf{A}}_{pc}$ presents a disadvantage. From an interpretative point of view, a very important issue of the economy is ignored when using the per capita GDP (the ratio between GDP and population) since the distribution of income and the level of development of each region of the country are disregarded when all people are considered equal in terms of earnings. An increase in the per capita GDP does not necessarily mean the country is more developed; it may also indicate that the richest people in the country have increased their income.

In the residualization procedure, the relationship between GDP and population is deleted (in this case, the relationship between GDP and industrialization, variable $\mathbf{T}$, is also deleted), and, on the other hand, it is assumed that all people have the same income. Indeed, ${\mathbf{e}}_{\mathbf{A}}$ coincides with the part of GDP that has no relationship with population and industrialization, as has been discussed. If ${\mathbf{A}}_{pc}$ could be interpreted as a tool to measure the enrichment of the people and not the enrichment of the country, ${\mathbf{e}}_{\mathbf{A}}$ would be interpreted as a tool that measures whether the countries, and not the people, are richer in economic terms that are unrelated to industrialization.

Furthermore, in model (21), nonessential collinearity represents a significant issue for this empirical example (see [56]) since $\mathrm{C}\mathrm{V}\left(\mathbf{P}\right)=0.053$ and $\mathrm{C}\mathrm{V}\left(\mathbf{T}\right)=0.026$. To mitigate the nonessential collinearity, the variables population and technology will be centered. Thus the following model is proposed:

$$\mathbf{I}={\mathrm{\text{β}}}_1+{\mathrm{\text{β}}}_2{\mathbf{P}}^{\ast }+{\mathrm{\text{β}}}_3{\mathbf{e}}_{\mathbf{A}}+{\mathrm{\text{β}}}_4{\mathbf{T}}^{\ast }+\mathbf{u},$$ (23)

where ${\mathbf{P}}^{\ast }=\mathbf{P}-\overline{\mathbf{P}}$ and ${\mathbf{T}}^{\ast }=\mathbf{T}-\overline{\mathbf{T}}$.

With this model, it is verified that the degree of the existing near-multicollinearity is not worrisome. The values of VIF are lower than the threshold of 10 (VIF${}_{\mathbf{P}}=1.622$, VIF${}_{{\mathbf{e}}_{\mathbf{A}}}=1.000$ and VIF${}_{\mathbf{T}}=1.622$) and the value of CN is lower than 30 $(\mathrm{C}\mathrm{N}=2.063)$.

The results obtained by using OLS estimation of models (20) and (23) are shown in Table 5. The reader will observe the following:

• In model (20), the intercept has a coefficient that is significantly different from zero and has a negative value. This means that if population and GDP were null, CO${}_2$ emissions would be negative. This situation is corrected with model (23).

• In model (20), the estimated coefficient for population is not significantly different from zero; by contrast, in model (23), the coefficient is significant, and it has a positive value. This means that when the population increases, the CO${}_2$ emissions also increase. This is in line with economic theory and the correlation matrix.

• In models (20) and (23), the GDP coefficient (obtained from ${\mathbf{A}}_{pc}$ and ${\mathbf{e}}_{\mathbf{A}}$, respectively) is significantly different from zero and has a positive value. However, the interpretations of the two estimated coefficients are different. Thus, in model (23), the conclusion is that the increase in the wealth of the country when the production of goods and services is unrelated to industrialization entails an increase in CO${}_2$ emissions.

• In model (20), the estimated coefficient for industrialization is significant and has a positive value. It is contrary to the sign expected by observing the correlation matrix; however, this situation is corrected by model (23), in which this coefficient is not significantly different from zero.

Table 5. Results of STIRPAT models (20) and (23).

	Model (20)		Model (23)
Intercept	−10,287,191 *	Intercept	5,405,875 ***
	(3,667,784)		(53,166)
$\mathbf{P}$	−1,790,259	${\mathbf{P}}^{\ast }$	35,883,988 ***
	(1,861,596)		(1,004,251)
${\mathbf{A}}_{pc}$	1,837,647	${\mathbf{e}}_{\mathbf{A}}$	1,300,875 ***
	(77,813)		(57,278)
$\mathbf{T}$	409,211 ***	${\mathbf{T}}^{\ast }$	24,250
	(80,784)		(82,153)
$R^2$	0.9924		0.9918
F statistic	918.2		851.2
p-value (of F)	$<2.2\cdot {10}^{-16}$		$<2.2\cdot {10}^{-16}$

***, **, * mean the coefficient is statistically significant at 0.001 (99.9% level of confidence), at 0.01 (99% level of confidence) and at 0.05 (95% level of confidence), respectively.

7. Conclusion

The estimation and inference of the multiple linear regression model estimated by the residualization procedure has been exhaustively developed in this paper and the results compared with those of the original model. In this sense, it is important to point out that the application of residualization leads to conclusion about a model different to the original even though they have several identical characteristics (such as the variance estimation of the random perturbation, the coefficient of determination or the significance statistics).

The main contributions when applying this technique are:

• The new interpretations of the coefficients. The residualized model can answer questions that could not be answered with the initial model.

• The possibility of reducing the degree of collinearity in the initial model.

This paper proposes different criteria to select the variable to be residualized and the option of a successive residualization is also revealed.

Finally, it is relevant to note that residualization is not always applicable because the interpretations of the new estimated coefficients are not always simple. This also occurs with the application of other well-known techniques, such as ridge regression, PCR or PLSR, whose interpretation of estimated parameters is also controversial. A Monte Carlo simulation was performed to conclude that ridge regression, PCR or PLSR presents a prediction capability better than residualization but not better than OLS. Other alternative methodologies, such as raise regression (see [21] and [23]), could also be recommended.

Disclosure statement

No potential conflict of interest was reported by the authors.