Ridge regression and its applications in genetic studies

Abstract

With the advancement of technology, analysis of large-scale data of gene expression is feasible and has become very popular in the era of machine learning. This paper develops an improved ridge approach for the genome regression modeling. When multicollinearity exists in the data set with outliers, we consider a robust ridge estimator, namely the rank ridge regression estimator, for parameter estimation and prediction. On the other hand, the efficiency of the rank ridge regression estimator is highly dependent on the ridge parameter. In general, it is difficult to provide a satisfactory answer about the selection for the ridge parameter. Because of the good properties of generalized cross validation (GCV) and its simplicity, we use it to choose the optimum value of the ridge parameter. The GCV function creates a balance between the precision of the estimators and the bias caused by the ridge estimation. It behaves like an improved estimator of risk and can be used when the number of explanatory variables is larger than the sample size in high-dimensional problems. Finally, some numerical illustrations are given to support our findings.

Introduction

High-dimensional statistical inference is essential whenever the number of unknown parameters is larger than sample size. Typically, high-throughput technology provides large-scale data of gene expressions in transcriptomics. As an example, the riboflavin production data set with Bacillus subtilis (Lee et al. [1] and Zamboni et al. [2]) includes the logarithm of the riboflavin production rate as the response variable along with 4088 covariates which are the logarithm of the expression levels of 4088 genes, which are normalized using the Affymetrix oligonucleotide arrays normalizing methods. One rather homogeneous data set exists from 71 samples that were hybridized repeatedly during a fed-batch fermentation process in which different engineered strains and strains grown under different fermentation conditions were analyzed.

A relevant family of methods for prediction of the response based on the high dimensional gene expression data are sparse linear regression models. The least absolute shrinkage and selection operator (LASSO), proposed by Tibshirani [3], is the most popular, while other relevant methods are SCAD penalization [4] and minimum concave penalty [5]. In spite of the suitable sparsity caused by these penalized methods, they have low prediction performance for high dimensional data sets, because of their shrinkage and bias. Hence, developing shrinkage strategies to improve prediction is an interest in genome studies.

The primary aim of this study is improving the prediction accuracy of the riboflavin production data, in genome regression modeling; secondly, we further focus on detecting outliers. Intuitive methods for labeling observations as outliers can be provided by diagnostic plots. Fig 1 gives the diagnostics plots to identify outliers for the riboflavin data set based on the ordinary least-squares model with effective genes. The plots suggest there exist some outliers in the data set. Hence, developing efficient robust estimation strategy is another aspect of our approach.

Fig 1. Diagnostic plots for the riboflavin production data set.

Rank regression

Jureckova [6] and Jaeckel [7] proposed rank-based estimators for linear models as highly efficient and robust methods to outliers in response space. In short, rank regression is a simple technique which consists of replacing the data with their corresponding ranks. Rank regression and related inferential methods are useful in situations where

1. the relation between the response and covariate variables is nonlinear and monotonic and a simple and practical nonlinear form is of interest rather than polynomial, spline, kernel and/or other forms

2. there are outliers present in the study and we need a nonparametric robust procedure

3. the mere presence of so many important input variables makes it difficult to think in terms to find an appropriate parametric nonlinear model

The package Rfit in R, developed by Kloke and McKean [8] is a convenient and efficient tool for estimation/testing, diagnostic procedures and measures of influential cases in rank regression.

Rank estimator

Consider the setting where observed data are realizations of ${\left\{({\mathit{X}}_i,y_i)\right\}}_{i=1}^n$ with p-dimensional covariates X_i ∈ R^p and univariate continuous response variables y_i ∈ R. A simple regression model has form

$$\begin{array}{c}\hfill y_i={\mathit{X}}_i^{\top }\mathit{\beta }+{ϵ}_i,\end{array}$$ (1)

where β is the vector of regression coefficients and ϵ_i is the i^th error component. For simplicity, we assume that the intercept is zero. In case it exists, by centering the observations one can eliminate it from the study.

We assume that:

1. Errors ϵ = (ϵ₁, …, ϵ_n)^⊤ are independently and identically distributed (i.i.d.) random variables with (unknown) cumulative distribution function (c.d.f.) F having absolutely continuous probability density function (p.d.f.) f with finite and nonzero Fisher information

   $$\begin{array}{c}\hfill 0<I\left(f\right)={\int }_{-\infty }^{+\infty }{[-\frac{f^{\prime }\left(x\right)}{f\left(x\right)}]}^{2}f\left(x\right)\mathrm{d}x<\infty .\end{array}$$
2. For obtaining the linear rank estimator, we consider the score generating function $\psi :(0,1)\to \mathbb{R}$ which is assumed to be non constant, nondecreasing, and square integrable on (0, 1). The scores are defined in either of the following ways:

   $$\begin{array}{c}\hfill a\left(i\right)=\mathrm{E}\psi \left({\mathrm{U}}_{\mathrm{i}:\mathrm{n}}\right)\text{or}\mathrm{a}\left(\mathrm{i}\right)=\psi \left(\frac{\mathrm{i}}{\mathrm{n}+1}\right),\mathrm{i}=1,\dots ,\mathrm{n}\end{array}$$

   for n ≥ 1, where U_1:n ≤ … ≤ U_{n: n} are order statistics from a sample of size n from the uniform distribution $\mathcal{U}(0,1)$.

To obtain the rank estimate of β, define the pseudo-norm

$$\begin{array}{c}\hfill {\parallel \mathit{v}\parallel }_{\psi }=\sum _{i=1}^{n}a\left(R\left(v_i\right)\right)v_i\end{array}$$ (2)

where for y = (y₁, …, y_n)^⊤, a(R(y)) = (a(R(y₁)), …, a(R(y_n)))^⊤ and R(y_i) is the rank of y_i, i = 1, …, n and a(1)≤a(2)≤…≤a(n). Then, the rank-estimate of β is given by

$$\begin{array}{ll}{\widehat{\mathit{\beta }}}_{\psi }\hfill & =\text{arg}\text{min}\parallel y-\mathit{X}\mathit{\beta }{\parallel }_{\psi }\hfill \\ \hfill & ={\left({\mathit{X}}^{\top }\mathit{X}\right)}^{-1}{\mathit{X}}^{\top }{\widehat{y}}_{\psi },\hfill \end{array}$$ (3)

where X = (X₁, …, X_n)^⊤ and ${\widehat{\mathit{y}}}_{\psi }$ is the minimizer of dispersion function D_ψ(η) = ‖y − η‖_ψ over $\eta ϵ\mathcal{C}\left(\mathit{X}\right)$, where $\mathcal{C}\left(\mathit{X}\right)$ is the column space spanned by the columns of X. Thus, ${\widehat{\mathit{\beta }}}_{\psi }$ is the solution to the rank-normal equations X^⊤ a(R(y − Xβ)) = 0 and $D_{\psi }\left(\mathit{X}{\widehat{\mathit{\beta }}}_{\psi }\right)={\parallel \mathit{y}-{\widehat{\mathit{y}}}_{\psi }\parallel }_{\psi }$. Refer to the “S1 File” for a simple example of rank estimator, the form of ψ, and references.

Regularization methods

Under situations in which the matrix X^⊤ X is singular, the usual estimators are not applicable. From a practical point of view, high empirical correlations among two or a few other covariates lead to unstable results for estimating β or for pursuing variable selection. To overcome this problem, the ridge version for estimation can be considered. Ridge estimation is a regularization technique initially introduced by Tikhonov [9] and followed by Hoerl and Kennard [10] to regularize parameters or linear combination of them in order to provide acceptable estimators with less variability than the usual estimator, in multicollinear situations (see [8, 11–14] for more details). On the other hand, when the response distribution is non-normal or there are some outliers present in the study, the usual least squares and maximum likelihood methods fail to provide efficient estimates. In such cases, there is a need to develop an estimation strategy which is applicable in multicollinear situations and has acceptable efficiency in the presence of outliers.

Our contribution

Our contribution is two fold. First, for the situations where both multicollinearity and outliers exist we develop a shrinkage estimation strategy based on the rank ridge regression estimator. This creates two tuning parameters that must be optimized. Then, we define a new generalized cross validation (GCV) criterion to select the induced tuning parameters. The GCV has been applied to obtain the optimal ridge parameter in a ridge regression model by Golub et al. [15] and to obtain the optimal ridge parameter and bandwidth of the kernel smoother in semiparametric regression model by Amini and Roozbeh [16] as well as in partial linear models by Speckman [17]. Here, we use the GCV criterion for selecting the optimal values of ridge and shrinkage parameters, simultaneously. Our proposed GCV criterion creates a balance between the precision of the estimators and the biasedness caused by the ridge and shrinkage parameters.

The following section provides a robust shrinkage estimator based on the improved rank-based test statistic with developing a generalized cross validation (GCV) criterion, to obtain optimal values of tuning parameters. Subsequently, application of the proposed improved estimation method is illustrated for two real world data sets and an extensive simulation study to demonstrate usefulness of the proposed improved methodology. Finally, our study is concluded.

Methodology

In this section, we first define a robust test-statistic for testing the null hypothesis ${\mathcal{H}}_o:\mathit{\beta }=\mathbf{0}$ in the rank-regression analysis. This test is further employed in the construction of a robust rank-based shrinkage estimator. Then, we consider the rank ridge regression estimator and by the aid of the proposed test-statistic, we define the Stein-type shrinkage estimator for β for robust analysis. Since this estimator will have two tuning parameters, we evaluate these parameters using a generalized cross validation (GCV) criterion.

Robust shrinkage estimator

In order to define the robust rank-based shrinkage estimator, we need to develop a robust test statistic to test the following set of hypotheses

$$\begin{array}{c}\hfill {\mathcal{H}}_o:\mathit{\beta }=\mathbf{0}vs{\mathcal{H}}_A:\mathit{\beta }\ne \mathbf{0}.\end{array}$$ (4)

Denote the i^th element of H = X(X^⊤ X)⁻¹ X^⊤, the projection matrix onto the space $\mathcal{C}\left(\mathit{X}\right)$, by h_iin. We also need the following regularity conditions to be held.

• (A1) lim_{n → ∞}max_1≤i≤n h_iin = 0, where h_iin is commonly called the leverage of the i^th data point.

• (A2) ${lim}_{n\to \infty }\frac{1}{n}{\mathit{X}}^{\top }\mathit{X}=\Sigma$, where Σ is a p × p positive definite matrix.

Theorem 1 Let

$$\begin{array}{c}\hfill R_n\left(k\right)={\sigma }_a^{-2}{\mathit{a}}^{\top }\left(R\left(\mathit{y}\right)\right)\mathit{X}{(n^{-1}{\mathit{X}}^{\top }\mathit{X}+k{\mathit{I}}_p)}^{-1}{\left[\mathit{X}\left(k\right)\right]}^{-1}{(n^{-1}{\mathit{X}}^{\top }\mathit{X}+k{\mathit{I}}_p)}^{-1}{\mathit{X}}^{\top }\mathit{a}\left(R\left(\mathit{y}\right)\right),\end{array}$$ (5)

where X(k) is an invertible matrix given by

$$\begin{array}{c}\hfill \mathit{X}\left(k\right)={(\frac{1}{n}{\mathit{X}}^{\top }\mathit{X}+{\mathit{I}}_p)}^{-1}-\left[k{(\frac{1}{n}{\mathit{X}}^{\top }\mathit{X}+{\mathit{I}}_p)}^{-1}{(\frac{1}{n}{\mathit{X}}^{\top }\mathit{X}+{\mathit{I}}_p)}^{-1}\right],\end{array}$$ (6)

${\sigma }_a^2=\frac{1}{n-1}{\sum }_{j=1}^n{a}^2\left(j\right)\doteq 1$, and k > 0. Assume (A1) and (A2). Then, reject ${\mathcal{H}}_o$ in favor of ${\mathcal{H}}_A$ at approximate level α iff $R_n\left(k\right)\ge {\chi }_p^2\left(\alpha \right)$, where ${\chi }_p^2\left(\alpha \right)$ denotes the upper level α critical value of χ² distribution with p d.f.

Proof 1 Refer to the “S1 File”.

Now, using a similar approach in formulating the ridge estimator, we use the following rank ridge regression estimator (Roozbeh et al. [18])

$$\begin{array}{ccc}\hfill {\widehat{\mathit{\beta }}}_{\psi }\left(k\right)& =& {(\frac{1}{n}{\mathit{X}}^{\top }\mathit{X}+k{\mathit{I}}_p)}^{-1}{\mathit{X}}^{\top }{\widehat{\mathit{y}}}_{\psi },\hfill \end{array}$$ (7)

where k > 0 is the ridge parameter.

In order to improve upon the rank ridge regression estimator, following Saleh [19], we use the Stein-type shrinkage estimator (SSE) as

$$\begin{array}{ll}{\widehat{\mathit{\beta }}}_{\psi }^{\left(S\right)}\left(k,d\right)\hfill & =\left(1-\frac{d}{R_n\left(k\right)}\right){\widehat{\mathit{\beta }}}_{\psi }\left(k\right)\hfill \\ \hfill & ={\widehat{\mathit{\beta }}}_{\psi }\left(k\right)-d{R}_n{\left(k\right)}^{-1}{\widehat{\mathit{\beta }}}_{\psi }\left(k\right),d>0.\hfill \end{array}$$ (8)

The SSE shrinks the coefficients towards the origin using the test statistic R_n(k). The amount of shrinkage is controlled by the shrinkage coefficient d.

In the following result we show that the SSE is a shrinkage estimator with respect to the l_q-norm, ${\parallel \mathit{a}\parallel }^q={\left({\sum }_{j=1}^n{\left|a_j\right|}^q\right)}^{1/q}$, q > 0, with a = (a₁, …, a_n)^⊤. The reason we take the l_q-norm is that we can simultaneously take l₁ and l₂ norms into consideration. One must consider l₁-norm keeps the scale of observation, however, l₂-norm is mathematically tractable.

Theorem 2 ${\widehat{\mathit{\beta }}}_{\psi }^{\left(S\right)}(k,d)$ is a shrinkage estimator under l_q-norm under some regularity conditions as stated below

• (i): Under the set of local alternatives ${\mathcal{K}}_n:\mathit{\beta }=n^{-\frac{1}{2}}\delta$, with δ = (δ₁, …, δ_p)^⊤, δ_i ≠ 0, i = 1, …, p, we have $\parallel {\widehat{\mathit{\beta }}}_{\psi }^{\left(S\right)}{(k,d)\parallel }^q<{\parallel {\widehat{\mathit{\beta }}}_{\psi }\left(k\right)\parallel }^q$.

• (ii) For k > n/2, d > 0, we have

  $$\begin{array}{c}\hfill \parallel {\widehat{\mathit{\beta }}}_{\psi }^{\left(S\right)}{(k,d)\parallel }^q<{\parallel (1-\frac{d}{R_n\left(k\right)}){\widehat{\mathit{y}}}_{\psi }\parallel }^q.\end{array}$$

• (iii) Assume λ_i = o(n), i = 1, …, n. For k > sup_{1≤i ≤ n} λ_i, (ii) holds in limit.

Proof 2 Refer to the “S1 File”.

The proposed SSE may be criticized since it depends on the two tuning parameters k and d and it may come to mind why we need an estimator with two tuning parameters, when we have the rank ridge regression estimator. In what follows we elaborate more on the advantages of the SSE ${\widehat{\mathit{\beta }}}_{\psi }^{\left(S\right)}(k,d)$ in our analysis. Accepting the fact that we need a robust rank estimator, apart from the justifications provided in Saleh [19], we give the following reasons.

1. Apparently, as d → 0, ${\widehat{\mathit{\beta }}}_{\psi }^{\left(S\right)}(k,d)\to {\widehat{\mathit{\beta }}}_{\psi }\left(k\right)$ and thus for small values d the gain in estimation is just the information provided by the robust ridge parameter, even if the null hypothesis ${\mathcal{H}}_o$ is not true. Thus, even if we agree that the rank ridge regression estimator shrink the coefficients to zero, the information provided by the test statistic R_n(k), which is controlled by d in the SSE, is useful.

2. Consider a situation in which we do not have strong evidence to reject the null hypothesis. Knowing the fact that the ridge estimator does not select variables (see Saleh et al. [20]), we can not estimate the zero vector using the rank ridge regression estimator, however, the shrinkage coefficient d maybe obtained such that for a given k, d = R_n(k) and the resulting shrinkage estimator becomes equal to zero. This might be a rare event, but theoretically sounds.

3. The last but not the least, for the set of local alternatives ${\mathcal{K}}_n$, as in Theorem 2, the proposed SSE shrinks more than the rank ridge regression estimator. Thus in order to have robust shrinkage estimator, the SSE with two tuning parameters is preferred.

The SSE depends on both the ridge parameter k and shrinkage parameter d. For optimization purposes, we use the GCV of Roozbeh et al. [18] in the forthcoming section.

Generalized cross validation

The GCV chooses the ridge and shrinkage parameters by minimizing an estimate of the unobservable risk function

$$\begin{array}{ccc}\hfill \mathbf{R}(\mathit{\beta };{\widehat{\mathit{\beta }}}_{\psi }^{\left(S\right)}(k,d))& =& \frac{1}{n}{(\mathit{X}\mathit{\beta }-{\widehat{\mathit{y}}}_{\psi }^{\left(S\right)}(k,d))}^{\top }(\mathit{X}\mathit{\beta }-{\widehat{\mathit{y}}}_{\psi }^{\left(S\right)}(k,d))\hfill \\ \hfill & =\hfill & \frac{1}{n}{\parallel \mathit{X}\mathit{\beta }-\mathit{X}{\widehat{\mathit{\beta }}}_{\psi }^{\left(S\right)}(k,d)\parallel }^2,\hfill \end{array}$$

where

$$\begin{array}{ccc}\hfill {\widehat{\mathit{y}}}_{\psi }^{\left(S\right)}(k,d)& =& \mathit{X}{\widehat{\mathit{\beta }}}_{\psi }^{\left(S\right)}(k,d)\hfill \\ & =& (1-\frac{d}{R_n\left(k\right)})2{\widehat{\tau }}_{\psi }\mathit{X}{(n^{-1}{\mathit{X}}^{\top }\mathit{X}+k{\mathit{I}}_p)}^{-1}{\mathit{X}}^{\top }\mathit{y}\hfill \\ & =& \mathit{L}(k,d)\mathit{y},\hfill \end{array}$$ (9)

with $\mathit{L}(k,d)=(1-\frac{d}{R_n\left(k\right)})2{\widehat{\tau }}_{\psi }\mathit{X}{(n^{-1}{\mathit{X}}^{\top }\mathit{X}+k{\mathit{I}}_p)}^{-1}{\mathit{X}}^{\top }$, termed as the hat matrix of $\mathit{y}$, and ${\widehat{\tau }}_{\psi }$ is a consistent estimator (see Hettmansperger and McKean [21]) for the scale parameter τ_ψ given by

$$\begin{array}{c}\hfill {\tau }_{\psi }^{-1}=\int \psi \left(u\right){\psi }_f\left(u\right)du,{\psi }_f\left(u\right)=-\frac{f^{\prime }\left(F^{-1}\left(u\right)\right)}{f\left(F^{-1}\left(u\right)\right)}.\end{array}$$

It is straightforward to show that (see Hettmansperger and McKean [21] and Roozbeh [22])

$$\begin{array}{ccc}\hfill \mathrm{E}\left(R\right(\mathit{\beta };{\widehat{\mathit{\beta }}}_{\psi }^{\left(S\right)}(k,d)\left)\right)& =& \frac{1}{n}\parallel ({\mathit{I}}_n-L(k,d))\mathit{X}\mathit{\beta }{\parallel }^2+\frac{{\tau }_{\psi }^2}{n}\text{tr}\left(\mathit{L}{(k,d)}^2\right)\hfill \\ & =& b^2(k,d)+{\tau }_{\psi }^2{\mu }_2(k,d),\hfill \end{array}$$

where $b^2(k,d)=\frac{1}{n}\parallel ({\mathit{I}}_n-\mathit{L}(k,d))\mathit{X}\mathit{\beta }{\parallel }^2$ and ${\mu }_2(k,d)=\frac{1}{n}\text{tr}\left(\mathit{L}{(k,d)}^2\right)$.

The GCV function is then defined as

$$\begin{array}{ccc}\hfill \text{GCV}\left({\widehat{\mathit{\beta }}}_{\psi }^{\left(S\right)}(k,d)\right)& =& \frac{\frac{1}{n}\parallel ({\mathit{I}}_n-\mathit{L}(k,d))\mathit{y}{\parallel }^2}{{(1-\frac{1}{n}\text{tr}(\mathit{L}(k,d)\left)\right)}^2}\hfill \\ & =& \frac{\frac{1}{n}\parallel ({\mathit{I}}_n-\mathit{L}(k,d))\mathit{y}{\parallel }^2}{{(1-{\mu }_1(k,d))}^2},\hfill \end{array}$$ (10)

where ${\mu }_1(k,d)=\frac{1}{n}\text{tr}\left(\mathit{L}\right(k,d\left)\right)$.

Corollary 3 Suppose that the eigenvalues {λ_νn, ν = 1, …, n} of XX^⊤ satisfy λ_νn ≃ nν^−m for some m > 1. Then, for the GCV function in (10)

$$\begin{array}{ccc}\hfill \underset{n\to \infty }{lim}\mathrm{E}\left(\text{GCV}\right({\widehat{\mathit{\beta }}}_{\psi }^{\left(S\right)}(k,d)\left)\right)& =& {\tau }_{\psi }^2+\underset{n\to \infty }{lim}\mathrm{E}\left(R\right(\mathit{\beta };{\widehat{\mathit{\beta }}}_{\psi }^{\left(S\right)}(k,d)\left)\right).\hfill \end{array}$$ (11)

Corollary 3 is an application of the GCV theorem of Craven and Wahba (1979) and Golub et al. (1979). It implies that the minimizer of $\mathrm{E}\left(\text{GCV}\right({\widehat{\mathit{\beta }}}_{\psi }^{\left(S\right)}(k,d)\left)\right)$ is essentially equivalent to the minimizer of $\mathrm{E}\left(R\right(\mathit{\beta };{\widehat{\mathit{\beta }}}_{\psi }^{\left(S\right)}(k,d)\left)\right)$ for SSE. Based on (11), $\text{GCV}\left({\widehat{\mathit{\beta }}}_{\psi }^{\left(S\right)}(k,d)\right)$ is an estimator of $\mathrm{E}\left(R\right(\mathit{\beta };{\widehat{\mathit{\beta }}}_{\psi }^{\left(S\right)}(k,d)\left)\right)$ with a nearly constant bias. Using the techniques of Section 3, this can be shown to be an estimator of ${\tau }_{\psi }^2$ with positive but asymptotically negligible bias, so the resulting “F” statistic can be expected to be conservative. The main result of this section is to obtain a good estimate of the minimizer of $\mathrm{E}\left(R\right(\mathit{\beta };{\widehat{\mathit{\beta }}}_{\psi }^{\left(S\right)}(k,d)\left)\right)$ from the data which does not require knowledge of ${\tau }_{\psi }^2$ so that, by minimizing it, we can extract the optimal values for the two tuning parameters simultaneously.

Applications

In this section we consider some numerical experiments to illustrate the usefulness of the suggested improved methodology in the regression model. We analyze the performance of the proposed estimators in a real-world examples related to the riboflavin production.

Application to riboflavin production data set

To support our assertions, we consider the data set about riboflavin (vitamin B2) production in Bacillus subtilis, which can be found in R package “hdi”. There is a single real valued response variable which is the logarithm of the riboflavin production rate and p = 4088 explanatory variables measuring the logarithm of the expression level of 4088 genes. There is one rather homogeneous data set from n = 71 samples that were hybridized repeatedly during a fed batch fermentation process where different engineered strains and strains grown under different fermentation conditions were analyzed.

Fig 2 shows the normal Q–Q plot based on the ridge estimation for the riboflavin production data set. Also, the bivariate boxplot for selected genes of this data is depicted in Fig 3. The bivariate boxplot is a two-dimensional analogue of the boxplot for univariate data. This diagram is based on calculating robust measures of location, scale, and correlation; it consists essentially a pair of concentric ellipses, one of which (the hinge) includes 50% of the data and the other (called the fence) delineates potentially troublesome outliers. In addition, robust regression lines of both response on predictor and vice versa are shown, with their intersection showing the bivariate location estimator. The acute (large) angle between the regression lines will be small (large) for a large (small) absolute value of correlations. Figs 2 and 3 clearly reveals that the data contains some outliers.

Fig 2. Q–Q plot based on the ridge estimator for the riboflavin production data set.

Fig 3. Bivariate boxplot of the riboflavin production data set for effective genes.

We use GCV to select the the ridge and shrinkage parameters for the proposed estimators, simultaneously. Similar to the SSE, the GCV score functions for $\widehat{\mathit{\beta }}\left(k\right)$ and ${\widehat{\mathit{\beta }}}_{\psi }\left(k\right)$ can be procured by setting

$$\begin{array}{c}\hfill {\mathit{L}}_1\left(k\right)=\mathit{X}{(n^{-1}{\mathit{X}}^{\top }\mathit{X}+k{\mathit{I}}_p)}^{-1}{\mathit{X}}^{\top },{\mathit{L}}_2\left(k\right)=2{\widehat{\tau }}_{\psi }\mathit{X}{(n^{-1}{\mathit{X}}^{\top }\mathit{X}+k{\mathit{I}}_p)}^{-1}{\mathit{X}}^{\top },\end{array}$$

in (10), respectively.

All computations were conducted using the statistical software R 3.4.3 to develop the package Rfit for calculating the proposed estimators, their test statistics and powers described in this paper. The R codes are available at https://mahdiroozbeh.profile.semnan.ac.ir/#downloads. The 3D diagram as well as the 2D slices of GCV of ${\widehat{\mathit{\beta }}}_{\psi }^{\left(S\right)}(k,d)$ versus k and d are plotted in Fig 4 for the riboflavin production data set. As it can be seen from Fig 4, the 2D (3D) diagrams of GCV are convex functions (surfaces) and hence they have a global minimum. This guarantees the existence of optimum values of k and d which minimize the GCV’s. The minimum of $\text{GCV}\left({\widehat{\mathit{\beta }}}_{\psi }^{\left(S\right)}(k,d)\right)$ approximately occurs at k_opt = 0.468759 and d_opt = 0.002332. We test the hypothesis ${\mathcal{H}}_o:\mathit{\beta }=\mathbf{0}$ using the ridge rank-based (RRB) test statistic. The test statistic for ${\mathcal{H}}_o$, given our observations, is R_n(k_opt) = 27.42. Thus, we conclude that there is not enough evidence to reject the null hypothesis ${\mathcal{H}}_o$ and so, the SSE can be efficient for prediction purposes according to the second comment right after Theorem 2.

Fig 4. The diagram of $\text{GCV}\left({\widehat{\mathit{\beta }}}_{\psi }^{\left(S\right)}(k,d)\right)$ and its counter plot versus k and d for the riboflavin production data set.

To measure the prediction accuracy of proposed estimators, the leave-one-out cross-validation (CV) criterion was used, which is defined by

$$\begin{array}{c}\hfill \text{CV}\left(\widehat{\mathit{\beta }}\right)=\frac{1}{\mathrm{n}}\sum _{\mathrm{s}=1}^{\mathrm{n}}{({\mathit{y}}_{(-\mathrm{s})}-{\mathit{X}}_{(-\mathrm{s})}{\widehat{\mathit{\beta }}}_{(-\mathrm{s})})}^2,\end{array}$$

where ${\widehat{\mathit{\beta }}}_{(-s)}$ is obtained by replacing $\mathit{X}$ and $\mathit{y}$ with ${\mathit{X}}_{(-s)}=\left({\tilde{x}}_{jk(-s)}\right)$, 1 ≤ k ≤ n, 1 ≤ j ≤ p, ${\mathit{y}}_{(-s)}={({\tilde{y}}_{1(-s)},\dots ,{\tilde{y}}_{n(-s)})}^{\top }$, ${\tilde{x}}_{lk(-s)}=x_{lk}-{\sum }_{j\ne s}^n{W}_{nj}\left(t_s\right)x_{lj}$, ${\tilde{y}}_{k(-s)}=y_k-{\sum }_{j\ne s}^n{W}_{nj}\left(t_s\right)y_j$. Here ${\mathit{y}}_{(-s)}$ is the predicted value of response variable where sth observation left out of the estimation of the β.

Table 1 displays a summary of the results. In this Table, a goodness of fit criterion R-squared is calculated for comparing the proposed estimators using the following formula

$$\begin{array}{c}\hfill {\mathrm{R}}^2\left(\widehat{\mathit{\beta }}\right)=1-\frac{\text{SSE}\left(\widehat{\mathit{\beta }}\right)}{{\mathrm{S}}_{yy}},\end{array}$$

where $\text{SSE}\left(\widehat{\mathit{\beta }}\right)={\sum }_{i=1}^n{({\widehat{y}}_i-\overline{y})}^2$ for ${\widehat{y}}_i={\mathit{X}}^{\top }\widehat{\mathit{\beta }}$ and ${\mathrm{S}}_{yy}={\sum }_{i=1}^n{(y_i-\overline{y})}^2$. From Table 1, it is seen that ${\widehat{\mathit{\beta }}}_{\psi }^{\left(S\right)}(k,d)$ performs better than ridge regression, since it offers smaller GCV and bigger R-squared values in the presence of multicollinearity and outliers. Moreover, because of the existence of outliers in the data set, it can be seen that R-squared’s of robust type estimators are more acceptable than the R-squared of non-robust type estimator.

Table 1. Evaluation of proposed estimators for the riboflavin production data set.

Estimator	$\widehat{\mathit{\beta }}\left(k\right)$	${\widehat{\mathit{\beta }}}_{\psi }\left(k\right)$	${\widehat{\mathit{\beta }}}_{\psi }^{\left(S\right)}(k,d)$
$\text{CV}$	13.10023	9.88070	8.01023
$\text{min}\left(\text{GCV}\right)$	22.88144	17.00815	14.23133
${\mathrm{R}}^2$	0.707080	0.759485	0.798853

For further illustrative purposes, we analyze some simulated data sets in the forthcoming section.

Monte-Carlo simulation

In this section, we perform some Monte-Carlo simulation studies to justify our assertions as well as examining the performance of the proposed estimators. As pointed and explained in Section 1, high-dimensional case p > n causes the matrix X^⊤ X to be ill-conditioned. To accommodate ill-conditioning, apart from generating multicollinear data, we will evaluate how our estimators work for the high-dimensional case p > n.

We also examine the robustness of the proposed estimators in the presence of contaminated data. The regressors are drawn a new in every replication. The efficiencies of ${\widehat{\mathit{\beta }}}_i$ relative to ${\widehat{\mathit{\beta }}}_1$ are defined by

$$\begin{array}{c}\hfill \text{eff}({\widehat{\mathit{\beta }}}_i,{\widehat{\mathit{\beta }}}_1)=\frac{\frac{1}{M}{\sum }_{m=1}^M{\parallel {\widehat{\mathit{\beta }}}_1\left(m\right)-\mathit{\beta }\parallel }^2}{\frac{1}{M}{\sum }_{m=1}^M{\parallel {\widehat{\mathit{\beta }}}_i\left(m\right)-\mathit{\beta }\parallel }^2},i=2,3,\end{array}$$ (12)

where M is the number of iterations and ${\widehat{\mathit{\beta }}}_i\left(m\right)$ is the ith estimator of β in the mth stage.

To examine the performance of the proposed estimators, we perform a Monte-Carlo simulation. To achieve different degrees of collinearity, following McDonald and Galarneau [23] and Gibbons [24] the explanatory variables were generated for (n, p) = {(180, 60), (180, 120)} (low-dimensional) and (n, p) = {(200, 240), (200, 360), (250, 10000)} (high-dimensional) from the following model:

$$\begin{array}{c}\hfill x_{ij}={(1-{\gamma }^2)}^{\frac{1}{2}}{z}_{ij}+\gamma z_{ip},i=1,2,\dots ,n,j=1,2,\dots ,p,\end{array}$$ (13)

where z_ij are independent standard normal pseudo-random numbers and γ is specified so that the correlation between any two explanatory variables is given by γ². These variables are then standardized so that X^⊤ X and X^⊤ y are in correlation forms. Two different sets of correlation corresponding to γ = 0.20, 0.50, 0.90 and 0.95 are considered. The observations for the dependent variable are determined by

$$\begin{array}{c}\hfill \mathit{y}=\mathit{X}\mathit{\beta }+ϵ,\end{array}$$ (14)

in sparse case: β_i, i = 1, …, 0.1p are generated from standard normal distribution and β_i = 0, i > 0.1p; non-sparse case: β_i, i = 1, …, 0.2 × p are generated from standard normal distribution and β_i = 0, i > 0.2p. Also, we considered $ϵ={({ϵ}_1^{\top },{ϵ}_2^{\top })}^{\top }$ where

$$\begin{array}{ccc}& & {ϵ}_{1(h\times 1)}\sim N_h(\mathbf{0},{\sigma }^2\mathit{V}),{\sigma }^2=0.44,v_{ij}=exp(-9|i-j\left|\right),\hfill \\ & & {ϵ}_{2\left(\right(n-h)\times 1)}{\sim }^{{}^{i.i.d.}}{\chi }_1^2\left(8\right),\hfill \end{array}$$

where ${\chi }_m^2\left(\delta \right)$ is the non-central chi-squared distribution with m degrees of freedom and non-centrality parameter δ. The main reason of selecting such structure for errors is to contaminate the data and evaluate the robustness of the estimators. We set the first h error terms as dependent normal random variables and the last (n − h) error terms as independent non-central chi-squared random variables. The non-centrality causes the outliers to lie on one side of the true regression model which then pulls the non-robust estimation toward them.

The Monte-Carlo simulation is performed with M = 10³ replications, obtaining the proposed estimators ${\widehat{\mathit{\beta }}}_1=\widehat{\mathit{\beta }}\left(k\right),{\widehat{\mathit{\beta }}}_2={\widehat{\mathit{\beta }}}_{\psi }\left(k\right)$ and ${\widehat{\mathit{\beta }}}_3={\widehat{\mathit{\beta }}}_{\psi }^{\left(S\right)}(k,d)$, in the sparse and non-sparse regression models.

To save space, the Tables have been reported in the S1 File of this paper and their results have been briefly shown by Fig 5. In the “S1 File”, we provided 12 tables (S2-S9 Tables in S1 File) to extensively analyze the numerical outputs. However, to save space here, we only report Fig 5 as an abstract of tables’ results. Fig 5 summarizes the empirical type I errors and powers at a 5% significance level under low-dimensional (based on F test statistic) and high-dimensional (based on R_n(k) test statistics) settings for γ = 0.20, 0.50, 0.90 and 0.95, respectively. The contaminated sample is the percentage of the sample contaminated with outliers ($\text{CS}=100\times \frac{n-h}{n}\%$). The F-test is valid when p is less than n. Please note in Tables S10-S13 Tables in “S1 File”, we numerically estimated the risks and efficiencies of the proposed estimators relative to ${\widehat{\mathit{\beta }}}_1$.

Fig 5. The diagram of empirical type I error and power for different (n, p) for the simulated data sets.

We apply the generalized cross-validation (GCV) method to select the optimal ridge parameter (k_opt) and shrinkage parameter (d_opt), which minimizes the GCV function. Since the results were similar across cases, to save space we report here only the results for the sparse case with γ = 0.95, n = 200, p = 240 and CS = 20%. For this case, the minimum of GCV approximately occurred at k_opt = 5.90 and d_opt = 0.006254 for the model (14). The 3D diagram as well as the 2D slices of GCV versus k and d are plotted in Fig 6. Fig 7 shows the results of the F test and RRB test for the non-sparse and sparse cases with parameter values γ = 0.95, n = 180, p = 120, CS = 40% and significance level α = 5% (there are a total of 25 realizations). Each point in the plot corresponds to one realization of this configuration.

Fig 6. The diagram of GCV and its counter plot versus k and d for the simulated data set.

Fig 7. Comparison between F test and RRB test for the non-sparse case (left diagram) and sparse case (right diagram) with γ = 0.95, n = 180, p = 120 and CS = 40%.

Comparison results based on the simulations are similar: Firstly, we observe the empirical type I errors for the F test, are not reasonable in comparison with the significance level α = 0.05, while the powers are slightly smaller than RRB test in most cases. Secondly, the RRB test is highly efficient for all cases under consideration. Its sizes are reasonable, while its powers compared to F test is high, in the low dimensional settings. Thirdly, RRB test is powerful in the high-dimensional settings, as we would expect.

As demonstrated, the RRB test is overly conservative. In other words, it achieves smaller empirical type I error and bigger empirical power compared to the F test.

Summary & conclusions

In this paper, we proposed a robust ridge test statistic to improve the predictions in a regression model. In the presence of multicollinearity and outliers, we introduced robust ridge type estimator and improved it by shrinking toward the origin to incorporate the information contained in the null-hypothesis. By defining a generalized cross validation criterion optimal values of ridge and shrinkage parameters obtained simultaneously. Figs 3 and 6 showed the global minimum archived by this criterion. Through nonlinear minimization algorithms, we found the global minimum of this criterion with respect to both parameters. Finally, a Monte-Carlo simulation study as well as a real data example were considered to compare performances of the proposed estimators numerically.

According to Fig 5 and the detailed tabulated numerical results in the “S1 File”, we observed that the proposed robust ridge-type test statistic is more powerful than the classical F test in the presence of multicollinearity and outliers. Moreover, we found that efficiencies of robust type estimators with respect to non-robust type increase when the percentage of outliers increases. Another factor affecting the efficiency of the estimators was the number of explanatory variables. We seen the estimator ${\widehat{\mathit{\beta }}}_{\psi }^{\left(S\right)}(k,d)$ is leading to be the best estimator among others, since it offers smaller risk and bigger efficiency values in all cases. Moreover, $\widehat{\mathit{\beta }}\left(k\right)$ was not a suitable estimator in the presence of outliers, especially, for the high percentage of outliers. For the real examples, from Table 1, we deduced ${\widehat{\mathit{\beta }}}_{\psi }^{\left(S\right)}(k,d)$ is quite efficient in the sense that it has significant value of goodness of fit.

Supporting information

S1 File

(PDF)

Data Availability

All used data sets are available with the R code uploaded to GitHub https://github.com/M-Arashi/R-codes-PONE-D-20-08942.

Funding Statement

This study was supported in part by Iran National Science Foundation (INSF) in the form of a grant awarded to M. Arashi (97019472), the National Research Foundation (NRF) of South Africa SARChI Research Chair (UID:71199, Ref.:IFR170227223754) in the form of a grant awarded to M. Arashi (109214), and the University of Malaya Research in the form of grants awarded to N.A. Hanzah and M. Roozbeh (RP009B-13AFR, IIRG009C-19FNW).