Robust Logistic and Probit Methods for Binary and Multinomial Regression

Abstract

In this paper we introduce new robust estimators for the logistic and probit regressions for binary, multinomial, nominal and ordinal data and apply these models to estimate the parameters when outliers or inluential observations are present. Maximum likelihood estimates don't behave well when outliers or inluential observations are present. One remedy is to remove inluential observations from the data and then apply the maximum likelihood technique on the deleted data. Another approach is to employ a robust technique that can handle outliers and inluential observations without removing any observations from the data sets. The robustness of the method is tested using real and simulated data sets.

Introduction

Binary and multinomial regressions are commonly used by medical scientists and researchers for analysis of binary or polytomous outcomes. These methods are routinely used as diagnostic tools in all areas of medicine including oncology and cardiology. Zhou et al. [1] used logistic regression to relate the gene expression with class labels. They also used logistic regression for their microarray-based analysis of cancer classification and prediction. Sator et al. [2] applied a logistic regression model to identify enriched biological groups in gene expression microarray studies. Majid et al. [3] performed logistic regression analysis to predict endoscopic lesions in iron deficiency anemia when there are no gastrointestinal symptoms.

Morris et al. [4] applied multinomial regression technique to analyze the sub-phenotypes by allowing for heterogeneity of genetic effects. Richman et al. [5] investigated the association between European ancestry and renal disease when compared with African Americans, East Asians, and Hispanics. They concluded that European ancestry is protective against the development of renal disease in systematic lupus erythematosus. Their data had some outliers but they were excluded in their final analysis. Timmerman et al. [6] used the logistic regression to distinguish between benign and malignant adnexal mass before surgery. Merritt et al. [7] used the binary and multinomial logistic regressions to investigate the role of dairy food intake and risk of ovarian cancer. The validity of estimation and testing procedures used in the analysis of binary data are heavily dependent on whether or not the model assumptions are satisfied. The maximum likelihood method of estimating binary regression parameters using logistic, probit and many other methods is extremely sensitive to outliers and influential observations.

There is a large literature on the robustness issue of the binary regression. Most of the existing methods attempt to achieve robustness by down weighting observations which are far from the majority of the data, that is, outliers. The reader is referred to papers published by Pregibon [8], Carroll and Pederson [9], and Bianco and Yohai [10]. Bianco and Martinez [11] modified the original score functions of the logistic regression to obtain bounded sensitivity, which is a concept introduced by Morgenthaler [12] using the L¹-norm instead of the L²-norm in the likelihood, resulting in a weighted score function of the original score function. Cantoni and Ronchetti [13] focused on robustness of inference rather than the model. Pregibon [8] suggested resistant fitting methods which taper the standard likelihood to reduce the influence of extreme observations. Kordzakhia et al. [14] introduced a robust logistic regression by minimizing the mean-squared deviance for the worst case contamination. Bergesio and Yohai [15] introduced projection estimators for generalized linear model. These estimators have the same asymptotic normal distribution as the M-estimators. Hobza et al. [16] introduced a median estimator to estimate the parameters of the logistic regression.

Robust binary and multinomial regression estimators for analysis of biomedical data are proposed. This robust method has a bounded influence and high breakdown point and efficiency under normal distribution and is able to estimate the parameters of logistic and probit regression models. The proposed model is computationally simple and can easily be used by researchers.

Binary Regression Model

Consider the model y_i=π(x_i;β)+ε_i where ε₁,ε₂,…, ε_n are independent random variables with E(ε_i)=0 and Var(ε_i)=π(x_i;β)(1−π(x_i;β)), and y₁,y₂,…,y_n are n independent Bernoulli random variables with Ey_i=π(x_i;β) and Var(y_i)=π(x_i;β)(1−π(x_i;β)) such that the conditional success probability is given by P(y_i=1| x_i)=π(x_i;β) and x_i=(x_i0,x_i01,…,x_ip)^t; 1≤i≤n is a p+1 dimensional vector of predictor variables with β=(β₀,β₁,…,β_p)^t as the parameters vector.

There are various estimation methods for the estimation of the parameter vector β. The most commonly used method is the logistic regression which is used to analyze the effects of explanatory variables on the binary response y. In the logistic regression the link function π_L(x_i;β) is assumed to have the following functional form

$${\pi }_L(x_i;\beta )=\frac{1}{1+Exp(-\sum _{j=0}^p{\beta }_j{x}_{ij})}$$

The logistic transformation of π_L(x_i;β) is called the logit function and is given by

$$\mathrm{logit}\left({\pi }_L(x_i;\beta )\right)=Log\frac{{\pi }_L(x_i;\beta )}{1-{\pi }_L(x_i;\beta )}=\sum _{j=0}^p{\beta }_j{x}_{ij}$$

The probit function is a link function of the form

$${\pi }_P(x_i;\beta )=\frac{1}{2}[Erf\left(\frac{\sum _{j=0}^p{\beta }_j{x}_{ij}}{\sqrt{2}}\right)+1]’$$

Where Erf function is defined as

$$Erf\left(x\right)=\frac{2}{\sqrt{\pi }}{\int }_0^x{e}^{-t^2}dt$$

which has the probit transformation function

$$probit\left({\pi }_P(x_i;\beta )\right)=\sqrt{2}Er{f}^{-1}(-1+2{\pi }_P(x_i;\beta ))=\sum _{j=0}^p{\beta }_j{x}_{ij}$$

The tabaistic model introduced by Tabatabai and Argyros [17] has the link function:

$${\pi }_T(x_i;\beta )=\frac{1}{1+arcsinh\left[Exp[-\sum _{j=0}^p{\beta }_j{x}_{ij}]\right],}$$

Where arcsinh (.) represents the inverse hyperbolic sine function.

The tabaistic transformation function is called the tabit function and is defined as

$$tabit\left({\pi }_T(x_i;\beta )\right)=Log\left(Csch\left(\frac{1-{\pi }_T(x_i;\beta )}{{\pi }_T(x_i;\beta )}\right)\right)=\sum _{j=0}^p{\beta }_j{x}_{ij}$$

Where Csch (.) denotes the hyperbolic cosecant function and the complementary log-log model link function has the form

$${\pi }_{CLL}(x_i;\beta )=1-Exp[-\left[Exp\left[\sum _{j=0}^p{\beta }_j{x}_{ij}\right]\right]]$$

with the complementary log-log transformation function (cllogit) defined as

$$\mathrm{cllogit}\left({\pi }_{CLL}(x_i;\beta )\right)=Log[-\left[Log[1-{\pi }_{CLL}(x_i;\beta )]\right]]=\sum _{j=0}^p{\beta }_j{x}_{ij}$$

Figure 1 shows the graph of π_L, π_CLL, π_P and π_T where of π_L, π_CCL, π_P and π_T take values between zero and one. The solid curve is the graph of π_L function, the dotted curve is the graph of π_P function, the dot-dashed curve is the graph of π_CLL function, and the dashed curve is the graph of π_T function. Figure 2 shows the graph of logit (π_L). cllogit (π_CLL), probit(π_P), and tabit(π_T). The solid curve, the dotted curve, the dot-dashed curve and the dashed curve are the graph of logit (π_L) function, the graph of probit(π_P) function, the graph of cllogit (π_CLL) function, and the graph of tabit(π_T) function, respectively. The principle of maximum likelihood is ordinarily used to estimate the model parameters by maximizing the log-likelihood function of the form

$$LL\left(\beta \right)=\sum _{i=1}^n[y_{i}Log\left(\pi (x_i;\beta )\right)+(1-y_i)Log(1-\pi (x_i;\beta ))]$$

Figure 1.

Graphs of π_L>, π_CLL, π_P and π_T functions.

Figure 2.

Graphs of logit (π_L). cllogit (π_CLL), probit (π_P), and tabit (π_T) functions.

In other words, the estimate $\widehat{\beta }$ of β is

$$\widehat{\beta }=\underset{\beta \in {\mathbb{ℝ}}^{p+1}}{\mathbf{arg}\mathbf{max}}\left(LL\left(\beta \right)\right)$$

Although the maximum likelihood estimator is asymptotically efficient, it is not recommended as a method of choice when outliers are present. The alternative techniques are robust statistical methods.

Tabatabai et al. [18] defined the one parameter family of differentiable functions ρ_ω(x) of the form ρ_ω(x)=1−Sech(ωx), where the positive real number ω is called the tuning constant.

The bounded function ρ_ω: R→ R is a differentiable function satisfying the following properties:

$$\begin{array}{cc}\hfill & {\rho }_{\omega }\left(0\right)=0\hfill \\ \hfill & \underset{x\to \infty }{\lim }{\rho }_{\omega }\left(x\right)=\underset{x\to -\infty }{\lim }{\rho }_{\omega }\left(x\right)=1\hfill \\ \hfill & (\forall x)(x\in \mathbb{ℝ}\Rightarrow {\rho }_{\omega }\left(x\right)\ge 0)\hfill \\ \hfill & (\forall x){(x\in \mathbb{ℝ}\Rightarrow {\rho }_{\omega }\left(x\right)={\rho }_{\omega }(-x))}_{,}\hfill \\ \hfill & (\forall a)(\forall b)(a\in \mathbb{ℝ}\wedge b\in \mathbb{ℝ}\wedge \mid a\mid >\mid b\mid \Rightarrow {\rho }_{\omega }\left(a\right)\ge {\rho }_{\omega }\left(b\right))\hfill \\ \hfill & (\forall \kappa )(\kappa >0\Rightarrow \underset{x\to \infty }{\lim }\frac{{\rho }_{\omega }\left(\kappa x\right)}{{\rho }_{\omega }\left(x\right)}=1)\hfill \\ \hfill & \underset{\mid x\mid \to \infty }{\lim }\frac{d{\rho }_{\omega }\left(x\right)}{dx}=0\hfill \end{array}$$

Under the normality assumption for the error term ε_i, the asymptotic efficiency (Aeff) is defined as

$$Aeff=\frac{{\left(E\left[{\psi }_{\omega }^{\prime }\left(t\right)\right]\right)}^2}{E\left[{\psi }_{\omega }^2\left(t\right)\right]}$$ (1)

Where ψ_ω is the derivative of ρ_ω and is equal to

$${\psi }_{\omega }\left(x\right)=\omega Sech\left(\omega x\right)Tanh\left(\omega x\right),$$

Where Sech and Tanh represent the hyperbolic secant and hyperbolic tangent, respectively.

$$\begin{array}{cc}\hfill & E\left[{\psi }_{\omega }^{\prime }\left(t\right)\right]={\int }_{-\infty }^{\infty }{\psi }_{\omega }^{\prime }\left(t\right)\frac{e^{-\frac{t^2}{2}}}{\sqrt{2\pi }}dt\mathrm{with}\hfill \\ \hfill & E\left[{\psi }_{\omega }^2\left(t\right)\right]={\int }_{-\infty }^{\infty }{\psi }_{\omega }^2\left(t\right)\frac{e^{-\frac{t^2}{2}}}{\sqrt{2\pi }}dt\hfill \end{array}$$

The tuning constant ω can be calculated by solving the following equation (1) for ω. The numerical values for ω at the efficiency levels 0.80, 0.85, 0.90, and 0.95 are approximately 0.721, 0.628, 0.525 and 0.405, respectively. Although the choice for tuning constant ω is left for the investigator to decide, we do recommend an efficiency of approximately 90 percent which corresponds to ω=1/2. We now consider the hat matrix of the form

$$H=X{\left(X^{t}X\right)}^{-1}{X}^t,$$

where X is the design matrix defined as

$$X=\left(\begin{array}{ccc}\hfill x_{10}\hfill & \hfill x_{11}\dots \hfill & \hfill x_{1k}\hfill \\ \hfill x_{20}\hfill & \hfill x_{21}\dots \hfill & \hfill x_{2k}\hfill \\ \hfill x_{n0}\hfill & \hfill x_{n1}\dots \hfill & \hfill x_{nk}\hfill \end{array}\right)=\left(X_0{X}_1\dots X_K\right).$$

If the model has intercept, then the column vector X₀ has the form

$$X_0=\left(\begin{array}{c}\hfill 1\hfill \\ \hfill .\hfill \\ \hfill .\hfill \\ \hfill .\hfill \\ \hfill 1\hfill \end{array}\right)$$

For j=1,2,…,k, define

$$M_j=Median\{\mid x_{1_j}\mid ,\mid x_{2_j}\mid ,\dots ,x_{nj}\mid \}$$

and for i=1,2,…,n, define

$$L_i=\sum _{j=1}^{k}Max\{M_j,\mid x_{ij}\mid \}$$

and for ω>0 define the function G_ω(u) as

$$G_{\omega }\left(u\right)={\int }_0^u{\psi }_{\omega }(-\ln \left(t\right))dt.$$ (2)

The estimator

$$\widehat{\beta }=\underset{\beta \in {\mathbb{ℝ}}^{p+1}}{\mathbf{arg}\mathbf{min}}\sum _{i=1}^n\frac{(1-h_{ii})[{\rho }_c\left(d_i\right)+s\left({\beta }^t{x}_i\right)]}{L\left(i\right)}$$ (3)

Where d_i = y_i ln[π(x_i ;β)]+(1−y_i)ln[1−π(x_i;β)]

$$s\left(t\right)=G\left(\pi \left(t\right)\right)+G(1-\pi \left(t\right))-G\left(1\right)$$

and h_ii is the ith diagonal element in the hat matrix. For the logistic model, we have

$$\pi \left(t\right)=\frac{1}{1+Exp(-t)}.$$

So that the above integral (2) can be written as

$$G_{\omega }\left(u\right)=\frac{2u^{1+\omega }H(1,\frac{1+\omega }{2\omega },\frac{1}{2}(3+\frac{1}{\omega }),-u^{2\omega })}{1+\omega }-uSech\left(\omega \ln \left(u\right)\right)’$$

Where H(k₁,k₂,k₃,t) is the Gauss hypergeometric function 2F1 with parameter k₁,k₂ and k₃. If ω=1, then we have

$$G_1\left(u\right)=u^{2}H(1,1,2,-u^2)-uSech\left(\ln \left(u\right)\right)=\frac{2u^2}{1+u^2}-\ln (1+u^2),$$

and if ω=1/2, then we have

$$G_{1∕2}\left(u\right)=-4\sqrt{u}+4Arc\tan \left(\sqrt{u}\right)+uSech\left(\frac{\ln \left(u\right)}{2}\right).$$

Define the Hessian matrix H_b for binary data as

$$H_b=\left(\begin{array}{ccc}\hfill \frac{{\partial }^{2}LL\left(\beta \right)}{\partial {\beta }_0^2}\hfill & \hfill \dots \hfill & \hfill \frac{{\partial }^{2}LL\left(\beta \right)}{\partial {\beta }_0{\beta }_p}\hfill \\ \hfill ⋮\hfill & \hfill \ddots \hfill & \hfill ⋮\hfill \\ \hfill \frac{{\partial }^{2}LL\left(\beta \right)}{\partial {\beta }_p^2}\hfill & \hfill \dots \hfill & \hfill \frac{{\partial }^{2}LL\left(\beta \right)}{\partial {\beta }_p{\beta }_0}\hfill \end{array}\right)$$

Then an estimate of the variance-covariance matrix for vector $\widehat{\beta }\mathrm{is}\widehat{Var\left(\widehat{\beta }\right)}={(-H_p)}^{-1}$ with an estimated variance σ² given by

$${\widehat{\sigma }}^2=\frac{\sum _{i=1}^n[{(y_i-\pi (x_i;\widehat{\beta }))}^2∕\pi (x_i;\widehat{\beta })].}{n-(p+1)}$$

To perform hypothesis testing, we let $\Omega \subseteq R^p$ be the parameter space and {β_j1, β_j2,…β_jq} be a subset of {β₀,β₁,…β_p}. Define

$${\Omega }_0=\{\beta \in \Omega :{\beta }_{j_1}={\beta }_{j_2}=\dots {\beta }_{j_q}=0\}.$$

To test the following hypothesis

$H_0:\beta \in {\Omega }_0$ against the alternative $H_1:\beta \in {\Omega }_0^c$, one can use the Wald type test statistic which is defined as

$$W_n^2=n({\widehat{\beta }}_{j_1},{\widehat{\beta }}_{j_2},\dots ,{\widehat{\beta }}_{j_q})V_q^{-1}{({\widehat{\beta }}_{j_1},{\widehat{\beta }}_{j_2},\dots ,{\widehat{\beta }}_{j_q})}^t.$$

The null distribution of the statistic $W_n^2$ is asymptotically a chi-square distribution with q degrees of freedom.

Robust Multinomial Logistic Regression Model

In this section we generalize the robust binary method to multinomial regression where the response y includes k categories. When k=2, this model reduces to the binary regression. Now, consider the response matrix Y given by

$$Y=\left(\begin{array}{ccc}\hfill y_{11}\hfill & \hfill \dots \hfill & \hfill y_{ik}\hfill \\ \hfill ⋮\hfill & \hfill \ddots \hfill & \hfill ⋮\hfill \\ \hfill y_{n1}\hfill & \hfill \dots \hfill & \hfill y_{nk}\hfill \end{array}\right),$$

where

$$y_{ij}=\{\begin{array}{cc}1\hfill & if{y}_i=j\hfill \\ 0\hfill & if{y}_i\ne j\hfill \end{array}\phantom{\}}.$$

This means that y_ij=1whenever the ith response is in category j.

For i =1,2,…,n, ${\sum }_{j=1}^k{y}_{ij}=1$ and x_i=(x_i1,x_i2,…,x_ip)^t. The parameter vector

β_j =(β_j0,β_j1,…,β_jp)^t and the vector β₀ is a zero vector in a p+1 dimensional space with β=(β₀,β₁,…,β_k−1)^t. The multinomial likelihood function of parameter vector β is defined by

$$ML\left(\beta \right)=\prod _{i=1}^n\prod _{j=1}^k\left[{\left({\pi }_j(x_i;\beta )\right)}^{y_{ij}}\right],$$

and the multinomial log- likelihood function is

$$MLL\left(\beta \right)=\sum _{i=1}^n(y_{i1}\ln \left[{\pi }_1(x_i;\beta )\right]+y_{i2}\ln \left[{\pi }_2(x_i;\beta )\right]+\dots +y_{ik}\ln [1-\sum _{s=1}^{k-1}{\pi }_s(x_i;\beta )]).$$

For the generalized logit when the first category is the designated reference category and the intercepts are β₀₁,β₀₂,…,β_0k−1, we get

$${\pi }_1(x_i;\beta )=P(y_i=1\mid x_i;\beta )=\frac{1}{1+\sum _{s=2}^{k}Exp[-{\eta }_s(x_i;\beta )]},$$

and for j = 2,…,k,

$${\pi }_j(x_i;\beta )=P(y_i=j\mid x_i;\beta )=\frac{Exp[-{\eta }_j(x_i;\beta )]}{1+\sum _{s=2}^k[Exp[-{\eta }_s(x_i;\beta )]},$$

and for j = 1,…,k−1 the logit function η_j(x_j;β) is the log-odds of membership in category j versus the reference category 1 and is equal to

$${\eta }_j(x_i;\beta )=\ln \left(\frac{{\pi }_j(x_i;\beta )}{{\pi }_1(x_i;\beta )}\right)={\beta }_{0j}+{\beta }_{1j}{x}_{i1}+{\beta }_{2j}{x}_{i2}+\dots +{\beta }_{pj}{x}_{ip}.$$

The principle of maximum likelihood can be used to estimate model parameters. The maximum likelihood estimate of the parameters vector β is

$$\widehat{\beta }=\underset{\beta \in {\mathbb{ℝ}}^{p+1}}{\mathbf{arg}\mathbf{max}}\sum _{i=1}^n{d}_i^{\ast },$$ (4)

where

$$d_i^{\ast }=\sum _{j=1}^k{y}_{ij}\ln ({\pi }_j(x_i;\beta )$$

The robust estimate of the model parameters is given by

$$\widehat{\beta }=\underset{\beta \in {\mathbb{ℝ}}^{p+1}}{\mathbf{arg}\mathbf{min}}\sum _{i=1}^n\frac{(1-h_{ij})[{\rho }_{\omega }\left(d_i^{\ast }\right)+\sum _{j=1}^k{G}_{\omega }\left({\pi }_j(x_i;\beta )\right)-G_{\omega }\left(1\right)]}{L\left(i\right)}$$ (5)

Define the Hessian matrix H_p as

$$H_p=\left(\begin{array}{ccc}\hfill \frac{{\partial }^{2}MLL\left(\beta \right)}{\partial {\beta }_0^2}\hfill & \hfill \dots \hfill & \hfill \frac{{\partial }^{2}MLL\left(\beta \right)}{\partial {\beta }_0{\beta }_p}\hfill \\ \hfill ⋮\hfill & \hfill \ddots \hfill & \hfill ⋮\hfill \\ \hfill \frac{{\partial }^{2}MLL\left(\beta \right)}{\partial {\beta }_p^2}\hfill & \hfill \dots \hfill & \hfill \frac{{\partial }^{2}MLL\left(\beta \right)}{\partial {\beta }_p{\beta }_0}\hfill \end{array}\right)$$

Then, for the multinomial response, an estimate of the variance-covariance matrix for vector $\widehat{\beta }\mathrm{is}\widehat{Var\left(\widehat{\beta }\right)}={(-H_p)}^{-1}$ with an estimated variance σ² given by

$${\widehat{\sigma }}^2=\frac{\sum _{j=1}^k\sum _{i=1}^n[{(y_{ij}-{\pi }_j(x_i;\widehat{\beta }))}^2∕{\pi }_j(x_i;\widehat{\beta })].}{(n-(p+1))(k-1)}$$

For the cumulative logit model for ordinal k-category response, the cumulative probability for the ith response belongs to the response category less than or equal j is

$$F_j\left(x_i\right)=P(y\le j\mid x_i),$$

and for j=1,…,k−1 the ordinal logit O_j(x_j;β) is the log-odds of falling into or below category j against falling above it and is given by

$$O_j(x_i;\beta )=\ln \left(\frac{F_j\left(x_i\right)}{1-F_j\left(x_i\right)}\right)={\beta }_{0j}+{\beta }_1{x}_{i1}+{\beta }_2{x}_{i2}+\dots +{\beta }_p{x}_{ip},$$

where β₀₁ < β₀₂ < … < β_0k−1. The ordered logit model is sometimes called the proportional odds model.

Let ${\pi }_j^o(x_i;\beta )=P(y_i=j\mid x_i;\beta )$. Then we have

$${\pi }_j^o(x_i;\beta )=\{\begin{array}{cc}\frac{1}{1+Exp[-O_1(x_i;\beta )]}\hfill & ifj=1\hfill \\ \frac{1}{1+Exp[-O_j(x_i;\beta )]}-\frac{1}{1+Exp[-O_{j-1}(x_i;\beta )]}\hfill & if2\le j\le k-1\hfill \\ 1-\frac{1}{1+Exp[-O_{k-1}(x_i;\beta )]}\hfill & ifj=k\hfill \end{array}\phantom{\}}$$

and for the ordinal probit we have

$${\pi }_j^o(x_i;\beta )=\{\begin{array}{cc}\frac{1}{2}(1+Erf\left[\frac{O_1(x_i;\beta )}{\sqrt{2}}\right])\hfill & ifj=1\hfill \\ \frac{1}{2}(1+Erf\left[\frac{O_j(x_i;\beta )}{\sqrt{2}}\right])-\frac{1}{2}(1+Erf\left[\frac{O_{j-1}(x_i;\beta )}{\sqrt{2}}\right])\hfill & if2\le j\le k-1\hfill \\ 1-\frac{1}{2}(1+Erf\left[\frac{O_{k-1}(x_i;\beta )}{\sqrt{2}}\right])\hfill & ifj=k\hfill \end{array}\phantom{\}}$$

The robust estimate of ordinal multinomial parameter vector is giveb

$${\widehat{\beta }}^O=\underset{\beta \in {\mathbb{ℝ}}^{p+1}}{\mathbf{arg}\mathbf{min}}\sum _{i=1}^n\frac{(1-h_{ii})[{\rho }_{\omega }\left(d_i^O\right)+\sum _{j=1}^k{G}_{\omega }\left({\pi }_j^O(x_i;\beta )\right)-G_{\omega }\left(1\right)]}{L\left(i\right)}$$ (6)

where $d_i^O={\sum }_{j=1}^k{y}_{ij}\ln \left({\pi }_j^O(x_i;\beta )\right)$

Application

Vasoconstriction example

Vasoconstriction and vasodilation are two important physiological mechanisms used to control the circulation of blood throughout the body. These mechanisms directly affect both the blood pressure and the distribution of the blood in the body. Vasodilation refers to the expansion of blood vessels through relaxation of smooth muscles in the vessel walls. This allows increased flow of blood through these vessels and also decreases the blood pressure. Contraction of the same muscles tightens the blood vessels, which decreases blood flow and increases pressure. Thus, vasodilation and vasoconstriction work in opposition to adjust both blood flow and blood pressure. The usual controls for vasoconstriction and vasodilation are done by smooth muscles and autonomic nervous system, triggered by the medulla. These responses can also be affected by drugs promoting either constriction or dilation. Furthermore, there is a means of control by circulating hormones in the bloodstream, as well as control by intrinsic mechanisms to vasculature, called the myogenic response. The antagonistic operation of vasoconstriction and vasodilation is used by the body for numerous purposes. Primary among these is regulation of the supply of oxygen and nutrients to the cells of the body, to meet their needs. Furthermore, this regulation of blood flow is also needed for thermoregulation within the body. At times of increased metabolic needs or needs for oxygen in certain organs or systems in the body, the blood flow to these regions will also be modulated. Finally, vasoconstriction is also important in restricting blood flow to regions of the body in cases of traumatic injury.

The data set was analyzed originally by Finney [19]. It consists of 39 observations where the binary response variable y=1 or y=0 represent the presence or absence of vasoconstriction of the skin respectively. This experimental data set considers the effect of inhalation of air in a single deep breath on the presence or absence of vasoconstriction in the digits. Presence or absence of vasoconstriction is considered as a categorical variable, and the study considers the effect of two variables, the volume of inhaled air and the rate of inhalation. (Data has hidden outliers so that the robust logistic regression will be useful in its analysis.) In the remainder of this work, we denote by ML and BY, the Maximum Likelihood and Bianco-Yohai methods, respectively. By examining Table 1 we conclude that the new robust estimator has produced the closest parameter estimates to the maximum likelihood estimates when the outliers were removed. In addition, the ${\chi }_{arc}^2$ is the lowest for the new robust method. The ${\chi }_{arc}^2$ is defined by Kordzakhia et al. [14] as

$${\chi }_{arc}^2=\sum _{i=1}^{n}4{(\arcsin \sqrt{y_i}-\arcsin \sqrt{\pi (x_i;\widehat{\beta })})}^2$$

Table 1.

Parameter estimates for Vaso Data Using ML, BY and New Robust Method.

Methods	Coefficients			Standard Error
	b0	b1	b2	b0	b1	b2
ML	−2.887	5.191	4.578	1.324	1.869	1.843
ML (Influential observations removed )	(${\chi }_{arc}^2$=48.3118)
	−24.590	39.539	31.928	13.974	23.153	17.687
	(${\chi }_{arc}^2$ =32.0159)
BY c=1.25	−5.3214	8.4454	7.4801	9.7647	14.1903	12.3199
	(${\chi }_{arc}^2$ =43.1853)
New Robust c=0.5	−24.1191	38.4639	30.9608	15.3232	24.9410	19.2365
	(${\chi }_{arc}^2$ =32.2671)

Plasma example

The erythrocyte sedimentation rate (ESR) is very important. It is a common hematology test which is simple and inexpensive but can be used to detect infection or acute phase response, which can alert physicians to a wide variety of conditions. The test is very versatile and can assist physicians in detecting conditions from rheumatoid arthritis to systemic lupus erythematosus to multiple myeloma; however it is non-specific and is usually combined with other tests. In practice, ESR is used widely to test for a range of conditions, including inflammation, trauma, and malignant disease. Studies have also suggested the utility of the ESR among the elderly as a general indicator of level of sickness or disease. Recently ESR has also attracted attention for a potential role as a predictor for the development of cardio-vascular disease and heart failure.

The ESR simply measures the rate at which red blood cells precipitate during a period of one hour. Anticoagulated blood is placed in an upright tube, and the rate at which the erythrocytes settle is measured in mm per hour. Although the test is a direct measurement of rate of sedimentation, the balance between factors stimulating sedimentation and factors resisting sedimentation allows for a number of clinically relevant factors to influence this rate. Fibrinogen is the most important factor promoting sedimentation, and the high level of fibrinogen in the blood during the inflammatory process makes this test sensitive to inflammation. High levels of fibrinogen in the blood decrease the repulsive forces experienced between the negatively charged erythrocytes and favor the formation of rouleaux. These stacks of erythrocytes that stick together will settle faster and lead to an increased ESR. Other acute phase reactants, or other large molecules, especially when positively charged, can have a similar effect, although fibrinogen has been observed to have the largest effect.

A recent focus on the inflammatory nature of artherosclerosis has been accompanied by a recent study of increased levels of ESR and elevated risk of coronary heart disease. Erikssen et al. [20] observed that elevated ESR is a strong predictor of mortality from heart failure, suggesting it may serve as a marker for aggressive forms of coronary heart disease. Andresdottir et al. [21] observed an increased risk of coronary heart disease among the top quintile or ESR rates, with a hazard ratio of 1.57 for men and 1.9 for women. The 2005 paper of Inglesson et al. [22] also observes a significant association between elevated ESR and heart failure, suggesting both that inflammation is involved in the processes leading to heart failure and that the ESR may be used in evaluating this process. In addition to the well-established uses of ESR, Saadeh [23] mentions some potential new applications of this test such as bacterial otitis media, acute hematogenous osteomyelitis, AIDS, pelvic inflammatory disease, prostate cancer, and early prediction of stroke severity.

Although the ESR usually detects acute phase response from fibrinogen in blood in conditions such as those mentioned above, in certain cases there are factors which decrease the rate of sedimentation. One important factor that can slow the rate of sedimentation is irregularity in the erythrocytes, either in shape or unusually small size. As a consequence, ESR can detect certain blood diseases (including sickle cell anemia and spherocytosis) which lead to a lower than normal rate of sedimentation, as observed in Bridgen [24]. Other conditions that may also lower ESR include the extreme levels of white blood cells as observed in chronic lymphocytic leukemia. Furthermore the surplus of erythrocytes found in patients with polycythemia makes rouleau formation difficult and decreases the ESR.

In clinical applications the erythrocyte sedimentation rate may in many cases be treated as a categorical variable, with a normal ESR for values less than some given α and an elevated ESR for values greater than α. When representing such a set of data where ESR depends on one or more variables the logistic regression may be used. For instance in the data set from Collett [25], the ESR is considered as a function of two variables, the level of fibrinogen and the level of γ-globulin. The data for 32 individuals represents the levels of fibrinogen and γ-globulin in the blood and whether the ESR level is healthy (< 20 mm/hr) or unhealthy (≥ 20 mm/hr), and the logistic regression is used to describe how both fibrinogen and γ-globulin affect the ESR variable. Since this data set contains (hidden/influential) outliers, both the probit method of regression and the logit method do not give accurate results. However we observed that our new methods for robust logistic regression do represent the data accurately. The logit, when all 32 observations are included in the study, is given by

$$Logit\left(\widehat{\pi }\left(X_i\right)\right)=-6.845+1.827f_i$$

When one removes the influential observations 15, and 23, the logit model becomes

$$Logit\left(\widehat{\pi }\left(X_i\right)\right)=-59.62+17.46f_i$$

The level of γ-globulin was not a statistically significant variable to be included in the model. Thus only the level of fibrinogen f is used in the variable selection.

Again, examining Table 2 reveals that the new robust estimator has produced the closest parameter estimates to the maximum likelihood estimates when the outliers were removed as well as the lowest value for the ${\chi }_{arc}^2$.

Table 2.

Parameter Estimates for Plasma Data Using ML, BY and New Robust Method.

Methods	Coefficients		Standard Error
	b0	b1	b0	b1
ML	−6.8451	1.8271	2.7703	0.9009
ML (Influential observations removed )	(${\chi }_{arc}^2$ =38.9405)		45.51	13.50
	−59.62	17.46
	(${\chi }_{arc}^2$ =29.1415)
BY
c=1.25	−8.3774	2.2870	5.4383	1.6632
	(${\chi }_{arc}^2$ =37.0421)
New method c=0.5	−60.5094	17.7654	49.7325	14.7409
	(${\chi }_{arc}^2$ 29.2719)

Mental health example

The following example involves the ordinal multinomial regression. The data comes from a mental health study for a random sample of adult residents of Alachua County, Florida. This data was appeared in Agresti [26]. The mental impairment is divided into four categories (well, mild symptom formation, moderate symptom formation, and impaired). The explanatory variables are life events index X₁ and socioeconomic status X₂, where X₂ is binary and takes high and low levels. There is no outlier in this data. We just want to show how the method works even when outliers are not present. The logit is given by

$$logit\left[P(Y\le j)\right]={\alpha }_j+{\beta }_1{X}_1+{\beta }_2{X}_2$$

Table 3 gives the parameter values using maximum likelihood method as well as the new robust ordinal multinomial method. The R program for this example is provided in the Appendix.

Table 3.

Parameter estimates for Mental Health data using robust ordinal method.

Parameter	Estimated	Standard Error	Robust estimate	Standard Error
Intercept 1	−0.2819	0.6423	−.2374	0.7265
Intercept 2	1.2128	0.6607	1.1923	0.7507
Intercept 3	2.2094	0.7210	2.2981	0.8364
Life	−0.3189	0.1210	−.3181	0.1618
Socioeconomic status	1.1112	0.6109	1.1423	0.7789

Simulation

To evaluate the performance of the new robust method for logistic regression we conduct a Monte Carlo simulation. In the first round of our simulation, we use one explanatory variable and in the next round, we increase the number of explanatory variables to two. We first generate an independent random sample of size 100 from the standard normal distribution with mean 0 and standard deviation equal to 0.5. We call the variable x. Then we generate a sample of error terms ε_i of size 100 from the logistic distribution with mean zero and standard deviation equal to 1. The dependent variable y is generated using the formula

$$y_i=\{\begin{array}{cc}1if\hfill & 1+3x_i+{\epsilon }_i>0\hfill \\ 0if\hfill & 1+3x_i+{\epsilon }_i\le 0\hfill \end{array}\phantom{\}}.$$

The model parameters are 1 (intercept) and 3 (coefficient for x). We then select a random sample (5%) from the generated sample x and contaminate the selected sample by multiplying each x value by a factor of 10. Then we repeat the above procedures 1000 times. Finally, we estimate both the bias and mean squared errors using the following equations

$$bias=\mid \frac{\sum _{l=1}^m{\widehat{\theta }}_l}{m}-\theta \mid ,$$

where m is the number of iterations in the simulation. The mean squared error is estimated by

$$MSE=\frac{\sum _{l=1}^m{(\widehat{\theta }-\theta )}^2}{m}$$

For the two explanatory variables, we generate two independent normal random samples of size 100 from a normal distribution with mean 0 and standard deviation 0.5 and call them x₁ and x₂ respectively. Then we select 5% of this random sample (3% from x₁ and 2% from x₂) and multiply the selected samples by 10. Then we generate a sample of error terms ε_i of size 100 from the logistic distribution with mean zero and standard deviation equal to 1. The dependent variable y is generated using the formula

$$y_i=\{\begin{array}{cc}1if\hfill & 1+.5x_{1i}+2x_{2i}+{\epsilon }_i>0\hfill \\ 0if\hfill & 1+.5x_{1i}+2x_{2i}+\epsilon \le 0\hfill \end{array}\phantom{\}}$$

We calculate the parameter estimates and continue the iteration 1000 times. In addition, we calculate the bias and mean squared errors. Tables 4 and 5 show the results of simulations using ML, BY and the new robust method with one and two explanatory variables, respectively. For binary logistic regression the simulation results indicate that our new robust method is as good as the BY method. The BY method only covers binary logistic regression whereas our method not only covers binary but also covers multinomial regression for both nominal and ordinal responses.

Table 4.

Simulation Results for Logistic Regression (b₀=1, b₁=3, N=100, m=1000).

Method	Bias	MSE	Bias (5% x)	MSE (5%)
ML
b0	0.0430	0.0876	0.08065	0.07938
b1	0.1784	0..6310	1.00777	2.16368
BY
b0	0.0463	0.0923	0.0027	0.0842
b1	0.1857	0.6662	0.2336	0.8885
New Robust
b0	0.0342	0.0897	0.0125	0.0779
b1	0.1125	0.5321	0.2642	0.9771

Table 5.

Simulation Results for Logistic Regression (b0=1, b1=0.5, b2=2, N=100, m=1000).

Method	Bias	MSE	Bias (5%)	MSE (5%)
ML
b0	0.04283	0.08152	0.00312	0.07526
b1	0.01071	0.25287	0.19770	0.23694
b2	0.10448	0.38209	0.36612	0.75513
BY
b0	0.0485	0.0856	0.0232	0.0788
b1	0.0128	0.2650	0.1711	0.2456
b2	0.1143	0.4028	0.0909	0.5125
New Robust
b0	0.0410	0.0786	0.0270	0.0755
b1	0.0255	0.2799	0.1045	0.2676
b2	0.1034	0.4037	0.0553	0.5032

Discussion and Conclusions

In this work we have proposed a new robust method to analyze binary and multinomial regression models. We believe that these new robust methods for binary and multinomial regressions have potential to play a key role in modeling categorical data in medical, biological and engineering sciences. We have shown the lack of robustness of the maximum likelihood technique when outliers are present. In both real examples and simulated ones and when the outliers are present, the new robust method performed well. In conclusion the motivation was to introduce a new robust loss function of residuals which can attain high breakdown value. The method has high efficiency and high breakdown points with bounded influence function.