Logistic regression with image covariates via the combination of L₁ and Sobolev regularizations

Abstract

The use of image covariates to build a classification model has lots of impact in various fields, such as computer science, medicine, and so on. The aim of this paper is to develop an estimation method for logistic regression model with image covariates. We propose a novel regularized estimation approach, where the regularization is a combination of L₁ regularization and Sobolev norm regularization. The L₁ penalty can perform variable selection, while the Sobolev norm penalty can capture the shape edges information of image data. We develop an efficient algorithm for the optimization problem. We also establish a nonasymptotic error bound on parameter estimation. Simulated studies and a real data application demonstrate that our proposed method performs very well.

Introduction

As one of the most important issues in machine learning field, classification plays a prominent role throughout various disciplines. Until now people have developed a large number of classification methods, such as KNN, Linear (Quadratic) discriminant analysis, logistic regression, naive bayes, decision tree, SVM, neural network, deep learning, and many others [1, 2]. Among all those methods logistic regression has a long history [3], and is one of the most popular approaches. Logistic regression model is a typical representative of generalized linear models and linear classification methods. Therefore, this article takes logistic regression as the research object. Traditionally, maximum likelihood method is usually used to obtain an estimator of the parameter in logistic regression model [4–6].

However, the big data era brings us massive complex data, one of whose most prominent characteristics is high dimensionality. The maximum likelihood estimation method in logistic regression model faces serious problems such as non-existence, non-uniqueness [7] in high dimensional settings. Regularization is a popular strategy to handle high dimensional problems [8]. Many regularized methods have been proposed over the past decades, including LASSO [9], the smoothly clipped absolute deviation (SCAD) penalty method [10], the minimax concave penalty (MCP) method [11], and so on. For high dimensional logistic regression, [12] considered L₁-regularization path algorithm. [13] proposed an interior-point method for large-scale L₁-regularized logistic regression. [14] proposed the group lasso for logistic regression. The L_1/2 regularized logistic regression is considered by [15] for gene selection in cancer classification.

Image data is a very popular form of data, and generated in many fields, such as computer science, medicine, and so on. In addition to high dimensionality, image data usually contains spatially smooth regions with relatively sharp edges, which leads to its own characteristics including local smoothness [16], jump discontinuity [17], and many others. Local smoothness leads to highly correlated features, which makes the image classification problem more challenging [18]. Jump discontinuity makes conventional smoothing techniques inefficient [17]. On the other hand, using these characteristics in modeling process is often helpful for model efficiency enhancement, and has received a lot of attention recently. For example, [19] introduced a locally adaptive smoothing method for image restoration. [16] proposed Propagation-Separation approach for local likelihood estimation, which can handle local smoothness of image data. [20] developed an adaptive regression model for the analysis of neuroimaging data, which is a generalization of the PS approach. [21] studied theoretical performance of nonlocal means for noise removal of image data. [17] considered a spatially varying coefficient model for neuroimaging data with jump discontinuities. [18] proposed a spatially weighted principal component analysis (SWPCA) for imaging classification. [22] developed a generalized scalar-on-image regression models via total variation regularization, which can keep the piecewise smooth nature of imaging data. [23] proposed an efficient nuclear norm penalized estimation method for matrix linear discriminant analysis.

In this paper, we consider a logistic regression model with image covariates, and develop a regularized estimation approach, which combines the L₁ regularization and the Sobolev norm regularization. The L₁ penalty performs variable selection and removes covariates unrelated to the response from models [9]. The Sobolev norm penalty keeps characteristics of image data (e.g. local smoothness) in model fitting. In fact, the Sobolev regularization is a popular technology in image data analysis, such as image denoising [24], edge detection of images [25], and many others. The proposed regularization method is different from the aforementioned regularized logistic regression models. It is also different from the elastic net method [26], which is a combination of Lasso and ridge regression. The elastic net encourages the grouping effect, where strongly correlated predictors tend to be in or out of the model. However, the elastic net can not exploit structure information of image covariates, and is not suitable for models with image covariates. There are differences between our proposed method and the fused lasso method [27]. In many real data analysis, such as gene expression data, covariates have a order. Adjacent covariates are often highly correlated and have similar effects on the response variable. The fused lasso tends to make adjacent covariates share common effect on the response. The proposed method can be treated as the extended version of the fused lasso from one dimension to multidimensions. Moreover, the fusion term here is Sobolev norm penalty. Furthermore, we develop a novel algorithm to solve the optimization problem. The theoretical property of our estimator is also studied, and a nonasymptotic estimate error bound is given. Numerical studies including simulations and a real data analysis are also considered to verify the performance of our method.

The rest of the article is organized as follows. Section 2 presents the methodology, including model setup, algorithm, and theoretical property. Section 3 is numerical studies, where simulated studies and a real data application are presented. Lastly, we make a short conclusion in Section 4. The proof details of theoretical studies are put in Appendix Section.

Methodology

Model setup

Suppose that we have observations (X_i, Y_i) with 1 ≤ i ≤ n, where Y_i ∈ {−1, + 1} is the class label, and ${\mathbf{X}}_i=(x_{jk}^{\left(i\right)}:j=1,\cdots ,p;k=1,\cdots ,q)\in {\mathbb{R}}^{p\times q}$ is the corresponding image covariate. We further assume that (X_i, Y_i) with 1 ≤ i ≤ n are independent and identically distributed. In order to predict Y_i with X_i, the following logistic regression model is assumed

$$\begin{array}{c}\hfill \text{log}\frac{P_i}{1-P_i}=<{\mathbf{X}}_i,\mathbf{B}>,\end{array}$$ (1)

where P_i = P(Y_i = +1|X_i), $\mathbf{B}=\left(b_{jk}\right)\in {\mathbb{R}}^{p\times q}$ is the corresponding coefficient image, and $<{\mathbf{X}}_i,\mathbf{B}>={\sum }_{j=1}^p{\sum }_{k=1}^q{x}_{jk}^{\left(i\right)}{b}_{jk}$ is the inner operator of two matrices. Let β = vec(B) = (β₁,⋯,β_pq)^T and X_i = vec(X_i) = (x_ij, j = 1,⋯,pq)^T for i = 1, ⋯, n, then < X_i, B >= β^T X_i, and the model (1) is equivalent to $P_i=1/(1+e^{-{\beta }^T{X}_i})$. The true value of β is denoted by ${\beta }^{*}={({\beta }_1^{*},\cdots ,{\beta }_{pq}^{*})}^T$.

Traditionally, maximum likelihood method is usually used to estimate coefficient image B. The likelihood function is

$$\begin{array}{c}\hfill L\left(\beta \right)=\prod _{i=1}^n{P}_i^{I(Y_i=+1)}{(1-P_i)}^{I(Y_i=-1)}=\prod _{i=1}^n{(1+e^{-Y_i{\beta }^T{X}_i})}^{-1},\end{array}$$

and the corresponding log-likelihood function is $\text{ln}\left(L\left(\beta \right)\right)=-{\sum }_{i=1}^n\text{log}(1+e^{-Y_i{\beta }^T{X}_i}).$ Denote logistic loss function as $l\left(\beta \right)=\text{log}(1+e^{-Y_i{\beta }^T{X}_i})$, and the associated risk is denoted by $\mathbb{P}l\left(\beta \right)=El\left(\beta \right)$. We assume that ${\beta }^{*}=\text{arg}{\text{min}}_{\beta }\mathbb{P}l\left(\beta \right)$. The empirical risk is denoted by ${\mathbb{P}}_{n}l\left(\beta \right)=n^{-1}{\sum }_{i=1}^n\text{log}(1+e^{-Y_i{\beta }^T{X}_i})$. Hence, maximizing the likelihood function is equivalent to minimizing the empirical risk

$$\begin{array}{c}\hfill \underset{\beta }{\text{min}}{\mathbb{P}}_{n}l\left(\beta \right).\end{array}$$

Many optimization methods, such as Newton-Raphson method [6], can be used to solve the above problem.

However, in the image covariate case, the corresponding coefficient image B is usually assumed to be a piecewise smooth image with unknown edges. This assumption is widely used in the imaging literature, and is critical for addressing various scientific questions [22]. The maximum likelihood method does not take advantage of these characteristics. Moreover, image covariate is usually high dimensional, and not every element of X_i is useful to predict Y_i. But the maximum likelihood method can not perform variable selection. Consequently, we propose a novel estimation method for B in the next subsection, which can keep characteristics of image covariate such as local smoothing, and perform variable selection simultaneously.

Estimation

For the coefficient image B, we define its discrete gradient $\nabla \mathbf{B}\in {\mathbb{R}}^{p\times q\times 2}$ as

$$\begin{array}{c}\hfill {(\nabla \mathbf{B})}_{jk}=\{\begin{array}{cc}(b_{j+1,k}-b_{j,k},b_{j,k+1}-b_{j,k}),\hfill & j<p,k<q,\hfill \\ (0,b_{j,k+1}-b_{j,k}),\hfill & j=p,k<q,\hfill \\ (b_{j+1,k}-b_{j,k},0),\hfill & j<p,k=q,\hfill \\ (0,0),\hfill & j=p,k=q.\hfill \end{array}\end{array}$$

(∇B)_jk = (b_j+1,k − b_j,k, b_j,k+1 − b_j,k) is the discrete gradient at the position (j, k). Furthermore, b_j+1,k − b_j,k is the discrete gradient in the vertical direction, and b_j,k+1 − b_j,k indicates the discrete gradient in the horizontal direction. The Sobolev norm of B is the L₂ norm of ∇B, which is written as

$$\begin{array}{c}\hfill {\Vert \mathbf{B}\Vert }_{Sob}=(\sum _{j=1}^p\sum _{k=1}^q{(\nabla \mathbf{B})}_{jk}^2{)}^{1/2}.\end{array}$$

In fact, we can rewrite ${\Vert B\Vert }_{Sob}^2$ as a quadratic form of β. Specifically, we define a matrix $D=\left(d_{ij}\right)\in {\mathbb{R}}^{(2pq-p-q)\times pq}$ with d_ij defined in the following formula (2)

$$\begin{array}{c}\hfill d_{ij}=\{\begin{array}{cc}-1,\hfill & i=(2p-1)(k-1)+s,j=p(k-1)+(s+1)/2,\hfill \\ & k=1,\dots ,(q-1),s=1,3,\dots ,(2p-3);\hfill \\ 1,\hfill & i=(2p-1)(k-1)+s,j=p(k-1)+(s+1)/2+1,\hfill \\ & k=1,\dots ,(q-1),s=1,3,\dots ,(2p-3);\hfill \\ -1,\hfill & i=(2p-1)(k-1)+s,j=p(k-1)+s/2,\hfill \\ & k=1,\dots ,(q-1),s=2,4,\dots ,(2p-2);\hfill \\ 1,\hfill & i=(2p-1)(k-1)+s,j=p(k-1)+s/2+p,\hfill \\ & k=1,\dots ,(q-1),s=2,4,\dots ,(2p-2);\hfill \\ -1,\hfill & i=(2p-1)k,j=kp,k=1,\dots ,(q-1);\hfill \\ 1,\hfill & i=(2p-1)k,j=kp+p,k=1,\dots ,(q-1);\hfill \\ -1,\hfill & i>(2p-1)(q-1),j=i-(p-1)(q-1);\hfill \\ 1,\hfill & i>(2p-1)(q-1),j=i-(p-1)(q-1)+1,\hfill \\ 0,\hfill & \text{else}.\hfill \end{array}\end{array}$$ (2)

Then one can easily verify that ${\Vert B\Vert }_{Sob}^2={\beta }^T{D}^{T}D\beta$. We also present the matrix D with a graph in the case p = q = 3 for the purpose of understanding. Please see Fig (1).

Fig 1. The matrix D in the case p = q = 3.

We then consider the following optimization problem

$$\begin{array}{c}\hfill \underset{\beta }{\text{min}}Q\left(\beta \right),\end{array}$$ (3)

where $Q\left(\beta \right)={\mathbb{P}}_{n}l\left(\beta \right)+{\lambda }_1{\beta }^T{D}^{T}D\beta +{\lambda }_2{\Vert \beta \Vert }_1$, and ‖⋅‖₁ is the L₁ norm. The term λ₁ β^T D^T Dβ shrinks adjacent elements of B to be similar, hence it can capture the local smoothing of B. The term λ₂‖β‖₁ shrinks the elements of B to 0, and performs variable selection. We next propose an algorithm to solve the optimization problem (3).

Algorithm

For the optimization problem (3), we define K = D^T D = (k_jl) and $H\left(\beta \right)={\mathbb{P}}_{n}l\left(\beta \right)+{\lambda }_1{\beta }^{T}K\beta$, then one can see that Q(β) = H(β) + λ₂‖β‖₁. This indicates that the function Q(β) is a convex function with the separable structure [28]. [29] shows that the coordinate descent algorithm can be guaranteed to converge to the global minimizer for any convex optimization function with the separable structure. Hence we here propose a coordinate descent algorithm to obtain the solution of the optimization problem (3).

For j = 1, ⋯, pq, we successively minimize Q(β) along β_j direction with other parameters fixed. Specifically, denote the current value of β as β^c, and $p_i^c=P\left(Y_i\right|{\mathbf{X}}_i,{\beta }^c)=1/(1+e^{-Y_i{X}_i^T{\beta }^c})$ for i = 1, ⋯, n. For j = 1, ⋯, pq, we use the second order Taylor expansion to approximate function $H({\beta }_{-j}^c,{\beta }_j)$ with ${\beta }_{-j}^c={({\beta }_1^c,\cdots ,{\beta }_{j-1}^c,{\beta }_{j+1}^c,\cdots ,{\beta }_{pq}^c)}^T$ fixed. Specifically,

$$\begin{array}{c}\hfill \frac{\partial H({\beta }_{-j}^c,{\beta }_j)}{\partial {\beta }_j}{|}_{{\beta }_j={\beta }_j^c}=n^{-1}\sum _{i=1}^n(p_i^c-1)Y_i{x}_{ij}+2{\lambda }_1\sum _{l=1}^{pq}{k}_{jl}{\beta }_l^c,\end{array}$$

$$\begin{array}{c}\hfill \frac{{\partial }^{2}H({\beta }_{-j}^c,{\beta }_j)}{\partial {\beta }_j^2}{|}_{{\beta }_j={\beta }_j^c}=n^{-1}\sum _{i=1}^n{p}_i^c(1-p_i^c)x_{ij}^2+2{\lambda }_1{k}_{jj}.\end{array}$$

Hence,

$$\begin{array}{ccc}\hfill H({\beta }_{-j}^c,{\beta }_j)& \approx & H({\beta }_{-j}^c,{\beta }_j^c)+(n^{-1}\sum _{i=1}^n(p_i^c-1)Y_i{x}_{ij}+2{\lambda }_1\sum _{l=1}^{pq}{k}_{jl}{\beta }_l^c)({\beta }_j-{\beta }_j^c)\hfill \\ & +& \frac{1}{2}(n^{-1}\sum _{i=1}^n{p}_i^c(1-p_i^c)x_{ij}^2+2{\lambda }_1{k}_{jj}){({\beta }_j-{\beta }_j^c)}^2.\hfill \end{array}$$

Moreover,

$$\begin{array}{ccc}\hfill Q({\beta }_{-j}^c,{\beta }_j)& \approx & (n^{-1}\sum _{i=1}^n(p_i^c-1)Y_i{x}_{ij}+2{\lambda }_1\sum _{l=1}^{pq}{k}_{jl}{\beta }_l^c)({\beta }_j-{\beta }_j^c)\hfill \\ & +& \frac{1}{2}(n^{-1}\sum _{i=1}^n{p}_i^c(1-p_i^c)x_{ij}^2+2{\lambda }_1{k}_{jj}){({\beta }_j-{\beta }_j^c)}^2+{\lambda }_2\left|{\beta }_j\right|+C,\hfill \end{array}$$

where C is a constant containing no information about β_j. Denote $(n^{-1}{\sum }_{i=1}^n(p_i^c-1)Y_i{x}_{ij}+2{\lambda }_1{\sum }_{l=1}^{pq}{k}_{jl}{\beta }_l^c)({\beta }_j-{\beta }_j^c)+\frac{1}{2}(n^{-1}{\sum }_{i=1}^n{p}_i^c(1-p_i^c)x_{ij}^2+2{\lambda }_1{k}_{jj}){({\beta }_j-{\beta }_j^c)}^2+{\lambda }_2\left|{\beta }_j\right|$ by $\tilde{Q}\left({\beta }_j\right)$. One can update β_j through minimizing $\tilde{Q}\left({\beta }_j\right)$. Specifically, by

$$\begin{array}{c}\begin{array}{c}\frac{\partial \tilde{Q}\left({\beta }_j\right)}{\partial {\beta }_j}=\left(n^{-1}{\displaystyle \sum _{i=1}^n}\left(p_i^c-1\right)Y_i{x}_{ij}+2{\lambda }_1{\displaystyle \sum _{l=1}^{pq}}{k}_{jl}{\beta }_l^c\right)\\ \hfill +\left(n^{-1}{\displaystyle \sum _{i=1}^n}{p}_i^c\left(1-p_i^c\right)x_{ij}^2+2{\lambda }_1{k}_{jj}\right)\left({\beta }_j-{\beta }_j^c\right)+{\lambda }_2\frac{\partial \left|{\beta }_j\right|}{\partial {\beta }_j}=0,\hfill \end{array}\end{array}$$

where $\frac{\partial |{\beta }_j|}{\partial {\beta }_j}$ is the subderivative, that is $\frac{\partial |{\beta }_j|}{\partial {\beta }_j}=sign\left({\beta }_j\right)$ if β_j ≠ 0 and $\frac{\partial |{\beta }_j|}{\partial {\beta }_j}\in [-1,1]$ otherwise, we have that

$$\begin{array}{c}\begin{array}{c}{\beta }_j-{\beta }_j^c+{\left(n^{-1}{\displaystyle \sum _{i=1}^n}{p}_i^c\left(1-p_i^c\right)x_{ij}^2+2{\lambda }_1{k}_{jj}\right)}^{-1}\\ \hfill ·\left(n^{-1}{\displaystyle \sum _{i=1}^n}\left(p_i^c-1\right)Y_i{x}_{ij}+2{\lambda }_1{\displaystyle \sum _{l=1}^{pq}}{k}_{jl}{\beta }_l^c\right)={\lambda }_2{\left(n^{-1}{\displaystyle \sum _{i=1}^n}{p}_i^c\left(1-p_i^c\right)x_{ij}^2+2{\lambda }_1{k}_{jj}\right)}^{-1}\frac{\partial \left|{\beta }_j\right|}{\partial {\beta }_j}.\hfill \end{array}\end{array}$$

Consequently, one can update β_j as

$$\begin{array}{c}\hfill {\beta }_j^c\leftarrow \text{sign}\left({\Delta }_j^c\right)(|{\Delta }_j^c|-{\lambda }_2{(n^{-1}\sum _{i=1}^n{p}_i^c(1-p_i^c)x_{ij}^2+2{\lambda }_1{k}_{jj})}^{-1}{)}_{+},\end{array}$$

where ${\Delta }_j^c={\beta }_j^c-{(n^{-1}{\sum }_{i=1}^n{p}_i^c(1-p_i^c)x_{ij}^2+2{\lambda }_1{k}_{jj})}^{-1}(n^{-1}{\sum }_{i=1}^n(p_i^c-1)Y_i{x}_{ij}+2{\lambda }_1{\sum }_{l=1}^{pq}{k}_{jl}{\beta }_l^c)$.

We summarize the algorithm as follows.

Coordinate

• Step 1. Initialization. Given initial value β.

• Step 2. For t = 1, 2, ⋯, update β.

  For j = 1, ⋯, p

  • Compute p_i for 1 ≤ i ≤ n;
  • Let ${\Delta }_j={\beta }_j-{(n^{-1}{\sum }_{i=1}^n{p}_i(1-p_i)x_{ij}^2+2{\lambda }_1{k}_{jj})}^{-1}(n^{-1}{\sum }_{i=1}^n(p_i-1)Y_i{x}_{ij}+2{\lambda }_1{\sum }_{l=1}^{pq}{k}_{jl}{\beta }_l)$;
  • Update ${\beta }_j\leftarrow \text{sign}\left({\Delta }_j\right)(|{\Delta }_j|-{\lambda }_2{(n^{-1}{\sum }_{i=1}^n{p}_i(1-p_i)x_{ij}^2+2{\lambda }_1{k}_{jj})}^{-1}{)}_{+}$;

  End for.

• Step 3. Repeat Step 2 until convergence.

By the proposed algorithm, we can obtain the solution of (3), which is denoted by $\widehat{\beta }$. As the estimator for β, the theoretical properties of $\widehat{\beta }$ are studied in the next subsection.

Theoretical properties

In this subsection, we consider the properties of $\widehat{\beta }$. A nonasymptotic error bound of $\widehat{\beta }$ is given. We assume that the true value β* is sparse. Let $I^{*}=\{1\le j\le pq:{\beta }_j^{*}\ne 0\}$, and $k^{*}={\sum }_{j=1}^{pq}I({\beta }_j^{*}\ne 0)$ be the cardinality of I*. For the purpose of theoretical studies, we make the following assumptions.

Assumption 1. Assume that there exists a constant L such that |x_ij| ≤ L for every 1 ≤ i ≤ n, 1 ≤ j ≤ pq.

Assumption 2. Assume that there exists a constant C such that ‖β*‖₁ ≤ C.

Assumption 3. For the matrix K, assume that there exists a constant C₀ such that λ_max(K) ≤ C₀, where λ_max(K) is the largest eigenvalue of K.

Assumption 4. Let $\Sigma =E\left(X_i{X}_i^T\right)$. Define the set $V_{\alpha ,ϵ}=\{\beta \in {\mathbb{R}}^{pq}:{\sum }_{j\notin I^{*}}|{\beta }_j|\le \alpha {\sum }_{j\in I^{*}}\left|{\beta }_j\right|+ϵ\}\text{for}\text{some}\alpha ,ϵ.$ Assume that there exists a constant 0 < b ≤ 1 such that for every β ∈ V_{α, ϵ},

$$\begin{array}{c}\hfill P({\beta }^T\Sigma \beta \ge b\sum _{j\in I^{*}}{\beta }_j^2-ϵ)=1.\end{array}$$

Assumption 1 makes a common bound L for all x_ij with i = 1, ⋯, n, j = 1, ⋯, pq. Assumption 2 gives a bound for ‖β*‖₁. Combining Assumptions 1 and 2, one can make sure that P_i with 1 ≤ i ≤ n are bounded away from zero and one. P_i equalling zero or one will cause the i-th subject to be either ignorable or dominant in the likelihood function, that is not expected to appear in statistical analysis. This case can be avoided by Assumptions 1 and 2. In Assumption 3, we assume that the largest eigenvalue of K is bounded. Assumption 4 is called Condition Stabil, which can be regarded as a stability requirement on the correlation structure [30]. Under these assumptions, we have the following theorem.

Theorem 1 Assume that Assumptions 1-3 are true and Assumption 4 holds for $\alpha =5,ϵ=\frac{\text{ln}2}{2^d}\times \frac{3}{{\lambda }_2}$ with d = max{pq, n}, let λ₁ = λ₂/(6CC₀), if

$$\begin{array}{c}\hfill {\lambda }_2\ge 3(7L\sqrt{\frac{2\text{ln}\left(2d\right)}{n}}+\frac{L}{2d}+2L\sqrt{\frac{-2\text{ln}\delta }{n}}),\end{array}$$ (4)

then we have that

$$\begin{array}{c}\hfill P(\Vert \widehat{\beta }-{\beta }^{*}{\Vert }_1\le \frac{3k^{*}{\lambda }_2}{sb}+(1+\frac{3s}{{\lambda }_2})ϵ)>1-\delta ,\end{array}$$

where s = (1 + e^A)⁻⁴ with A = 8CL is a constant.

The proof of Theorem 1 is put in the appendix section. The theorem shows that with a high probability, the L₁ norm of estimate error is bounded by 3k*λ₂/(sb) + (1 + 3s/λ₂)ϵ. One can see that the term (1 + 3s/λ₂)ϵ = O(d/2^d), which can be negligible for large d. Hence, the term 3k*λ₂/(sb) dominates the upper bound, which becomes larger when b becomes smaller. If further assume that ln(pq) = o(n), by the condition (4) one can see that λ₂ can tend to 0. Further 3k*λ₂/(sb) → 0, that means the upper bound can tend to zero. Consequently, the consistency of $\widehat{\beta }$ can be guaranteed.

The selection of tuning parameters

The optimization function (3) contains two tuning parameters λ₁ and λ₂, which should be determined by some criteria, such as BIC, cross validation method. In our simulation studies, we select the tuning parameters by a validation set. And in real data analysis, the cross validation method is used. Before applying these methods, one should firstly determine the value range of tuning parameters. Specifically, we here make a transformation of λ₁ and λ₂. Let λ = λ₁ + λ₂, and α = λ₂/λ. Then the penalty terms in (3) can be rewritten as λ(α‖β‖₁ + (1 − α)β^T Kβ). Because α ∈ [0, 1], the alternative values of α are set as 0.02κ for κ = 1, ⋯, 50. With a given α, we denote λ₀ as the threshold value. Once λ ≥ λ₀, the solution of (3) is exactly zero. By

$$\begin{array}{c}\hfill \frac{\partial Q\left(\beta \right)}{\partial \beta }=\frac{1}{n}\sum _{i=1}^n\frac{e^{-Y_i{\beta }^T{X}_i}}{1+e^{-Y_i{\beta }^T{X}_i}}(-Y_i{X}_i)+2\lambda (1-\alpha )K\beta +\lambda \alpha \frac{{\partial \Vert \beta \Vert }_1}{\beta }=0,\end{array}$$

one can see that once β = 0 is the solution, every element of $1/(2\lambda \alpha n){\sum }_{i=1}^n(-Y_i{X}_i)$ belongs to [−1, 1]. This means that ${\lambda }_0=1/\left(2\alpha n\right){\Vert {\sum }_{i=1}^n\left(Y_i{X}_i\right)\Vert }_{\infty }$. Following the idea of [14], the alternative values of λ are set as 0.001 and 0.96^νλ₀ for ν = 0, 1, ⋯, 160. For the validation set method, the prediction error on the validation set of our approach with tuning parameters α, λ is denoted by PE_{(α, λ)}. The final α, λ are selected as the minimizer of PE_{(α, λ)}.

For the M-fold cross validation method, the data are randomly divided into M folds of approximately equal size. For m = 1, ⋯, M, we treat the m fold as the validation set, and fit the model with tuning parameters α, λ on the remaining M − 1 flods. The corresponding prediction error on the validation set is denoted by $P{E}_{(\alpha ,\lambda )}^{\left(m\right)}$ and the cross validation prediction error is defined as

$$\begin{array}{c}\hfill P{E}_{(\alpha ,\lambda )}^{\left(cv\right)}=\frac{1}{M}\sum _{m=1}^{M}P{E}_{(\alpha ,\lambda )}^{\left(m\right)}.\end{array}$$

The α, λ are selected as the minimizer of the cross validation prediction error [6].

Numerical studies

In this section, we evaluate the performance of our proposed method by two simulated examples and a real data analysis. For the purpose of comparison, we also consider the logistic regression model with L₁ penalty [12, 13], the logistic regression model with fused lasso penalty, and linear support vector machine, which are denoted by LG-L₁, LG-fused, and Linear SVM respectively for convenience. Meanwhile, our proposed method is abbreviated as LG-sob.

Simulation studies

Example 1. We generate data from the following model

$$\begin{array}{c}\hfill \text{log}\frac{P(Y_i=+1|{\mathbf{X}}_i)}{P(Y_i=-1|{\mathbf{X}}_i)}=<{\mathbf{X}}_i,B_0>,i=1,\cdots ,n,\end{array}$$

where X_i and B₀ both belong to ${\mathbb{R}}^{32\times 32}$. One result caused by image covariates is that the corresponding regression coefficient can be treated as a image too. Hence we here just treat B₀ as images, while X_i is generated from a multivariate normal distribution. Specifically, we define the vectorization of X_i as X_i, and X_i is generated from a multivariate normal distribution with mean 0 and covariance $\text{cov}(x_{i{j}_1},x_{i{j}_2})=0.5^{|j_1-j_2|}$ for any 1 ≤ j₁, j₂ ≤ 1024. The parameter image B₀ is considered in two cases, which have been shown in Fig 2. The first case of B₀ denoted by B₀₁ is a bird picture, in which the blue region takes value 0, and the yellow region takes value 1. The other case of B₀ denoted by B₀₂ is a butterfly picture, which is more complicated and takes values in interval [−0.0197, 0.0628]. Given X_i and B₀, the response Y_i is generated from a two-point distribution $P(Y_i=+1|{\mathbf{X}}_i)=1-P(Y_i=-1|{\mathbf{X}}_i)=1/(1+e^{-<{\mathbf{X}}_i,B_0>})$.

Fig 2. Simulated example.

The true parameter images B₀.

Example 2. In this example, the mechanism of data generation is similar to that for Example 1, the only difference is that we generate X_i in a more complex way. In particular, we follow the simulation scheme of [22] and generate X_i from a 32 × 32 phantom map with 1024 pixels according to a spatially correlated random process X_i = ∑_l l⁻¹ η_il φ_l, among which the η_il are standard normal random variables and the φ_l are bivariate Haar wavelet basis functions.

For these two simulated examples, along with the training set with sample size n, we also generate a validation set and a test set with sample sizes both being 500. We train the model on the training set, select tuning parameters through the validation set, and calculate the classification accuracy on the test set to evaluate the performance of the model.

For every specification of the parameter B₀ and sample size n, we replicate the simulation 100 times for each example, and the average prediction errors are computed and summarized in Table 1 for Example 1, and Table 2 for Example 2 respectively. Besides the prediction errors, we also calculate the average estimation errors ${\sum }_{i=1}^{100}{\Vert {\widehat{B}}_i-B_0\Vert }^2/100$ for LG-sob, LG-L₁ and LG-fused, where ${\widehat{B}}_i$ is the parameter image estimator in the i-th time. From the results, one can see that our proposed method performs better than the other three methods in all cases from the prediction perspective. As sample size n becomes larger, the prediction errors will become smaller, but the estimation errors do not decrease congruously. The reason may be that the tuning parameters are selected based on minimization of prediction error.

Table 1. Results of simulated example 1: Prediction error (PE) and estimation error (EE).

(n, B0)	(500, B01)		(1000, B01)		(500, B02)		(1000, B02)
	PE	EE	PE	EE	PE	EE	PE	EE
LG-sob	0.099	337.645	0.075	336.173	0.107	8.153	0.080	13.505
LG-L1	0.272	404.589	0.199	375.926	0.272	17.354	0.204	23.143
LG-fused	0.248	423.242	0.190	406.866	0.248	7.016	0.190	9.400
Linear SVM	0.221	NA	0.174	NA	0.223	NA	0.172	NA

Table 2. Results of simulated example 2: Prediction error (PE) and estimation error (EE).

(n, B0)	(500, B01)		(1000, B01)		(500, B02)		(1000, B02)
	PE	EE	PE	EE	PE	EE	PE	EE
LG-sob	0.028	181.06	0.023	176.19	0.049	582.94	0.038	1096.1
LG-L1	0.073	1712.6	0.058	1501.7	0.116	2190.9	0.090	3022.4
LG-fused	0.052	438.57	0.044	495.73	0.097	482.19	0.076	775.99
Linear SVM	0.050	NA	0.038	NA	0.084	NA	0.071	NA

Moreover, we also randomly select one simulated result from the 100 replications of Example 1, and show the parameter image estimations in Fig 3. One can see that our proposed LG-sob method can capture the shapes of images, but LG-L₁ and LG-fused do not have this property.

Fig 3. Simulated example.

One of randomly selected parameter images estimations. The first row is the results of our proposed LG-sob, the second row is the results of LG-L₁, the third row is the results of LG-fused.

A real data analysis

The classification of the ZIP Code Dataset is a benchmark problem in machine learning community [6]. One can obtain the ZIP Code Dataset from the following website https://web.stanford.edu/~hastie/StatLearnSparsity_files/DATA/zipcode.html [28]. The Dataset contains normalized handwritten digits, which are automatically scanned from envelopes by the U.S. Postal Service. Every observation is a handwritten digit, and is represented as a size normalized 16 × 16 grayscale image [31]. The purpose is to use the 256 pixel values to predict the corresponding digit. The Dataset contains a training set with 7291 observations and a test set with 2007 observations. Because this article only considers the binary response prediction by logistic regression models, and it looks like that numbers 3 and 8 have more similar characteristics, hence we only consider handwritten 3’s and 8’s in this paper. The sizes of handwritten 3’s and 8’s are 658 and 542 respectively in the train set, while they are both 166 in the test set. Fig 4 shows some examples of handwritten 3’s and 8’s.

Fig 4. Real data analysis.

Some examples of handwritten 3’s and 8’s.

More specifically, we denote the i-th observation by ${\mathbf{X}}_i\in {\mathbb{R}}^{16\times 16}$, and define the corresponding class label Y_i = −1 if X_i represents handwritten 3 and Y_i = + 1 if X_i represents handwritten 8. Our proposed method is applied to construct the classifier for the prediction of Y_i (i.e. handwritten numeral) based on the grayscale image X_i. We train the model on the training set, and evaluate the performance of the proposed method on the test set by classification accuracy. For the purpose of comparison, we also consider the logistic regression model with only L₁ penalty.

The tuning parameters are selected by 10-fold cross validation (CV) method. The CV prediction errors in various parameters setting are calculated and plotted in Fig 5. Finally, our proposed method selects the tuning parameters as α = 0.04, λ = 0.0118, while the method with L₁ penalty selects the tuning parameter as λ = 0.0014. The parameter image estimations of the two methods are shown in Fig 6. One can see that our proposed method tends to make adjacent pixels have similar effects on the model. Meanwhile, LG-L₁ tries to obtain a more sparse parameter estimation, and LG-fused method tries to make pixels only adjacent in the vertical direction have similar effects. The top-left region of parameter image has positive effects on handwritten numeral 8, and the bottom-right region has positive effects on handwritten numeral 3. The classification accuracy on the test set of our proposed method is 96.99%, while the accuracies of LG-L₁, LG-fused, and Linear SVM are 96.39%, 96.08% and 96.39%, respectively. The proposed method performs better.

Fig 5. Real data analysis.

The results of 10-fold CV: Prediction error in various parameters settings.

Fig 6. Real data analysis.

The parameter image estimations. Left: the estimation of our proposed LG-sob; Middle: the estimation of LG-L₁; Right: the estimation of LG-fused.

Conclusion

We have developed a novel estimation method for logistic regression with image covariates. This method can not only perform variable selection, but also capture the shape features of the parameter images. Both theoretical results and numerical studies show that our method performs well. We have proposed a coordinated descent algorithm to solve the optimization problem, and the global convergence of the algorithm is guaranteed. However, as pointed out by one referee, the coordinated descent algorithm is very time consuming, especially in the case of high dimensional image covariates. Many more efficient optimalization approaches, such as Nesterov accelerated gradient methods [32], interior-point methods [13], may be more suitable. We will research this issue in future. Furthermore, our method is mainly based on Sobolev norm regularization, compared to which total variation regularization is more sensitive to capture sharp edges and jumps of parameter images. However, the algorithm of total variation regularization based estimation method is more complex, which can be our future work to study.

Appendix: Proof of Theorem 1

Before giving the proof of Theorem 1, we first list the bounded differences inequality as the following lemma without proof.

Lemma 1 (the Bounded Differences Inequality) Suppose that $X_1,\cdots ,X_n\in \mathcal{H}$ are independent, and the function $f:{\mathcal{H}}^n\to \mathbb{R}$ satisfies the bounded difference assumption

$$\begin{array}{c}\hfill \underset{x_1,\cdots ,x_n,x_i^{{}^{\prime }}\in \mathcal{H}}{sup}|f(x_1,\cdots ,x_n)-f(x_1,\cdots ,x_{i-1},x_i^{{}^{\prime }},x_{i+1},\cdots ,x_n)|\le c_i,\end{array}$$

for i = 1, ⋯, n. Then for all t > 0,

$$\begin{array}{c}\hfill P(f-E\left(f\right)\ge t)\le e^{-2t^2/{\sum }_{i=1}^n{c}_i^2}.\end{array}$$

For more details about Lemma 1 and its proof, one can refer to [33]. The following is the proof of Theorem 1.

Proof of Theorem 1. By the definitions of $\widehat{\beta }$ and β*, one can see that

$$\begin{array}{c}\hfill \mathbb{P}l\left(\widehat{\beta }\right)\ge \mathbb{P}l\left({\beta }^{*}\right),\end{array}$$

and

$$\begin{array}{c}\hfill {\mathbb{P}}_{n}l\left(\widehat{\beta }\right)+{\lambda }_1{\widehat{\beta }}^{T}K\widehat{\beta }+{\lambda }_2\Vert \widehat{\beta }{\Vert }_1\le {\mathbb{P}}_{n}l\left({\beta }^{*}\right)+{\lambda }_1{\beta }^{*T}K{\beta }^{*}+{\lambda }_2{\Vert {\beta }^{*}\Vert }_1.\end{array}$$

Hence, we have that

$$\begin{array}{ccc}\hfill 0& \le & \mathbb{P}l\left(\widehat{\beta }\right)-\mathbb{P}l\left({\beta }^{*}\right)\hfill \\ & \le & ({\mathbb{P}}_n-\mathbb{P})(l\left({\beta }^{*}\right)-l\left(\widehat{\beta }\right))+{\lambda }_1({\beta }^{*T}K{\beta }^{*}-{\widehat{\beta }}^{T}K\widehat{\beta })+{\lambda }_2(\Vert {\beta }^{*}{\Vert }_1-\Vert \widehat{\beta }{\Vert }_1).\hfill \end{array}$$ (5)

Moreover,

$$\begin{array}{ccc}& & {\lambda }_2/3{\Vert \widehat{\beta }-{\beta }^{*}\Vert }_1\hfill \\ & \le & {\lambda }_2/3{\Vert \widehat{\beta }-{\beta }^{*}\Vert }_1+{\lambda }_1{(\widehat{\beta }-{\beta }^{*})}^{T}K(\widehat{\beta }-{\beta }^{*})\hfill \\ & \le & {\lambda }_2/3{\Vert \widehat{\beta }-{\beta }^{*}\Vert }_1+{\lambda }_1{(\widehat{\beta }-{\beta }^{*})}^{T}K(\widehat{\beta }-{\beta }^{*})+\mathbb{P}l\left(\widehat{\beta }\right)-\mathbb{P}l\left({\beta }^{*}\right)\hfill \\ & \le & {\lambda }_2/3{\Vert \widehat{\beta }-{\beta }^{*}\Vert }_1+{\lambda }_1{(\widehat{\beta }-{\beta }^{*})}^{T}K(\widehat{\beta }-{\beta }^{*})+{\lambda }_1({\beta }^{*T}K{\beta }^{*}-{\widehat{\beta }}^{T}K\widehat{\beta })\hfill \\ & & +({\mathbb{P}}_n-\mathbb{P})(l\left({\beta }^{*}\right)-l\left(\widehat{\beta }\right))+{\lambda }_2(\Vert {\beta }^{*}{\Vert }_1-\Vert \widehat{\beta }{\Vert }_1).\hfill \end{array}$$ (6)

We first consider the term $({\mathbb{P}}_n-\mathbb{P})(l\left({\beta }^{*}\right)-l\left(\widehat{\beta }\right))$. Specifically, define $L_n={sup}_{\beta }\frac{({\mathbb{P}}_n-\mathbb{P})(l\left({\beta }^{*}\right)-l\left(\beta \right))}{\Vert \beta -{\beta }^{*}{\Vert }_1+ϵ}$. Let $l(\beta ;Y_i,X_i)=\text{log}(1+e^{-Y_i{\beta }^T{X}_i})$, and

$$\begin{array}{c}\hfill {\mathbb{P}}_{nl}l\left(\beta \right)=\frac{1}{n}(\sum _{i=1,i\ne l}^{n}l(\beta ;Y_i,X_i)+l(\beta ;Y_l^{{}^{\prime }},X_l^{{}^{\prime }})),\end{array}$$

which is the empirical measure corresponding to replacing (Y_l, X_l) by $(Y_l^{{}^{\prime }},X_l^{{}^{\prime }})$. Then

$$\begin{array}{ccc}& & \frac{({\mathbb{P}}_n-\mathbb{P})(l\left({\beta }^{*}\right)-l\left(\beta \right))}{\Vert \beta -{\beta }^{*}{\Vert }_1+ϵ}-\frac{({\mathbb{P}}_{nl}-\mathbb{P})(l\left({\beta }^{*}\right)-l\left(\beta \right))}{\Vert \beta -{\beta }^{*}{\Vert }_1+ϵ}\hfill \\ & =& \frac{1}{n}\frac{l({\beta }^{*};Y_l,X_l)-l(\beta ;Y_l,X_l)-l({\beta }^{*};Y_l^{{}^{\prime }},X_l^{{}^{\prime }})+l(\beta ;Y_l^{{}^{\prime }},X_l^{{}^{\prime }})}{\Vert \beta -{\beta }^{*}{\Vert }_1+ϵ}\hfill \\ & \le & \frac{4L}{n}\frac{\Vert \beta -{\beta }^{*}{\Vert }_1}{\Vert \beta -{\beta }^{*}{\Vert }_1+ϵ}\le \frac{4L}{n},\hfill \end{array}$$

among which the inequality is obtained by a first order Taylor expansion and the assumption 1. Then by Lemma 1, we can obtain that

$$\begin{array}{c}\hfill P(L_n-E\left(L_n\right)\ge u)\le \text{exp}\{-\frac{n{u}^2}{8L^2}\}.\end{array}$$

Let $\delta =\text{exp}\{-\frac{n{u}^2}{8L^2}\}$, then we have that $u=2L\sqrt{\frac{-2\text{ln}\delta }{n}}$, and P(L_n − E(L_n) ≥ u) ≤ δ.

Let d = max{pq, n}. Taking $ϵ=\frac{\text{ln}2}{2^d}\times \frac{3}{{\lambda }_2}$, by the lemma 3 of [34] with C_φ = 1, C_F = L, we have

$$\begin{array}{c}\hfill E\left(L_n\right)\le 7L\sqrt{\frac{2\text{ln}\left(2d\right)}{n}}+\frac{L}{2d}.\end{array}$$

Consequently, we have that

$$\begin{array}{c}\hfill P(L_n\le 7L\sqrt{\frac{2\text{ln}\left(2d\right)}{n}}+\frac{L}{2d}+2L\sqrt{\frac{-2\text{ln}\delta }{n}})\ge 1-\delta .\end{array}$$

By the condition of Theorem 1, we know ${\lambda }_2\ge 3(7L\sqrt{\frac{2\text{ln}\left(2d\right)}{n}}+\frac{L}{2d}+2L\sqrt{\frac{-2\text{ln}\delta }{n}})$, hence P(L_n ≤ λ₂/3) ≥ 1 − δ.

On the event {L_n ≤ λ₂/3}, we have that

$$\begin{array}{c}\hfill ({\mathbb{P}}_n-\mathbb{P})(l\left({\beta }^{*}\right)-l\left(\widehat{\beta }\right))\le \frac{{\lambda }_2}{3}(\Vert \beta -{\beta }^{*}{\Vert }_1+ϵ).\end{array}$$ (7)

Secondly, we consider the term ${\lambda }_1{(\widehat{\beta }-{\beta }^{*})}^{T}K(\widehat{\beta }-{\beta }^{*})+{\lambda }_1({\beta }^{*T}K{\beta }^{*}-{\widehat{\beta }}^{T}K\widehat{\beta })$. Based on Assumptions 2 and 3, one can see that

$$\begin{array}{ccc}& & {\lambda }_1{(\widehat{\beta }-{\beta }^{*})}^{T}K(\widehat{\beta }-{\beta }^{*})+{\lambda }_1({\beta }^{*T}K{\beta }^{*}-{\widehat{\beta }}^{T}K\widehat{\beta })\hfill \\ & =& 2{\lambda }_1{\beta }^{*T}K({\beta }^{*}-\widehat{\beta })\hfill \\ & \le & 2{\lambda }_1{\lambda }_{max}\left(K\right)\Vert {\beta }^{*}{\Vert }_2{\Vert \widehat{\beta }-{\beta }^{*}\Vert }_2\hfill \\ & \le & 2{\lambda }_1{\lambda }_{max}\left(K\right)\Vert {\beta }^{*}{\Vert }_1{\Vert \widehat{\beta }-{\beta }^{*}\Vert }_1\hfill \\ & \le & 2{\lambda }_{1}C{C}_0{\Vert \widehat{\beta }-{\beta }^{*}\Vert }_1.\hfill \end{array}$$

One can see that if λ₁ = λ₂/(6CC₀), we have

$$\begin{array}{c}\hfill {\lambda }_1{(\widehat{\beta }-{\beta }^{*})}^{T}K(\widehat{\beta }-{\beta }^{*})+{\lambda }_1({\beta }^{*T}K{\beta }^{*}-{\widehat{\beta }}^{T}K\widehat{\beta })\le {\lambda }_2/3{\Vert \widehat{\beta }-{\beta }^{*}\Vert }_1.\end{array}$$ (8)

Consequently, on the event {L_n ≤ λ₂/3} we combine (6), (7), (8), and obtain

$$\begin{array}{ccc}\hfill {\lambda }_2/3{\Vert \widehat{\beta }-{\beta }^{*}\Vert }_1& \le & {\lambda }_2\Vert \widehat{\beta }-{\beta }^{*}{\Vert }_1+{\lambda }_2(\Vert {\beta }^{*}{\Vert }_1-\Vert \widehat{\beta }{\Vert }_1)+{\lambda }_2/3ϵ\hfill \\ & \le & {\lambda }_2(\Vert \widehat{\beta }{\Vert }_1+\Vert {\beta }^{*}{\Vert }_1)+{\lambda }_2(\Vert {\beta }^{*}{\Vert }_1-\Vert \widehat{\beta }{\Vert }_1)+{\lambda }_2/3ϵ\hfill \\ & \le & 2{\lambda }_2{\Vert {\beta }^{*}\Vert }_1+{\lambda }_2/3ϵ.\hfill \end{array}$$ (9)

Hence we have that

$$\begin{array}{c}\hfill \Vert \widehat{\beta }-{\beta }^{*}{\Vert }_1\le 6{\Vert {\beta }^{*}\Vert }_1+ϵ\le 7C.\end{array}$$ (10)

By (9) one can also obtain that

$$\begin{array}{ccc}\hfill \Vert \widehat{\beta }-{\beta }^{*}{\Vert }_1& \le & 3\Vert \widehat{\beta }-{\beta }^{*}{\Vert }_1+3(\Vert {\beta }^{*}{\Vert }_1-\Vert \widehat{\beta }{\Vert }_1)+ϵ\hfill \\ & =& 3\left(\sum _{j\in I^{*}}\right|{\widehat{\beta }}_j-{\beta }_j^{*}|+\sum _{j\notin I^{*}}|{\widehat{\beta }}_j|+\sum _{j\in I^{*}}|{\beta }_j^{*}|-\sum _{j=1}^{pq}|{\widehat{\beta }}_j\left|\right)+ϵ\hfill \\ & =& 3\left(\sum _{j\in I^{*}}\right|{\widehat{\beta }}_j-{\beta }_j^{*}|+\sum _{j\in I^{*}}|{\beta }_j^{*}|-\sum _{j\in I^{*}}|{\widehat{\beta }}_j\left|\right)+ϵ\hfill \\ & \le & 6\sum _{j\in I^{*}}|{\widehat{\beta }}_j-{\beta }_j^{*}|+ϵ.\hfill \end{array}$$

Consequently, we have ${\sum }_{j\notin I^{*}}|{\widehat{\beta }}_j-{\beta }_j^{*}|\le 5{\sum }_{j\in I^{*}}|{\widehat{\beta }}_j-{\beta }_j^{*}|+ϵ$. This means that $\widehat{\beta }-{\beta }^{*}\in V_{5,ϵ}$.

By the example 4.5 in [35], we have that $\mathbb{P}l\left(\widehat{\beta }\right)-\mathbb{P}l\left({\beta }^{*}\right)\ge E_X{(P\left(\widehat{\beta }\right)-P\left({\beta }^{*}\right))}^2$, where P(β) = 1/(1+ e^−XTβ) and E_X(⋅) is the expectation with respect to the distribution of X. Using Taylor expansion, one can obtain that

$$\begin{array}{c}\hfill P\left(\widehat{\beta }\right)-P\left({\beta }^{*}\right)=\frac{e^{{\tilde{\beta }}^{T}X}}{{(1+e^{{\tilde{\beta }}^{T}X})}^2}{X}^T(\widehat{\beta }-{\beta }^{*}),\end{array}$$

where $\tilde{\beta }=\tau \widehat{\beta }+(1-\tau ){\beta }^{*}$ for some τ ∈ (0, 1). Moreover, by (10) and Assumptions 1-2, we have ${\tilde{\beta }}^{T}X\le \Vert \tilde{\beta }{\Vert }_{1}L\le (\tau \Vert \widehat{\beta }-{\beta }^{*}{\Vert }_1+\Vert {\beta }^{*}{\Vert }_1)L\le 8CL.$ This means that

$$\begin{array}{c}\hfill {(P\left(\widehat{\beta }\right)-P\left({\beta }^{*}\right))}^2\ge s{(\widehat{\beta }-{\beta }^{*})}^{T}X{X}^T(\widehat{\beta }-{\beta }^{*}),\end{array}$$

where s = (1 + e^A)⁻⁴ and A = 8CL, then by Assumption 4 we have that

$$\begin{array}{c}\hfill \mathbb{P}l\left(\widehat{\beta }\right)-\mathbb{P}l\left({\beta }^{*}\right)\ge E_X{(P\left(\widehat{\beta }\right)-P\left({\beta }^{*}\right))}^2\ge s{(\widehat{\beta }-{\beta }^{*})}^T\Sigma (\widehat{\beta }-{\beta }^{*})\ge sb\sum _{j\in I^{*}}{(\widehat{{\beta }_j}-{\beta }_j^{*})}^2-sϵ.\end{array}$$ (11)

Furthermore, we have

$$\begin{array}{ccc}& & \frac{{\lambda }_2}{3}{\Vert \widehat{\beta }-{\beta }^{*}\Vert }_1+sb\sum _{j\in I^{*}}{(\widehat{{\beta }_j}-{\beta }_j^{*})}^2-sϵ\hfill \\ & \le & \frac{{\lambda }_2}{3}{\Vert \widehat{\beta }-{\beta }^{*}\Vert }_1+\mathbb{P}l\left(\widehat{\beta }\right)-\mathbb{P}l\left({\beta }^{*}\right)\hfill \\ & \le & {\lambda }_2/3{\Vert \widehat{\beta }-{\beta }^{*}\Vert }_1+{\lambda }_1({\beta }^{*T}K{\beta }^{*}-{\widehat{\beta }}^{T}K\widehat{\beta })+({\mathbb{P}}_n-\mathbb{P})(l\left({\beta }^{*}\right)-l\left(\widehat{\beta }\right))\hfill \\ & & +{\lambda }_2(\Vert {\beta }^{*}{\Vert }_1-\Vert \widehat{\beta }{\Vert }_1)\hfill \\ & \le & {\lambda }_2/3{\Vert \widehat{\beta }-{\beta }^{*}\Vert }_1+{\lambda }_1{(\widehat{\beta }-{\beta }^{*})}^{T}K(\widehat{\beta }-{\beta }^{*})+{\lambda }_1({\beta }^{*T}K{\beta }^{*}-{\widehat{\beta }}^{T}K\widehat{\beta })\hfill \\ & & +({\mathbb{P}}_n-\mathbb{P})(l\left({\beta }^{*}\right)-l\left(\widehat{\beta }\right))+{\lambda }_2(\Vert {\beta }^{*}{\Vert }_1-\Vert \widehat{\beta }{\Vert }_1)\hfill \\ & \le & {\lambda }_2\Vert \widehat{\beta }-{\beta }^{*}{\Vert }_1+{\lambda }_2(\Vert {\beta }^{*}{\Vert }_1-\Vert \widehat{\beta }{\Vert }_1)+{\lambda }_2/3ϵ\hfill \\ & =& {\lambda }_2(\sum _{j\in I^{*}}|{\widehat{\beta }}_j-{\beta }_j^{*}|+\sum _{j\notin I^{*}}|\widehat{{\beta }_j}\left|\right)+{\lambda }_2(\sum _{j\in I^{*}}|{\beta }_j^{*}|-\sum _{j\in I^{*}}|\widehat{{\beta }_j}|-\sum _{j\notin I^{*}}|\widehat{{\beta }_j}\left|\right)\hfill \\ & & +{\lambda }_2/3ϵ\hfill \\ & =& {\lambda }_2\sum _{j\in I^{*}}|{\widehat{\beta }}_j-{\beta }_j^{*}|+{\lambda }_2(\sum _{j\in I^{*}}|{\beta }_j^{*}|-\sum _{j\in I^{*}}|\widehat{{\beta }_j}\left|\right)+{\lambda }_2/3ϵ\hfill \\ & \le & 2{\lambda }_2\sum _{j\in I^{*}}|{\widehat{\beta }}_j-{\beta }_j^{*}|+{\lambda }_2/3ϵ,\hfill \end{array}$$

where the first inequality follows by (11), the second inequality follows by (5), and the fourth inequality is obtained by combining the results of (7) and (8). Consequently, we have

$$\begin{array}{c}\hfill \Vert \widehat{\beta }-{\beta }^{*}{\Vert }_1+\frac{3}{{\lambda }_2}sb\sum _{j\in I^{*}}{(\widehat{{\beta }_j}-{\beta }_j^{*})}^2\le 6\sum _{j\in I^{*}}|\widehat{{\beta }_j}-{\beta }_j^{*}|+(1+\frac{3s}{{\lambda }_2})ϵ\\ \hfill \le 9a{k}^{*}+\frac{1}{a}\sum _{j\in I^{*}}{(\widehat{{\beta }_j}-{\beta }_j^{*})}^2+(1+\frac{3s}{{\lambda }_2})ϵ,\hfill \end{array}$$

where a is a positive constant and the second inequality follows by

$$\begin{array}{c}\hfill 6|\widehat{{\beta }_j}-{\beta }_j^{*}|=2·3a^{1/2}·a^{-1/2}|\widehat{{\beta }_j}-{\beta }_j^{*}|\le 6a+{(\widehat{{\beta }_j}-{\beta }_j^{*})}^2/a.\end{array}$$

Let a = λ₂/(3sb), then

$$\begin{array}{c}\hfill \Vert \widehat{\beta }-{\beta }^{*}{\Vert }_1\le \frac{3k^{*}{\lambda }_2}{sb}+(1+\frac{3s}{{\lambda }_2})ϵ.\end{array}$$

This completes the proof of the Theorem.

Data Availability

The real ZIP Code Dataset from the following website: https://web.stanford.edu/~hastie/StatLearnSparsity_files/DATA/zipcode.html.

Funding Statement

The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.