Structured sparse support vector machine with ordered features

ABSTRACT

In the application of high-dimensional data classification, several attempts have been made to achieve variable selection by replacing the ${\ell }_2$-penalty with other penalties for the support vector machine (SVM). However, these high-dimensional SVM methods usually do not take into account the special structure among covariates (features). In this article, we consider a classification problem, where the covariates are ordered in some meaningful way, and the number of covariates p can be much larger than the sample size n. We propose a structured sparse SVM to tackle this type of problems, which combines the non-convex penalty and cubic spline estimation procedure (i.e. penalizing second-order derivatives of the coefficients) to the SVM. From a theoretical point of view, the proposed method satisfies the local oracle property. Simulations show that the method works effectively both in feature selection and classification accuracy. A real application is conducted to illustrate the benefits of the method.

1. Introduction

Classification is one of the most important research fields in statistics and machine learning, and it is also a common practical problem. The support vector machine (SVM)[19] is a powerful classification tool with high accuracy and great flexibility. In this article, we will focus on a classification with n cases having class labels $\{y_i\in \{1,-1\};i=1,\dots n\}$ and features $\{x_{ij};i=1,2,\dots ,n,j=1,2,\dots ,p\}$. The SVM has an equivalent formulation as the ${\ell }_2$ penalized hinge loss [11]:

$$\underset{({\mathrm{\text{β}}}_0,\mathrm{\text{β}})}{min}\frac{1}{n}\sum _{i=1}^n\ell \{y_i({\mathrm{\text{β}}}_0+{\mathit{x}}_i^{\mathrm{\top }}\mathrm{\text{β}})\}+\frac{\lambda }{2}\parallel \mathrm{\text{β}}{\parallel }^2,$$ (1)

where the loss $\ell \left(t\right)=[1-t{]}_{+}$ is called hinge loss, and $\parallel \cdot \parallel$ is the ${\ell }_2$-norm. λ is the tuning parameter, which is used to control the tradeoff between loss and penalty.

In the application of high-dimensional data classification, several attempts have been made to achieve variable selection by replacing the ${\ell }_2$-penalty with other penalties for the SVM, such as ${\ell }_1$-SVM [1,28], ${\ell }_0$-SVM and ${\ell }_{\mathrm{\infty }}$-SVM [10], ${\ell }_p$-SVM [4], SCAD-SVM [25,27], Hybrid Huberized SVM [20,21], and MCP-SVM [27]. Since the hinge loss does not have the first derivative, it causes some difficulties in the calculation. [6,15] considered square hinge loss in the SVM; [17,20–22] suggested using the Huberized hinge loss in the SVM. [17] point out that the Huberized regularized model paths are both less affected by the outlier than the non-Huberized squared loss.

This paper concerns a class of structured sparse classification problems with ordering features, i.e. $x_j$ that can be ordered as $x_1,x_2,\dots ,x_p$ in some sense. A motivating example comes from protein mass spectroscopy. For each blood serum sample i, we observed the intensity $x_{ij}$ for many time-of-flight values $t_j$. Time of flight is related to the mass over charge ratio m/z of the constituent proteins in the blood. Figure 1 shows an example that protein mass spectroscopy taken from [16]. We plot intensity $x_{ij}$ on the vertical Y -axis against m/z on the horizontal x-axis. The features are ordered in a meaningful way, i.e. $x_{ij}$ are ordered by m/z, which may lead to a high correlation among closely located variables. The SVM methods mentioned above for processing high-dimensional data classification, do not consider the structure in which the variables are arranged in order. Our goal is to predict the label from the ordered features, especially for $p\gg n$.

Figure 1.

Protein mass spectroscopy data: average profiles from control (–) and ovarian cancer patients (–).

Besides the above example, there are also many data like this, such as the gene expression data in microarray studies, single nucleotide polymorphisms (SNPs) data in genome-wide association studies (GWAS), graph and image data[9]. Those special structures among variables may lead to successive coefficients vary slowly. Fused lasso [18] encourages flatness of the coefficients by penalizing the ${\ell }_1$-norm of coefficients' successive differences. However, it may not perform well when the features vary smoothly, rather than being like a step function. To capture the smooth features in a group, smooth-lasso [12] replaces the ${\ell }_1$-penalty of the difference of the adjacent coefficients in the fused lasso by the ${\ell }_2$-penalty. Recently, [9] proposed the spline-lasso and spline-MCP, in which they imposed an ${\ell }_2$-penalty on the discrete version of the second derivatives of coefficients. However, these methods mentioned above are mainly used in the regression.

To the best of our knowledge, the present article is the first to develop theory and methodology for SVM to incorporate the ordered structure among predictors. This study may advance from the existing ones along the following aspects. First, the structured sparse SVM can achieve variable selection as well as capture the ordered structure of features. The subsequent numerical analysis proves that ignoring this data structure has an impact on the accuracy of the classification and the variable selection. Second, we can theoretically prove that our method has local oracle properties. Even when the number of covariates grows exponentially with the sample size, the local oracle property still holds for the structured sparse SVM. Finally, the algorithm and asymptotic properties are established based on the general form, and we can advocate many kinds of loss functions in the formulation of structured sparse SVM, such as Huberized hinge loss and squared hinge loss.

The rest of the article is organized as follows. In Section 2, we describe the model, an efficient algorithm, and local oracle properties for structured sparse SVM. Simulation results and an application of the proposed method to a protein mass spectroscopy dataset are presented in Sections 3 and 4. Discussions of the proposed method and results are given in Section 5. Proofs for the oracle properties of structured sparse SVM are provided in the Appendix.

2. Structured sparse support vector machine

2.1. Methodology

In this paper, we allow the number of covariates p to increase with the sample size n. It is even possible that p is much larger than n. We assume that the true parameter is sparse, and the features are ordered in some meaningful way. Thus, we need to get an estimator that enjoys the structured sparse property.

The structured sparse SVM is formulated in terms of a loss function that is regularized by penalty terms. Our proposed minimization objective function is

$$\underset{({\mathrm{\text{β}}}_0,\mathrm{\text{β}})}{min}\frac{1}{n}\sum _{i=1}^n\ell \{y_i({\mathrm{\text{β}}}_0+{\mathit{x}}_i^{\mathrm{\top }}\mathrm{\text{β}})\}+\sum _{j=1}^p{p}_{{\lambda }_1}(|{\mathrm{\text{β}}}_j|)+{\lambda }_2\sum _{j=2}^{p-1}({\mathrm{\Delta }}_j^{\left(2\right)}\mathrm{\text{β}}{)}^2.$$ (2)

In (2), the first part is a convex loss function. We can advocate many kinds of loss functions. Huberized hinge loss,

$${\ell }_{\delta }\left(t\right)=\left\{\begin{array}{ll}0,& t>1,\\ (1-t{)}^2/(2\delta ),& 1-\delta <t\le 1,\\ 1-t-\delta /2,& t\le 1-\delta ,\end{array}\right.$$

is adopted in this paper. We fix the pre-specified constant $\delta =2$ following by [21]. The results with other losses are provided in the supplemental materials.

The second part is used to achieve variable selection. We consider the penalized SVM with a general class of non-convex penalties, such as the smoothly clipped absolute deviation (SCAD) penalty [7] and the minimax concave penalty (MCP) [24]. The SCAD penalty is defined by $p_{\lambda }\left(x\right)=\lambda {\int }_0^{|x|}min\{1,(a-t/\lambda {)}_{+}/(a-1)\}\mathrm{d}t$ for some a>2. The MCP is defined by $\lambda {\int }_0^{|x|}\{1-t/(a\lambda ){\}}_{+}\mathrm{d}t$ for some a>1. The experiments with different a values are presented in supplemental materials. We find our results to be insensitive to these choices, and for brevity, we fixed a = 3.7 for SCAD penalty and a = 3 for MCP as suggested in the literature [3,7,27,29].

The third part mimics the cubic spline to encourage the smoothness of coefficients. As the coefficient of the variables might vary smoothly, we encourage the smoothness of coefficients by penalizing the ${\ell }_2$ norm of the discrete version of the second-order derivatives of the coefficients. Denote the first- and second-order difference (or discrete versions of derivatives) of the coefficients by ${\mathrm{\Delta }}_j\mathrm{\text{β}}=:({\mathrm{\text{β}}}_{j+1}-{\mathrm{\text{β}}}_j)$ and ${\mathrm{\Delta }}_j^{\left(2\right)}\mathrm{\text{β}}={\mathrm{\Delta }}_j\mathrm{\text{β}}-{\mathrm{\Delta }}_{j-1}\mathrm{\text{β}}={\mathrm{\text{β}}}_{j+1}-2{\mathrm{\text{β}}}_j+{\mathrm{\text{β}}}_{j-1}$. Then the ${\ell }_2$ norm of the discrete version of the second-order derivatives of the coefficients is $\sum _{j=2}^{p-1}({\mathrm{\Delta }}_j^{\left(2\right)}\mathrm{\text{β}}{)}^2$. The estimator by minimizing (2) enjoys structured sparse property.

Remark 2.1

The idea of the third part in (2) is similar to spline-lasso [9], which is used in the regression. The computation of spline-lasso is converted to lasso through a certain transformation. However, the conversion is no longer applicable when we solve the SVM problem. The computation is more complicated for the SVM problem than regression. The details of the algorithm are shown in Section 2.2.

2.2. Algorithm

Then we give the algorithm to solve this problem. Without loss of generality, we assume that the input data are standardized: $\sum _{i=1}^n{x}_{ij}/n=0$, $\sum _{i=1}^n{x}_{ij}^2/n=1$, $j=1,2,\dots ,p.$ We use the generalized coordinate descent (GCD) algorithm [22] to calculate the structured sparse SVM problem.

When using the GCD algorithm, the loss function $\ell (\cdot )$ should satisfy the following condition

$$\ell (t+a)\le \ell \left(a\right)+{\ell }^{\prime }\left(a\right)t+\frac{M}{2}{t}^2\mathrm{\forall }a,t,$$ (3)

where M is a constant greater than 0. The corresponding M value is $2/\delta$ for Huberized hinge loss. It can be proved that the common loss functions such as Huberized hinge loss, logistic loss, and square hinge loss satisfy the above condition. Although the hinge loss, the loss for the standard SVM, does not satisfy (3), [21] showed that Huberized hinge function with the parameter $\delta =0.01$ is nearly identical to the hinge loss.

Let D be a $(p-2)\times p$ matrix with $D_{ii}=D_{i,i+2}=1$, $D_{i,i+1}=-2$, and $D_{ij}=0$ otherwise. Given current estimate $\{{\widetilde{\mathrm{\text{β}}}}_0,\stackrel{\mathbf{~}}{\mathrm{\text{β}}}\}$. Define the current margin $r_i=y_i({\widetilde{\mathrm{\text{β}}}}_0+{\mathit{x}}_i^{\mathrm{\top }}\stackrel{\mathbf{~}}{\mathrm{\text{β}}})$. The coordinate descent algorithm cyclically minimizes

$$\begin{array}{rl}F({\mathrm{\text{β}}}_j|{\widetilde{\mathrm{\text{β}}}}_0,\stackrel{\mathbf{~}}{\mathrm{\text{β}}})& =\frac{1}{n}\sum _{i=1}^n\ell \{r_i+y_i{x}_{ij}({\mathrm{\text{β}}}_j-{\widetilde{\mathrm{\text{β}}}}_j)\}+p_{{\lambda }_1}(|{\mathrm{\text{β}}}_j|)\\ & +{\lambda }_2(D^{\mathrm{\top }}D{)}_{jj}{\mathrm{\text{β}}}_j^2+2{\lambda }_2\sum _{l=1,i\ne j}^n{\left(D^{\mathrm{\top }}D\right)}_{lj}{\widetilde{\mathrm{\text{β}}}}_l{\mathrm{\text{β}}}_j\end{array}$$ (4)

with respect to ${\mathrm{\text{β}}}_j$. According to local linear approximation (LLA) [29], we have $p_{{\lambda }_1}\left(|{\mathrm{\text{β}}}_j|\right)\approx p_{{\lambda }_1}\left(|{\widetilde{\mathrm{\text{β}}}}_j|\right)+p_{{\lambda }_1}^{\prime }\left(|{\widetilde{\mathrm{\text{β}}}}_j|\right)(|{\mathrm{\text{β}}}_j|-|{\widetilde{\mathrm{\text{β}}}}_j|)$, for ${\mathrm{\text{β}}}_j\approx {\widetilde{\mathrm{\text{β}}}}_j$. As pointed out by a referee, CCCP (constrained concave–convex procedure) algorithm is also an efficient algorithm for solving this problem, which is worth investigating as future work. When $\ell (\cdot )$ satisfies (3), we can get $F({\mathrm{\text{β}}}_j|{\widetilde{\mathrm{\text{β}}}}_0,\stackrel{\mathbf{~}}{\mathrm{\text{β}}})\le \hat{F}({\mathrm{\text{β}}}_j|{\widetilde{\mathrm{\text{β}}}}_0,\stackrel{\mathbf{~}}{\mathrm{\text{β}}})$, where

$$\begin{array}{rl}\hat{F}({\mathrm{\text{β}}}_j|{\widetilde{\mathrm{\text{β}}}}_0,\stackrel{\mathbf{~}}{\mathrm{\text{β}}})& =\frac{1}{n}\sum _{i=1}^n\ell \left(r_i\right)+\frac{1}{n}\sum _{i=1}^n{\ell }^{\prime }\left(r_i\right)y_i{x}_{ij}({\mathrm{\text{β}}}_j-{\widetilde{\mathrm{\text{β}}}}_j)+\frac{M}{2}({\mathrm{\text{β}}}_j-{\widetilde{\mathrm{\text{β}}}}_j{)}^2\\ & +p_{{\lambda }_1}^{\prime }\left(|{\widetilde{\mathrm{\text{β}}}}_j|\right)|{\mathrm{\text{β}}}_j|+{\lambda }_2(D^{\mathrm{\top }}D{)}_{jj}{\mathrm{\text{β}}}_j^2+2{\lambda }_2\sum _{l=1,l\ne j}^p{\left(D^{\mathrm{\top }}D\right)}_{lj}{\widetilde{\mathrm{\text{β}}}}_l{\mathrm{\text{β}}}_j.\end{array}$$ (5)

Since $\hat{F}$ is a quadratic majorization function of F, we can get the new update by minimizing $\hat{F}$:

$${\widehat{\mathrm{\text{β}}}}_j^{new}=\underset{{\mathrm{\text{β}}}_j}{\arg min}\hat{F}({\mathrm{\text{β}}}_j|{\widetilde{\mathrm{\text{β}}}}_0,\stackrel{\mathbf{~}}{\mathrm{\text{β}}})=\frac{S(z,p_{{\lambda }_1}^{\prime }\left(|{\widetilde{\mathrm{\text{β}}}}_j|\right))}{M+2{\lambda }_2{\left(D^{\mathrm{\top }}D\right)}_{jj}},$$ (6)

where $S(z,t)=(|z|-t{)}_{+}\mathrm{s}\mathrm{i}\mathrm{g}\mathrm{n}\left(z\right),z=M{\widetilde{\mathrm{\text{β}}}}_j-\frac{1}{n}\sum _{i=1}^n{\ell }^{\prime }\left(r_i\right)y_i{x}_{ij}-2{\lambda }_2\sum _{l=1,i\ne j}^n{\left(D^{\mathrm{\top }}D\right)}_{lj}{\widetilde{\mathrm{\text{β}}}}_l$.

Likewise, we can update the intercept by minimizing

$$\hat{F}({\mathrm{\text{β}}}_0|{\widetilde{\mathrm{\text{β}}}}_0,\stackrel{\mathbf{~}}{\mathrm{\text{β}}})=\frac{1}{n}\sum _{i=1}^n\ell \left(r_i\right)+\frac{1}{n}\sum _{i=1}^n{\ell }^{\prime }\left(r_i\right)y_i({\mathrm{\text{β}}}_0-{\widetilde{\mathrm{\text{β}}}}_0)+\frac{M}{2}({\mathrm{\text{β}}}_0-{\widetilde{\mathrm{\text{β}}}}_0{)}^2.$$ (7)

Then the intercept is updated by

$${\widehat{\mathrm{\text{β}}}}_0^{new}=\underset{{\mathrm{\text{β}}}_0}{\arg min}Q({\mathrm{\text{β}}}_0|{\widetilde{\mathrm{\text{β}}}}_0,\stackrel{\mathbf{~}}{\mathrm{\text{β}}})={\widetilde{\mathrm{\text{β}}}}_0-\frac{\sum _{i=1}^n{\ell }^{\prime }\left(r_i\right)y_i}{Mn}.$$ (8)

Then we can iterate (4)–(8) until convergence.

Remark 2.2

In this paper, we specify the initial value by $L_1$ penalized SVM following by [27], and it leads to a satisfactory result.

Remark 2.3

The above algorithm satisfies the majorization–minimization (MM) principle [5,13,14], and the MM principle ensures the descent property of the GCD algorithm. The proof is similar to [21,22] and we omit here.

2.3. Asymptotic properties

In this subsection, we establish the theory of the local oracle property for the structured sparse SVM, namely the oracle estimator is one of the local minimizers of (2).

Since ${\mathrm{\text{β}}}_0$ does not affect variable selection, we make ${\mathrm{\text{β}}}_0=0$ for the convenience of expression without loss of generality in this section. Let ${\mathrm{\text{β}}}^{\ast }=({\mathrm{\text{β}}}_1^{\ast },{\mathrm{\text{β}}}_2^{\ast },\dots ,{\mathrm{\text{β}}}_p^{\ast }{)}^{\mathrm{\top }}$ denote the true parameter value, which is defined as the minimizer of the population loss: ${\mathrm{\text{β}}}^{\ast }=\underset{\mathrm{\text{β}}}{\arg min}L\left(\mathrm{\text{β}}\right)=\underset{\mathrm{\text{β}}}{\arg min}\mathbb{E}\left\{\ell \right(y{\mathit{x}}^{\mathrm{\top }}\mathrm{\text{β}}\left)\right\}.$

We use $p_n$ in this section to denote the number of features. Let $\mathcal{A}=\{j:{\mathrm{\text{β}}}_j^{\ast }\ne 0,1\le j\le p_n\}$ be the index set of the non-zero coefficients, and ${\mathcal{A}}^c=\{j:{\mathrm{\text{β}}}_j^{\ast }=0,1\le j\le p_n\}$ be the index set of the zero coefficients. $q_n=|\mathcal{A}|$ is the cardinality of set  $\mathcal{A}$. $D_{\mathcal{A}}$ is a submatrix formed by D removing the column corresponding to the element in the ${\mathcal{A}}^c$. ${\mathrm{\text{β}}}_{{\mathcal{A}}^c}=({\mathrm{\text{β}}}_j{)}_{j\in {\mathcal{A}}^c}$ is a vector composed of the components of $\mathrm{\text{β}}$ corresponding to the elements in ${\mathcal{A}}^c$. Then the oracle estimate $\stackrel{\mathbf{ˆ}}{\mathrm{\text{β}}}$ is defined as $\stackrel{\mathbf{ˆ}}{\mathrm{\text{β}}}=\underset{{\mathrm{\text{β}}}_{{\mathcal{A}}^c}=0}{\arg min}\left\{\frac{1}{n}\sum _{i=1}^n\ell \left(y_i{\mathit{x}}_{i\mathcal{A}}^{\mathrm{\top }}{\mathrm{\text{β}}}_{\mathcal{A}}\right)+{\lambda }_{2n}{\mathrm{\text{β}}}_{\mathcal{A}}^{\mathrm{\top }}{D}_{\mathcal{A}}^{\mathrm{\top }}{D}_{\mathcal{A}}{\mathrm{\text{β}}}_{\mathcal{A}}\right\}.$

Theorem 2.1

Assume that the conditions 1–5 listed in the Appendix hold. Let $B_n({\lambda }_{1n},{\lambda }_{2n})$ be the set of local minimizers of the objective function

$$Q_n\left(\mathrm{\text{β}}\right)=L_n\left(\mathrm{\text{β}}\right)+\sum _{j=1}^p{p}_{{\lambda }_{1n}}\left(|{\mathrm{\text{β}}}_j|\right)+{\lambda }_{2n}{\mathrm{\text{β}}}^{\mathrm{\top }}{D}^{\mathrm{\top }}D\mathrm{\text{β}}$$

with regularization parameter ${\lambda }_{1n}$, ${\lambda }_{2n}$. The oracle estimator $\stackrel{\mathbf{ˆ}}{\mathrm{\text{β}}}=({\stackrel{\mathbf{ˆ}}{\mathrm{\text{β}}}}_{\mathcal{A}}^{\mathrm{\top }},0^{\mathrm{\top }}{)}^{\mathrm{\top }}$ satisfies

$$Pr\{\stackrel{\mathbf{ˆ}}{\mathrm{\text{β}}}\in B_n({\lambda }_{1n},{\lambda }_{2n})\}\to 1$$

as $n\to \mathrm{\infty }$, if $q_n{n}^{-1/2}\log p_n\log n=o\left({\lambda }_{1n}\right)$, ${\lambda }_{2n}{q_n}^{1/2}{n}^{-1/2}=o\left({\lambda }_{1n}\right)$, and ${\lambda }_{1n}=o\left(n^{-(1-c_3)/2}\right)$.

From Theorem 2.1, we can see that if we take ${\lambda }_{1n}=n^{-1/2+\tau }$ for some $c_1<\tau <c_3/2$, then the oracle property holds even for $p=o\{\exp (n^{(\tau -c_1)/2}\left)\right\}$. Thus, even when the number of covariates grows exponentially with the sample size, the local oracle property still holds for the structured sparse SVM.

3. Simulations

In this section, numerical experiments are conducted to study the performance of our proposed method. We use Spline-penalty-HSVM, where penalty includes SCAD and MCP, to represent our proposed method (i.e. Spline-SCAD-HSVM and Spline-MCP-HSVM). To investigate the performance, we compared performances of the proposed method with other alternatives without considering structured sparsity: SCAD-HSVM and MCP-HSVM.

Three data generation processes are considered in this paper. We set n = 100 and p = 1000. In Example 3.1, the non-zero coefficients of the variables are completely smooth by position. Partial non-zero coefficients are smooth in Example 3.2. For Example 3.3, the non-zero coefficients are not smooth. Within each example, our simulated data consist of a training set and a testing set. Models are fitted on training data only, and the testing set with sample size 500 is used to show the predictions of each method. The optimal regularization parameters ${\lambda }_1$ and ${\lambda }_2$ are selected on a 15-by-20 meshgrid through a 5-fold cross validation. Possible values of ${\lambda }_2$ are from $[0.1,0.2,\dots ,1.9,2]$. For each fixed ${\lambda }_2$, we compute the solutions for a fine grid of ${\lambda }_1$s. Following [21], we start with ${\lambda }_{1(max)}$ which is the smallest ${\lambda }_1$ to set all ${\mathrm{\text{β}}}_j$ to be zero, and set ${\lambda }_{1(min)}=0.01{\lambda }_{1(max)}$. Between ${\lambda }_{1(min)}$ and ${\lambda }_{1(max)}$, 15 points are placed uniformly in the log-scale. Then we select the optimal regularization parameters that achieve the maximum of the classification accuracy rate. Here are the details of the three scenarios.

Example 3.1

Consider $\mathit{x}\sim N(0,\mathbf{\Sigma })$ with $\mathbf{\Sigma }=({0.5}^{|i-j|}{)}_{p\times p}$, ${\mathrm{\text{β}}}_j=j/40$ for $j=1,2,\dots ,20$; ${\mathrm{\text{β}}}_j=1-j/40$ for $j=21,\dots ,40$; ${\mathrm{\text{β}}}_j=-\sin \left(\pi j/40\right)$ for $j=81,\dots ,120$; ${\mathrm{\text{β}}}_j=0.5(1-\cos (\pi j/20\left)\right)$, for $j=161,\dots ,200$; and ${\mathrm{\text{β}}}_j=0$ for the otherwise. $Pr(y=-1)=1/\left(1+\exp (-{\mathit{x}}^{\mathrm{\top }}\mathrm{\text{β}})\right)$, and $Pr(y=1)=1/\left(1+\exp \left({\mathit{x}}^{\mathrm{\top }}\mathrm{\text{β}}\right)\right)$. The Bayes rule is $\mathrm{s}\mathrm{g}\mathrm{n}\left({\mathit{x}}^{\mathrm{\top }}\mathrm{\text{β}}\right)$ with Bayes error $5.2\mathrm{\%}$.

Example 3.2

The setting of $\mathit{x}$ is the same as in Example 3.1. ${\mathrm{\text{β}}}_j$ takes the same value as in Example 1 for $j=1,2,\dots ,120$; ${\mathrm{\text{β}}}_j\sim \mathrm{U}\mathrm{n}\mathrm{i}\mathrm{f}\mathrm{o}\mathrm{r}\mathrm{m}(-0.5,0.5)$, for $j=160,\dots ,200$; and ${\mathrm{\text{β}}}_j=0$ for the otherwise. $Pr(y=1|\mathit{x})=\mathrm{\Phi }\left({\mathit{x}}^{\mathrm{\top }}\mathrm{\text{β}}\right)$, where $\mathrm{\Phi }(\cdot )$ is the distribution function of standard normal distribution. The Bayes rule is $\mathrm{s}\mathrm{g}\mathrm{n}\left({\mathit{x}}^{\mathrm{\top }}\mathrm{\text{β}}\right)$ with Bayes error $9.1\mathrm{\%}$.

Example 3.3

The generations of $\mathit{x}$ and $\mathit{y}$ are the same as in Example 3.1. ${\mathrm{\text{β}}}_j\sim \mathrm{U}\mathrm{n}\mathrm{i}\mathrm{f}\mathrm{o}\mathrm{r}\mathrm{m}(0,1)$, for $j=1,2,\dots ,40$; and ${\mathrm{\text{β}}}_j=0$ for the otherwise. The Bayes rule is $\mathrm{s}\mathrm{g}\mathrm{n}\left({\mathit{x}}^{\mathrm{\top }}\mathrm{\text{β}}\right)$ with Bayes error $8.3\mathrm{\%}$.

The performance of different methods will be examined in two aspects: classification prediction and feature selection. In the evaluation of classification prediction, the classification accuracy rate (ACC), area under curve (AUC), true positive rate (TPR) and false positive rate (FPR) are adopted. As for feature selection, we compare TPR and FPR for different methods.

The results of the simulations are shown in Table 1. When the features vary smoothly, the SVM model with the spline penalty is significantly better in classification prediction and variable selection than the SVM model without the spline penalty (Example 3.1 and 3.2). The proposed method performs better, especially when non-zero coefficients of the variables are completely smooth by position in terms of classification prediction. In Figure 2, we present the estimation results for the correlated features in Example 3.1, by four different methods. From the figure, we can conclude that both the Spline-SCAD-HSVM and Spline-MCP-HSVM give a good estimate of the coefficients, while SCAD-HSVM and MCP-HSVM do not clean out the noisy signals very well. The improvement is not surprising since Spline-SCAD-HSVM and Spline-MCP-HSVM can capture the smoothing changes in coefficients. When the non-zero coefficients are not smooth, which means structural information described in this paper does not exist, the performances of both methods are similar (Example 3.3). This observation indicates that the proposed models are also applicable even when the features do not vary smoothly.

Table 1.

The simulation results obtained from 100 Monte Carlo repetitions (with standard errors in parentheses).

	Classification prediction				Variable selection
Method	ACC	AUC	TPR	FPR	TPR	FPR
Example 3.1
SCAD-HSVM	0.782 (0.021)	0.874 (0.018)	0.769 (0.046)	0.205 (0.040)	0.732 (0.038)	0.309 (0.029)
MCP-HSVM	0.775 (0.025)	0.867 ( 0.024)	0.789 (0.031)	0.237 (0.047)	0.761 (0.028)	0.304 (0.036)
Spline-SCAD-HSVM	0.922 (0.022)	0.983 (0.011)	0.911 (0.034)	0.067 (0.030)	0.874 (0.019)	0.254 (0.021)
Spline-MCP-HSVM	0.932 (0.010)	0.987 (0.010)	0.920 (0.033)	0.055 (0.028)	0.868 (0.024)	0.143 (0.019)
Example 3.2
SCAD-HSVM	0.756 (0.011)	0.845 (0.008)	0.723 (0.010)	0.258 (0.008)	0.611 (0.019)	0.253 (0.008)
MCP-HSVM	0.738 (0.015)	0.834 (0.007)	0.715 (0.009)	0.263 (0.008)	0.606 (0.020)	0.265 (0.009)
Spline-SCAD-HSVM	0.798 (0.013)	0.853 (0.011)	0.729 (0.009)	0.259 (0.008)	0.799 (0.019)	0.223 (0.010)
Spline-MCP-HSVM	0.804 (0.012)	0.854 (0.012)	0.731 (0.015)	0.257 (0.008)	0.799 (0.020)	0.225 (0.011)
Example 3.3
SCAD-HSVM	0.788 (0.009)	0.880 (0.012)	0.799 (0.010)	0.244 (0.002)	0.782 (0.018)	0.215 (0.005)
MCP-HSVM	0.788 (0.009)	0.880 (0.012)	0.799 (0.010)	0.244 (0.002)	0.781 (0.020)	0.215 (0.005)
Spline-SCAD-HSVM	0.801 (0.006)	0.878 (0.011)	0.791 (0.010)	0.290 (0.003)	0.790 (0.019)	0.229 (0.011)
Spline-MCP-HSVM	0.808 (0.006)	0.878 (0.011)	0.795 (0.010)	0.281 (0.003)	0.790 (0.015)	0.229 (0.012)

Figure 2.

The average estimation results for the correlated features in Example 1 of 100 Monte Carlo repetitions, by four different methods: SCAD-HSVM, MCP-HSVM, Spline-SCAD-HSVM and Spline-MCP-HSVM. The solid line curve is the true $\mathrm{\text{β}}$, and the scatter dot represents the estimation for each method.

4. Real data analysis

In this section, we apply our methods to a dataset of Ovarian Dataset 8-7-02. The dataset is provided by the US Food and Drug Administration (FDA) and the National Cancer Institute (NCI), which can be downloaded and accessed at http://home.ccr.cancer.gov/. The data were collected as serum samples from normal and cancer patients, and the mass spectrometry technique was combined with the WCX2 protein chip and SELDI-TOF. The sample set included 91 controls and 162 ovarian cancers, which were not randomized. Each mass spectrometer sample contains a 15154-dimensional mass-to-charge ratio (m/z) /intensity characteristic. As mentioned in Section 1, the features are ordered in a meaningful way. Following the original researchers, we ignored m/z-sites below 100, where chemical artifacts can occur [16].

We randomly choose 173 samples from data as the training set, and the remaining 80 samples are used as the testing set. Four methods with Huberized hinge loss, i.e. SCAD-HSVM, MCP-HSVM, Spline-SCAD-HSVM, and Spline-MCP-HSVM, are fitted using the training set. Additional results with other losses are provided in the supplemental materials. Tuning parameters are chosen by 5-fold cross validation base on the training set. We select the optimal regularization parameters that achieve the maximum of the classification accuracy rate among grid points using two-dimensional grid search. We run the sample-splitting method 100 times, and the results are summarized in Table 2. We can see that the performance of the Spline-SCAD-HSVM and Spline-MCP-HSVM is slightly better than the SCAD-HSVM and MCP-HSVM. The ACC, AUC and TPR are slightly higher when we consider the model that explicitly incorporates the special structures among the features.

Table 2.

Results of 100 random splits of the ovarian cancer dataset (with standard errors in parentheses).

Method	ACC	AUC	TPR	FPR
SCAD-HSVM	0.919 (0.025)	0.968 (0.010)	0.930 (0.021)	0.061 (0.012)
MCP-HSVM	0.921 (0.026)	0.971 (0.008)	0.932 (0.022)	0.060 (0.015)
Spline-SCAD-HSVM	0.947 (0.018)	0.989 (0.004)	0.972 (0.009)	0.043 (0.011)
Spline-MCP-HSVM	0.947 (0.019)	0.992 (0.003)	0.975 (0.009)	0.041 (0.010)

To complement the estimation and identification analysis, we also evaluate the stability of analysis by computing the observed occurrence index (OOI). For each feature identified using the training data, we compute its probability of being identified out of the 100 resamplings; this probability has been referred to as the OOI. The median OOI values of SCAD-HSVM, MCP-HSVM, Spline-SCAD-HSVM, and Spline-MCP-HSVM are 0.736, 0.739, 0.857, and 0.862, respectively. Figure 3 shows the number of selected proteins versus the selection frequency of four different methods out of the 100 random splits. We can conclude that the OOI value of the model with the spline item is significantly higher, which indicates that Spline-SCAD-HSVM and Spline-MCP-HSVM are more stable than SCAD-HSVM and MCP-HSVM.

Figure 3.

The figure shows the number of selected proteins versus the selection frequency of four different methods: (a) SCAD-HSVM, (b) MCP-HSVM, (c) Spline-SCAD-HSVM, and (d) Spline-MCP-HSVM.

5. Discussion

In this article, we consider a high-dimensional data classification problem, where the features are ordered in some meaningful way. When the coefficients are sparse and change smoothly, we propose a structured sparse SVM, which combines the non-convex penalty and cubic spline estimation procedure (i.e. penalizing second-order derivatives of the coefficients) to the SVM, and proved that it satisfies the local oracle property under some conditions. The simulation and empirical results show that the proposed method has a higher accuracy of classification and prediction compared with the existing methods.

In the future, we want to work on more complex data. For example, when the high-dimensional data variables have group structure information and the intra-group features are ordered in some meaningful way. Moreover, our approach could also be extended to the framework of semisupervised learning and multi-class classification.

Appendix 1. Regularity conditions.

To facilitate our technical proofs, we impose the following regularity conditions.

Condition A.1

The loss function $\ell (\cdot )$ is convex and it has a first order continuous derivative. There exist constants $M_1$ and $M_2$, such that $|{\ell }^{\prime }\left(t\right)|\le M_1(|t|+1),|\mathrm{\partial }\left({\ell }^{\prime }\right(t\left)\right)|\le M_2,\mathrm{\forall }t,$ where $\mathrm{\partial }(\cdot )$ represents the subgradient.

Condition A.2

$q_n=O\left(n^{c_1}\right),0\le c_1<1/2$; ${\lambda }_{2n}\parallel D{\mathrm{\text{β}}}^{\ast }\parallel =O\left(n^{-c_2}\right)$, $(1-c_1)/2<c_2\le 1/2$.

Condition A.3

The Hessian matrix $H\left({\mathrm{\text{β}}}_{\mathcal{A}}\right)=\mathbb{E}\left[{\mathrm{\nabla }}^2\ell \right(y{\mathit{x}}_{\mathcal{A}}^{\mathrm{\top }}{\mathrm{\text{β}}}_{\mathcal{A}}\left)\right]$ satisfies conditions

$$0<M_3<{\lambda }_{min}\left\{H\left({\mathrm{\text{β}}}_{\mathcal{A}}^{\ast }\right)\right\}\le {\lambda }_{max}\left\{H\left({\mathrm{\text{β}}}_{\mathcal{A}}^{\ast }\right)\right\}<M_4<\mathrm{\infty },$$

where ${\mathit{x}}_{\mathcal{A}}$ is the matrix formed by $\mathit{x}$ removing the column corresponding to the element in the ${\mathcal{A}}^c$. where ${\lambda }_{min}$ and ${\lambda }_{max}$ denote the smallest and largest eigenvalue, respectively.

Condition A.4

There is a constant $M_5>0$ such that ${\lambda }_{max}\left(n^{-1}{\mathit{x}}_{\mathcal{A}}^{\mathrm{\top }}{\mathit{x}}_{\mathcal{A}}\right)\le M_5$. It is further assumed that $x_{ij}$ are sub-Gaussian random variables for $1\le i\le n$, $j\in {\mathcal{A}}^c$.

Condition A.5 Condition on the true model dimension —

There exist positive constants $c_3$ and $M_6$ such that $1-c_1\le c_3\le 1$ and $n^{(1-c_3)/2}\underset{j\in \mathcal{A}}{min}|{\mathrm{\text{β}}}_j^{\ast }|\ge M_6$.

Remark A.1

Condition 1 requires that the loss function be smooth and that the change is gentle, which is satisfactory for some common SVM loss functions, such as Huberized hinge loss function and square hinge loss function. Condition 2 states that the divergence rate of the number of non-zero coefficients cannot be faster than $n^{1/2}$, and the coefficient of the variable is slowly changing in position, which supports our introduction of spline penalty. Under Conditions 3, the Hessian matrix of the loss function is assumed to be positive definite, and its eigenvalues are uniformly bounded. The condition on the largest eigenvalues of the design matrix, which is assumed in Condition 4, is similar to that of [23,26,27]. Condition 5 simply states that the signals cannot decay too quickly.

Appendix 2. Some lemmas.

The proof of Theorem 2.1 relies on the following lemmas.

Lemma A.1

Assume that Conditions 1–5 are satisfied. Then the oracle estimator satisfies $\parallel {\stackrel{\mathbf{ˆ}}{\mathrm{\text{β}}}}_{\mathcal{A}}-{\mathrm{\text{β}}}_{\mathcal{A}}^{\ast }\parallel =O_p\left(\sqrt{q_n/n}\right)$.

Proof.

Let ${\alpha }_n=\sqrt{q_n/n}$, and $Q_n\left({\mathrm{\text{β}}}_{\mathcal{A}}\right)=\frac{1}{n}\sum _{i=1}^n\ell \left(y_i{\mathit{x}}_{i\mathcal{A}}^{\mathrm{\top }}{\mathrm{\text{β}}}_{\mathcal{A}}\right)+{\lambda }_{2n}{\mathrm{\text{β}}}_{\mathcal{A}}^{\mathrm{\top }}{D}_{\mathcal{A}}^{\mathrm{\top }}{D}_{\mathcal{A}}{\mathrm{\text{β}}}_{\mathcal{A}}$, we want to show that for any given $ϵ>0$, there exists a constant C>0 such that

$$Pr\left\{\underset{\parallel \mathit{u}\parallel =C}{inf}{Q}_n({\mathrm{\text{β}}}_{\mathcal{A}}^{\ast }+{\alpha }_n\mathit{u})>Q_n\left({\mathrm{\text{β}}}_{\mathcal{A}}^{\ast }\right)\right\}\ge 1-ϵ.$$ (A.1)

This implies that there exists a local minimum in the ball $\{{\mathrm{\text{β}}}_{\mathcal{A}}^{\ast }+{\alpha }_n\mathit{u}:\parallel \mathit{u}\parallel \le C\}$ with probability at least $1-ϵ$. Hence, there exists a local minimizer such that $\parallel {\stackrel{\mathbf{ˆ}}{\mathrm{\text{β}}}}_{\mathcal{A}}-{\mathrm{\text{β}}}_{\mathcal{A}}^{\ast }\parallel =O_p\left({\alpha }_n\right)$.

Let

$$\begin{array}{rl}{\mathrm{\Lambda }}_n\left(\mathit{u}\right)& =Q_n({\mathrm{\text{β}}}_{\mathcal{A}}^{\ast }+{\alpha }_n\mathit{u})-Q_n\left({\mathrm{\text{β}}}_{\mathcal{A}}^{\ast }\right)\\ & =\frac{1}{n}\sum _{i=1}^n[\ell \{y_i{\mathit{x}}_{i\mathcal{A}}^{\mathrm{\top }}({\mathrm{\text{β}}}_{\mathcal{A}}^{\ast }+{\alpha }_n\mathit{u})\}-\ell \left(y_i{\mathit{x}}_{i\mathcal{A}}^{\mathrm{\top }}{\mathrm{\text{β}}}_{\mathcal{A}}^{\ast }\right)]\\ & +{\lambda }_{2n}\{{({\mathrm{\text{β}}}_{\mathcal{A}}^{\ast }+{\alpha }_n\mathit{u})}^{\mathrm{\top }}{D}_{\mathcal{A}}^{\mathrm{\top }}{D}_{\mathcal{A}}({\mathrm{\text{β}}}_{\mathcal{A}}^{\ast }+{\alpha }_n\mathit{u})-{\mathrm{\text{β}}}_{\mathcal{A}}^{\mathrm{\top }\ast }{D}_{\mathcal{A}}^{\mathrm{\top }}{D}_{\mathcal{A}}{\mathrm{\text{β}}}_{\mathcal{A}}^{\ast }\}.\end{array}$$

By applying Taylor series expansion around ${\mathrm{\text{β}}}^{\ast }$, we have

$$\begin{array}{rl}{\mathrm{\Lambda }}_n\left(\mathit{u}\right)& =\frac{1}{n}\sum _{i=1}^n[{\alpha }_n{\mathrm{\nabla }}^{\mathrm{\top }}\{\ell \left(y_i{\mathit{x}}_{i\mathcal{A}}^{\mathrm{\top }}{\mathrm{\text{β}}}_{\mathcal{A}}^{\ast }\right)\}\mathit{u}+\frac{{\alpha }_n^2}{2}{\mathit{u}}^{\mathrm{\top }}{\mathrm{\nabla }}^2\{\ell \left(y_i{\mathit{x}}_{i\mathcal{A}}^{\mathrm{\top }}{\stackrel{\mathbf{~}}{\mathrm{\text{β}}}}_{\mathcal{A}}\right)\}\mathit{u}]\\ & +2{\lambda }_{2n}{\alpha }_n{\mathrm{\text{β}}}_{\mathcal{A}}^{\ast \mathrm{\top }}{D}_{\mathcal{A}}^{\mathrm{\top }}{D}_{\mathcal{A}}\mathit{u}+{\lambda }_{2n}{\alpha }_n^2{\mathit{u}}^{\mathrm{\top }}{D}_{\mathcal{A}}^{\mathrm{\top }}{D}_{\mathcal{A}}\mathit{u}\\ & \ge \frac{{\alpha }_n}{n}\sum _{i=1}^n{\mathrm{\nabla }}^{\mathrm{\top }}\{\ell \left(y_i{\mathit{x}}_{i\mathcal{A}}^{\mathrm{\top }}{\mathrm{\text{β}}}_{\mathcal{A}}^{\ast }\right)\}\mathit{u}+\frac{{\alpha }_n^2{\mathit{u}}^{\mathrm{\top }}}{2n}\sum _{i=1}^n{\mathrm{\nabla }}^2\{\ell \left(y_i{x}_{i\mathcal{A}}^{\mathrm{\top }}{\stackrel{\mathbf{~}}{\mathrm{\text{β}}}}_{\mathcal{A}}\right)\}\mathit{u}\\ & +2{\lambda }_{2n}{\alpha }_n{\mathrm{\text{β}}}_{\mathcal{A}}^{\ast \mathrm{\top }}{D}_{\mathcal{A}}^{\mathrm{\top }}{D}_{\mathcal{A}}\mathit{u}\\ & \stackrel{\wedge }{=}{I}_1+I_2+I_3,\end{array}$$ (A.2)

where ${\stackrel{\mathbf{~}}{\mathrm{\text{β}}}}_{\mathcal{A}}={\mathrm{\text{β}}}_{\mathcal{A}}^{\ast }+{\alpha }_{n}t\mathit{u}$, 0<t<1. By Conditions 1–3, we have

$$\begin{array}{rl}|I_1|& =|\frac{{\alpha }_n}{n}\sum _{i=1}^n{\mathrm{\nabla }}^{\mathrm{\top }}\{\ell \left(y_i{\mathit{x}}_{i\mathcal{A}}^{\mathrm{\top }}{\mathrm{\text{β}}}_{\mathcal{A}}^{\ast }\right)\}\mathit{u}|\le {\alpha }_n\parallel \frac{1}{n}\sum _{i=1}^n{\mathrm{\nabla }}^{\mathrm{\top }}\{\ell \left(y_i{\mathit{x}}_{i\mathcal{A}}^{\mathrm{\top }}{\mathrm{\text{β}}}_{\mathcal{A}}^{\ast }\right)\}\parallel \cdot \parallel \mathit{u}\parallel \\ & ={\alpha }_n\cdot O_p\left(\sqrt{q_n/n}\right)\parallel \mathit{u}\parallel =O_p\left({\alpha }_n^2\right)\parallel \mathit{u}\parallel .\end{array}$$

With Conditions 1 and 4, using Chebyshev inequality similarly as that in [8], we have when $q_n=O\left(n^{c_1}\right),0\le c_1<1/2$

$$Pr\{∥\frac{1}{n}\sum _{i=1}^n{\mathrm{\nabla }}^2\{\ell \left(y_i{\mathit{x}}_{i\mathcal{A}}^{\mathrm{\top }}{\mathrm{\text{β}}}_{\mathcal{A}}^{\ast }\right)\}-H\left({\mathrm{\text{β}}}_{\mathcal{A}}^{\ast }\right)∥\ge ϵq_n^{-1}\}\le \frac{q_n^2}{n{ϵ}^2}=o\left(1\right),$$

Thus

$$\parallel \frac{1}{n}\sum _{i=1}^n{\mathrm{\nabla }}^2\{\ell \left(y_i{\mathit{x}}_{i\mathcal{A}}^{\mathrm{\top }}{\mathrm{\text{β}}}_{\mathcal{A}}^{\ast }\right)\}-H\left({\mathrm{\text{β}}}_{\mathcal{A}}^{\ast }\right)\parallel =o_p\left(q_n^{-1}\right),$$

Then

$$I_2=\frac{1}{2}{\alpha }_n^2{\mathit{u}}^{\mathrm{\top }}H\left({\mathrm{\text{β}}}_{\mathcal{A}}^{\ast }\right)\mathit{u}\{1+o_p(1\left)\right\}.$$

By choosing a sufficiently large C, the second term $I_2$ dominates the first term $I_1$ uniformly in $\parallel \mathit{u}\parallel =C$. By Cauchy–Schwarz inequality and Condition 2, we have

$$\begin{array}{rl}|I_3|& =|2{\lambda }_{2n}{\alpha }_n{\mathrm{\text{β}}}_{\mathcal{A}}^{\ast \mathrm{\top }}{D}_{\mathcal{A}}^{\mathrm{\top }}{D}_{\mathcal{A}}\mathit{u}|\le 2{\lambda }_{2n}{\alpha }_n\parallel D_{\mathcal{A}}{\mathrm{\text{β}}}_{\mathcal{A}}^{\ast }\parallel \parallel D_{\mathcal{A}}\mathit{u}\parallel \\ & =2{\lambda }_{2n}{\alpha }_n\parallel D{\mathrm{\text{β}}}^{\ast }\parallel \sqrt{{\mathit{u}}^{\mathrm{\top }}{D}_{\mathcal{A}}^{\mathrm{\top }}{D}_{\mathcal{A}}\mathit{u}}\\ & \le 2{\lambda }_{2n}{\alpha }_n\parallel D{\mathrm{\text{β}}}^{\ast }\parallel \parallel \mathit{u}\parallel \sqrt{{\lambda }_{max}\left(D_{\mathcal{A}}^{\mathrm{\top }}{D}_{\mathcal{A}}\right)}=o\left({\alpha }_n^2\right)\parallel \mathit{u}\parallel .\end{array}$$

This is also dominated by the second term of (A2). Hence, by choosing a sufficiently large C, (A1) holds. This completes the proof of the lemma.

Lemma A.2

Assume that Conditions 1–5 hold and that $q_n{n}^{-1/2}\log p_n\log n=o\left({\lambda }_{1n}\right)$. Then

$$Pr\left\{\underset{j\in {\mathcal{A}}^c}{max}|\frac{1}{n}\sum _{i=1}^n{y}_i{x}_{ij}{\ell }^{\prime }\left(y_i{\mathit{x}}_{i\mathcal{A}}^{\mathrm{\top }}{\mathrm{\text{β}}}_{\mathcal{A}}^{\ast }\right)|>\frac{{\lambda }_{1n}}{2}\right\}\to 0,$$

as $n\to \mathrm{\infty }$.

Proof.

Recall that $\mathbb{E}\left[n^{-1}\sum _{i=1}^n{y}_i{\mathit{x}}_{ij}{\ell }^{\prime }\left(y_i{x}_{i\mathcal{A}}^{\mathrm{\top }}{\mathrm{\text{β}}}_{\mathcal{A}}^{\ast }\right)\right]=0$, and the fact that $\underset{i}{max}|x_{ij}|=O_p\left(\sqrt{\log n}\right)$ for sub-Gaussian random variables. For some positive constans C, we have

$$|y_i{x}_{ij}{\ell }^{\prime }\left(y_i{\mathit{x}}_{i\mathcal{A}}^{\mathrm{\top }}{\mathrm{\text{β}}}_{\mathcal{A}}^{\ast }\right)|\le M_1|x_{ij}|(|{\mathit{x}}_{i\mathcal{A}}^{\mathrm{\top }}{\mathrm{\text{β}}}_{\mathcal{A}}^{\ast }|+1)\le M_1|x_{ij}|(\parallel {\mathit{x}}_{i\mathcal{A}}\parallel \parallel {\mathrm{\text{β}}}_{\mathcal{A}}^{\ast }\parallel +1)\le C{q}_n\log n.$$

By Lemma 14.11 of [2], we have

$$Pr\left\{|\frac{1}{n}\sum _{i=1}^n{y}_i{x}_{ij}{\ell }^{\prime }\left(y_i{\mathit{x}}_{i\mathcal{A}}^{\mathrm{\top }}{\mathrm{\text{β}}}_{\mathcal{A}}^{\ast }\right)|>\frac{{\lambda }_{1n}}{2}\right\}\le 2\exp \left\{-\frac{n{\lambda }_{1n}^2}{8C^2{q}_n^2{\log }^{2}n}\right\}.$$

Then

$$\begin{array}{rl}& Pr\left\{\underset{j\in {\mathcal{A}}^c}{max}|\frac{1}{n}\sum _{i=1}^n{y}_i{x}_{ij}{\ell }^{\prime }\left(y_i{\mathit{x}}_{i\mathcal{A}}^{\mathrm{\top }}{\mathrm{\text{β}}}_{\mathcal{A}}^{\ast }\right)|>\frac{{\lambda }_{1n}}{2}\right\}\\ & =Pr\left[\bigcup _{j\in {\mathcal{A}}^c}\left\{|\frac{1}{n}\sum _{i=1}^n{y}_i{x}_{ij}{\ell }^{\prime }\left(y_i{\mathit{x}}_{i\mathcal{A}}^{\mathrm{\top }}{\mathrm{\text{β}}}_{\mathcal{A}}^{\ast }\right)|>\frac{{\lambda }_{1n}}{2}\right\}\right]\le 2p_n\exp \left\{-\frac{n{\lambda }_{1n}^2}{8C^2{q}_n^2{\log }^{2}n}\right\}\\ & =2\exp \left[\log p_n\left\{1-\frac{n{\lambda }_{1n}^2}{8C^2{q}_n^2\log p_n{\log }^{2}n}\right\}\right]\to 0\end{array}$$

as $n\to \mathrm{\infty }$ by the fact that $q_n{n}^{-1/2}\log p_n\log n=o\left({\lambda }_{1n}\right)$.

Lemma A.3

Suppose that Conditions 1–5 hold, $q_n{n}^{-1/2}\log p_n\log n=o\left({\lambda }_{1n}\right)$, ${\lambda }_{2n}{q_n}^{1/2}{n}^{-1/2}=o\left({\lambda }_{1n}\right)$, and ${\lambda }_{1n}=o\left(n^{-(1-c_3)/2}\right)$. For $j=1,2,\dots ,p$, denote

$$s_j\left(\stackrel{\mathbf{ˆ}}{\mathrm{\text{β}}}\right)=\frac{\mathrm{\partial }[L_n\left(\stackrel{\mathbf{ˆ}}{\mathrm{\text{β}}}\right)+{\lambda }_{2n}{\mathrm{\text{β}}}_{\mathcal{A}}^{\mathrm{\top }}{D}_{\mathcal{A}}^{\mathrm{\top }}{D}_{\mathcal{A}}{\mathrm{\text{β}}}_{\mathcal{A}}]}{\mathrm{\partial }{\mathrm{\text{β}}}_j}.$$

For the oracle estimator $\stackrel{\mathbf{ˆ}}{\mathrm{\text{β}}}$ and $s_j\left(\stackrel{\mathbf{ˆ}}{\mathrm{\text{β}}}\right)$, with probability approaching 1, we have

$$\begin{array}{rl}& s_j\left(\stackrel{\mathbf{ˆ}}{\mathrm{\text{β}}}\right)=0,|{\widehat{\mathrm{\text{β}}}}_j|\ge (a+\frac{1}{2}){\lambda }_{1n},j\in \mathcal{A},\\ & |s_j\left(\stackrel{\mathbf{ˆ}}{\mathrm{\text{β}}}\right)|\le {\lambda }_{1n},|{\widehat{\mathrm{\text{β}}}}_j|=0,j\in {\mathcal{A}}^c.\end{array}$$

Proof.

The objective function $L_n\left(\mathrm{\text{β}}\right)+{\lambda }_{2n}{\mathrm{\text{β}}}^{\mathrm{\top }}{D}_{\mathcal{A}}^{\mathrm{\top }}{D}_{\mathcal{A}}\mathrm{\text{β}}$ is convex derivative function. By the convex optimization theorem, we have $s_j\left(\stackrel{\mathbf{ˆ}}{\mathrm{\text{β}}}\right)=0$, $j\in \mathcal{A}$.

Note that $\underset{j\in \mathcal{A}}{min}|{\widehat{\mathrm{\text{β}}}}_j|\ge \underset{j\in \mathcal{A}}{min}|{\mathrm{\text{β}}}_j^{\ast }|-\underset{j\in \mathcal{A}}{max}|{\widehat{\mathrm{\text{β}}}}_j-{\mathrm{\text{β}}}_j^{\ast }|$. Furthermore, we have $\underset{j\in \mathcal{A}}{min}|{\mathrm{\text{β}}}_j^{\ast }|\ge M_6{n}^{-(1-c_3)/2}$ by Condition 5, and $\underset{j\in \mathcal{A}}{max}|{\widehat{\mathrm{\text{β}}}}_j-{\mathrm{\text{β}}}_j^{\ast }|\le \parallel {\stackrel{\mathbf{ˆ}}{\mathrm{\text{β}}}}_{\mathcal{A}}-{\mathrm{\text{β}}}_{\mathcal{A}}^{\ast }\parallel =O_p\left(\sqrt{q_n/n}\right)=O_p\left(n^{-(1-c_1)/2}\right)=o_p\left(n^{-(1-c_3)/2}\right).$ Then according to ${\lambda }_{1n}=o\left(n^{-(1-c_3)/2}\right)$, we have

$$Pr\left\{|{\widehat{\mathrm{\text{β}}}}_j|\ge \left(a+\frac{1}{2}\right){\lambda }_{1n}\right\}\to 1,\mathrm{f}\mathrm{o}\mathrm{r}j\in \mathcal{A}.$$

For $j\in {\mathcal{A}}^c$, we have

$$s_j\left(\stackrel{\mathbf{ˆ}}{\mathrm{\text{β}}}\right)=\frac{1}{n}\sum _{i=1}^n{y}_i{x}_{ij}{\ell }^{\prime }\left(y_i{\mathit{x}}_{i\mathcal{A}}^{\mathrm{\top }}{\stackrel{\mathbf{ˆ}}{\mathrm{\text{β}}}}_{\mathcal{A}}\right)+2{\lambda }_{2n}\sum _{i=1}^{p_n}{\left(D^{\mathrm{\top }}D\right)}_{ij}{\widehat{\mathrm{\text{β}}}}_i$$ (A.3)

We observe that

$$\begin{array}{rl}& Pr\{\underset{j\in {\mathcal{A}}^c}{max}|n^{-1}\sum _{i=1}^n{y}_i{x}_{ij}{\ell }^{\prime }\left(y_i{\mathit{x}}_{i\mathcal{A}}^{\mathrm{\top }}{\stackrel{\mathbf{ˆ}}{\mathrm{\text{β}}}}_{\mathcal{A}}\right)|>{\lambda }_{1n}\}\\ & \le Pr\{\underset{j\in {\mathcal{A}}^c}{max}|n^{-1}\sum _{i=1}^n{y}_i{x}_{ij}{\ell }^{\prime }\left(y_i{\mathit{x}}_{i\mathcal{A}}^{\mathrm{\top }}{\mathrm{\text{β}}}_{\mathcal{A}}^{\ast }\right)|>\frac{{\lambda }_{1n}}{2}\}\\ & +Pr\{\underset{j\in {\mathcal{A}}^c}{max}|n^{-1}\sum _{i=1}^n{y}_i{x}_{ij}\left[{\ell }^{\prime }\right(y_i{\mathit{x}}_{i\mathcal{A}}^{\mathrm{\top }}{\stackrel{\mathbf{ˆ}}{\mathrm{\text{β}}}}_{\mathcal{A}})-{\ell }^{\prime }(y_i{\mathit{x}}_{i\mathcal{A}}^{\mathrm{\top }}{\mathrm{\text{β}}}_{\mathcal{A}}^{\ast }\left)\right]|>\frac{{\lambda }_{1n}}{2}\}.\end{array}$$ (A.4)

By Lemma A.2, the first term of inequality (A.4) is $o_p\left(1\right)$. From Lemma A.1, the second term of inequality (A.4) is bounded by

$$\begin{array}{rl}& Pr\{\underset{j\in {\mathcal{A}}^c}{max}|n^{-1}\sum _{i=1}^n{y}_i{x}_{ij}[{\ell }^{\prime }\left(y_i{\mathit{x}}_{i\mathcal{A}}^{\mathrm{\top }}{\stackrel{\mathbf{ˆ}}{\mathrm{\text{β}}}}_{\mathcal{A}}\right)-{\ell }^{\prime }\left(y_i{\mathit{x}}_{i\mathcal{A}}^{\mathrm{\top }}{\mathrm{\text{β}}}_{\mathcal{A}}^{\ast }\right)]|>\frac{{\lambda }_{1n}}{2}\}\\ & \le Pr\left\{\underset{j\in {\mathcal{A}}^c}{max}\underset{\parallel {\mathrm{\text{β}}}_{\mathcal{A}}-{\mathrm{\text{β}}}_{\mathcal{A}}^{\ast }\parallel \le C\sqrt{q_n/n}}{sup}|n^{-1}\sum _{i=1}^n{y}_i{x}_{ij}[{\ell }^{\prime }\left(y_i{\mathit{x}}_{i\mathcal{A}}^{\mathrm{\top }}{\stackrel{\mathbf{ˆ}}{\mathrm{\text{β}}}}_{\mathcal{A}}\right)-{\ell }^{\prime }\left(y_i{\mathit{x}}_{i\mathcal{A}}^{\mathrm{\top }}{\mathrm{\text{β}}}_{\mathcal{A}}^{\ast }\right)]|>\frac{{\lambda }_{1n}}{2}\right\}\end{array}$$ (A.5)

Together with Conditions 1 and 4, we have

$$\begin{array}{rl}& \underset{j\in {\mathcal{A}}^c}{max}\underset{\parallel {\mathrm{\text{β}}}_{\mathcal{A}}-{\mathrm{\text{β}}}_{\mathcal{A}}^{\ast }\parallel \le C\sqrt{q_n/n}}{sup}|n^{-1}\sum _{i=1}^n{y}_i{x}_{ij}[{\ell }^{\prime }\left(y_i{\mathit{x}}_{i\mathcal{A}}^{\mathrm{\top }}{\mathrm{\text{β}}}_{\mathcal{A}}\right)-{\ell }^{\prime }\left(y_i{\mathit{x}}_{i\mathcal{A}}^{\mathrm{\top }}{\mathrm{\text{β}}}_{\mathcal{A}}^{\ast }\right)]|\\ & \le M_2\underset{\parallel {\mathrm{\text{β}}}_{\mathcal{A}}-{\mathrm{\text{β}}}_{\mathcal{A}}^{\ast }\parallel \le C\sqrt{q_n/n}}{sup}\underset{i,j}{max}|x_{ij}|n^{-1}\sum _{i=1}^n\sqrt{{({\mathrm{\text{β}}}_{\mathcal{A}}-{\mathrm{\text{β}}}_{\mathcal{A}}^{\ast })}^{\mathrm{\top }}{\mathit{x}}_{i\mathcal{A}}{\mathit{x}}_{i\mathcal{A}}^{\mathrm{\top }}({\mathrm{\text{β}}}_{\mathcal{A}}-{\mathrm{\text{β}}}_{\mathcal{A}}^{\ast })}\\ & \le M_2\underset{\parallel {\mathrm{\text{β}}}_{\mathcal{A}}-{\mathrm{\text{β}}}_{\mathcal{A}}^{\ast }\parallel \le C\sqrt{q_n/n}}{sup}\underset{i,j}{max}|x_{ij}|\sqrt{{({\mathrm{\text{β}}}_{\mathcal{A}}-{\mathrm{\text{β}}}_{\mathcal{A}}^{\ast })}^{\mathrm{\top }}\left(n^{-1}{\mathit{x}}_{\mathcal{A}}^{\mathrm{\top }}{\mathit{x}}_{\mathcal{A}}\right)({\mathrm{\text{β}}}_{\mathcal{A}}-{\mathrm{\text{β}}}_{\mathcal{A}}^{\ast })}\\ & \le M_2\underset{\parallel {\mathrm{\text{β}}}_{\mathcal{A}}-{\mathrm{\text{β}}}_{\mathcal{A}}^{\ast }\parallel \le C\sqrt{q_n/n}}{sup}\underset{i,j}{max}|x_{ij}|\cdot \parallel {\mathrm{\text{β}}}_{\mathcal{A}}-{\mathrm{\text{β}}}_{\mathcal{A}}^{\ast }\parallel \cdot [\sqrt{{\lambda }_{max}\left(n^{-1}{\mathit{x}}_{\mathcal{A}}^{\mathrm{\top }}{\mathit{x}}_{\mathcal{A}}\right)}]\\ & =O\{\sqrt{\log \left(p_{n}n\right)}\}\cdot \sqrt{q_n/n}=o\left({\lambda }_{1n}\right),\end{array}$$ (A.6)

as $n\to \mathrm{\infty }$ by the fact that $q_n{n}^{-1/2}\log p_n\log n=o\left({\lambda }_{1n}\right)$.

By (A.4)–(A.6), as $n\to \mathrm{\infty }$, we have

$$Pr\{\underset{j\in {\mathcal{A}}^c}{max}|n^{-1}\sum _{i=1}^n{y}_i{x}_{ij}{\ell }^{\prime }\left(y_i{\mathit{x}}_{i\mathcal{A}}^{\mathrm{\top }}{\stackrel{\mathbf{ˆ}}{\mathrm{\text{β}}}}_{\mathcal{A}}\right)|>{\lambda }_{1n}\}\to 0.$$ (A.7)

Then according to the nature of the matrix $D^{\mathrm{\top }}D$,

$$\begin{array}{rl}2{\lambda }_{2n}|\sum _{i=1}^{p_n}{\left(D^{\mathrm{\top }}D\right)}_{ij}{\widehat{\mathrm{\text{β}}}}_i|& \le 2{\lambda }_{2n}|\sum _{i=1}^{p_n}{\left(D^{\mathrm{\top }}D\right)}_{ij}({\widehat{\mathrm{\text{β}}}}_i-{\mathrm{\text{β}}}_i^{\ast })|+2{\lambda }_{2n}|\sum _{i=1}^{p_n}{\left(D^{\mathrm{\top }}D\right)}_{ij}{\mathrm{\text{β}}}_i^{\ast }|\\ & \le 2\sqrt{70}{\lambda }_{2n}\parallel {\stackrel{\mathbf{ˆ}}{\mathrm{\text{β}}}}_{\mathcal{A}}-{\mathrm{\text{β}}}_{\mathcal{A}}^{\ast }\parallel +2\sqrt{6}{\lambda }_{2n}\parallel D{\mathrm{\text{β}}}^{\ast }\parallel =o_p\left({\lambda }_{1n}\right).\end{array}$$ (A.8)

By (A.3), (A.7) and (A.8), we have $|s_j\left(\stackrel{\mathbf{ˆ}}{\mathrm{\text{β}}}\right)|\le {\lambda }_{1n}$ for $j\in {\mathcal{A}}^c$.

As the oracle estimate $\stackrel{\mathbf{ˆ}}{\mathrm{\text{β}}}$ is defined as

$$\stackrel{\mathbf{ˆ}}{\mathrm{\text{β}}}=\arg \underset{{\mathrm{\text{β}}}_{{\mathcal{A}}^c}=0}{min}\left\{\frac{1}{n}\sum _{i=1}^n\ell \left(y_i{\mathit{x}}_{i\mathcal{A}}^{\mathrm{\top }}{\mathrm{\text{β}}}_{\mathcal{A}}\right)+{\lambda }_{2n}{\mathrm{\text{β}}}_{\mathcal{A}}^{\mathrm{\top }}{D}_{\mathcal{A}}^{\mathrm{\top }}{D}_{\mathcal{A}}{\mathrm{\text{β}}}_{\mathcal{A}}\right\},$$

$|{\widehat{\mathrm{\text{β}}}}_j|=0$ for $j\in {\mathcal{A}}^c$ naturally.

Appendix 3. Proof of Theorem 2.1.

Proof.

Let

$$Q_n\left(\mathrm{\text{β}}\right)=L_n\left(\mathrm{\text{β}}\right)+\sum _{j=1}^p{p}_{{\lambda }_{1n}}\left(|{\mathrm{\text{β}}}_j|\right)+{\lambda }_{2n}{\mathrm{\text{β}}}^{\mathrm{\top }}{D}^{\mathrm{\top }}D\mathrm{\text{β}}\stackrel{\mathrm{\Delta }}{=}g\left(\mathrm{\text{β}}\right)-h\left(\mathrm{\text{β}}\right),$$

where

$$g\left(\mathrm{\text{β}}\right)=L_n\left(\mathrm{\text{β}}\right)+{\lambda }_{1n}\sum _{j=1}^p|{\mathrm{\text{β}}}_j|+{\lambda }_{2n}{\mathrm{\text{β}}}^{\mathrm{\top }}{D}^{\mathrm{\top }}D\mathrm{\text{β}},h\left(\mathrm{\text{β}}\right)={\lambda }_{1n}\sum _{j=1}^p|{\mathrm{\text{β}}}_j|-\sum _{j=1}^p{p}_{{\lambda }_{1n}}\left(|{\mathrm{\text{β}}}_j|\right).$$

By writing as $g\left(\mathrm{\text{β}}\right)-h\left(\mathrm{\text{β}}\right)$, we need to show that $\stackrel{\mathbf{ˆ}}{\mathrm{\text{β}}}$ is a local minimizer of $Q_n\left(\mathrm{\text{β}}\right)$. Based on Lemma A3, the proof is similar to that of Theorem 3.2 in [27]. We omit the proof here.

Funding Statement

This study was supported by the National Natural Science Foundation of China [grant number 11971404], [grant number 71471152], Humanity and Social Science Youth Foundation of Ministry of Education of China [grant number 19YJC910010], [grant number 20YJC910004], the 111 Project (B13028) and Fundamental Research Funds for the Central Universities [grant number 20720181003].

Disclosure statement

No potential conflict of interest was reported by the author(s).