Error Bound of Mode-Based Additive Models

Abstract

Due to their flexibility and interpretability, additive models are powerful tools for high-dimensional mean regression and variable selection. However, the least-squares loss-based mean regression models suffer from sensitivity to non-Gaussian noises, and there is also a need to improve the model’s robustness. This paper considers the estimation and variable selection via modal regression in reproducing kernel Hilbert spaces (RKHSs). Based on the mode-induced metric and two-fold Lasso-type regularizer, we proposed a sparse modal regression algorithm and gave the excess generalization error. The experimental results demonstrated the effectiveness of the proposed model.

1. Introduction

Regression estimation and variable selection are two important tasks for high-dimensional data mining [1]. Sparse additive models [2,3], aiming to deal with the above tasks simultaneously, have been extensively investigated in the mean regression setting. As a class of models between linear and nonparametric regression, these methods inherit the flexibility from nonparametric regression and the interpretability from linear regression. Typical methods include COSSO [4] and SpAM [2] and its variants, such as Group SpAM [3], SAM [5], Group SAM [6], SALSA [7], MAM [8], SSAM [9], and ramp-SAM [10]. From the lens of nonparametric regression, the additive structure on the hypothesis space is crucial to overcome the curse of dimensionality [7,11,12].

Usually, the aforementioned models are limited to the estimation of the conditional mean under the mean-squared error (MSE) criterion. However, for the complex non-Gaussian noises (e.g., the skewed noise, the heavy-tailed noise), it is difficult to extract the intrinsic trends from the mean-based approaches, resulting in degraded performance. Beyond the traditional mean regression, it is interesting to formulate a new regression framework under the (conditional) mode-based criterion. With the help of the recent works in [13,14,15,16,17,18,19], this paper aimed to propose a new robust sparse additive model, rooted in modal regression associated with the RKHS.

As an alternative approach to mean regression, modal regression has been investigated on statistical behavior [14,15,17] and real-world applications [20,21]. Yao [14] proposed a modal linear regression algorithm and characterized its theoretical properties under the global mode assumption. As a natural extension of Lasso [22], Wang et al. [15] considered the regularized modal regression and established its analysis on the generalization bound and variable selection consistency. Feng et al. [17] studied modal regression by a learning theory approach and illustrated its relation with MCC [23,24]. Different from the above global approaches, some local modal regression algorithms were formulated in [16,25] with convergence guarantees. Recent literature [26] gave a general overview of modal regression, and a more comprehensive list of references can be found there.

The proposed robust additive models are formulated under the Tikhonov regularization scheme, which involves three building blocks, including the mode-based metric, the RKHS-based hypothesis space, and two Lasso-type penalties. Since the linear function space, polynomial function space, and Sobolev/Besov space are special cases of the RKHS, the kernel-based function space is more flexible than the traditional spline-based spaces or other dictionary-based hypotheses [2,5,27,28,29]. The mode-induced regression metric is robust to the non-Gaussian noise according to the theoretical and empirical evaluations [14,15,17]. The regularized penalty addresses the sparsity and smoothness of the estimator, which has shown promising performance for mean regression [2,29,30,31]. Therefore, different from mean-based kernel regression and additive models, the mode-based approach enjoys robustness and interpretability simultaneously due to its metric criterion and trade-off penalty. The estimator of our approach can be obtained by integrating the half-quadratic (HQ) optimization [32] and the second-order cone programming (SOCP) [33].

The rest of this article is organized as follows. After introducing the robust additive model in Section 2, we state its generalization error bound in Section 3. Finally, Section 5 ends this paper with a brief conclusion.

2. Methodology

2.1. Modal Regression

In this section, we recall the basic background on modal regression [19,34]. Let $\mathcal{X}$ be a compact subset of ${\mathbb{R}}^p$ associated with the input covariate vector and $\mathcal{Y}\in \mathbb{R}$ be the response variable set. In this paper, we considered the following nonparametric model:

$$Y=f^{*}(X)+ϵ,$$ (1)

where $X={(X_1,\dots ,X_p)}^T\in \mathcal{X}$, $Y\in \mathcal{Y}$, and $ϵ$ is a random noise. For feasibility, we denote by $\rho$ the underlying joint distribution of $(X,Y)$ generated by (1).

Being different from the traditional mean regression under the noise condition $E(ϵ|X=x)=0$ (e.g., Gaussian noise), we just require that the mode of the conditional distribution of $ϵ$ equal zero at each $x\in \mathcal{X}$. That is:

$$\forall x\in \mathcal{X},mode(ϵ|X=x)=arg\underset{t\in \mathbb{R}}{max}{P}_{ϵ|X}(t|X=x)=0,$$ (2)

where $P_{ϵ|X}$ is the conditional density of $ϵ$ given X. Notice that the zero condition is not specified to the homogeneity or symmetry distribution of noise $ϵ$, and some non-Gaussian noises (e.g., the skewed noise, the heavy-tailed noise) are not excluded.

From (1), we further deduce that:

$$\begin{array}{ccc}\hfill f^{*}(u)& :=& \sum _{j=1}^p{f}_j^{*}(u_j)=mode(Y|X=u)=arg\underset{t}{max}{P}_{Y|X}(t|X=u),\hfill \end{array}$$

where $u={(u_1,\dots ,u_p)}^T\in \mathcal{X}$ and $P_{Y|X}$ denotes the density of Y conditional on X. Then, the purpose of modal regression is to find the target function $f^{*}$ according to the empirical data $\mathbf{z}={\{z_i\}}_{i=1}^n={\{(x_i,y_i)\}}_{i=1}^n$ drawn independently from $\rho$.

For modal regression, the performance of a predictor $f:\mathcal{X}\to \mathbb{R}$ is measured by the mode-based metric:

$$\mathcal{R}(f)={\int }_{\mathcal{X}}{P}_{Y|X}(f(x)|X=x)d{\rho }_{\mathcal{X}}(x),$$ (3)

where ${\rho }_{\mathcal{X}}$ is the marginal distribution of $\rho$ with respect to input space $\mathcal{X}$.

Although the target function $f^{*}$ is the maximizer of $\mathcal{R}(f)$ over all measurable functions, it cannot be estimated directly via maximizing (3) due to the unknown $P_{Y|X}$ and ${\rho }_{\mathcal{X}}$. Fortunately, some indirect density-estimation-based strategies were proposed in [14,15,17]. As shown in Theorem 5 of [17], $\mathcal{R}(f)$ equals the density function of random variable $E_f=Y-f(X)$ at zero, e.g.,

$$\mathcal{R}(f)=P_{E_f}(0).$$

Therefore, we can find an approximation of $f^{*}$ by maximizing the empirical version of $P_{E_f}(0)$ with the help of kernel density estimation (KDE).

Let $K_{\sigma }:\mathbb{R}\times \mathbb{R}\to {\mathbb{R}}_{+}$ be a kernel with bandwidth $\sigma$, and its representing function $\varphi :\mathbb{R}\to [0,\infty )$ satisfies $\varphi (\frac{u-u^{\prime }}{\sigma })=K_{\sigma }(u,u^{\prime }),\forall u,u^{\prime }\in \mathbb{R}$. Typical kernels used in KDE include the Gaussian kernel, the Epanechnikov kernel, the logistic kernel, and the sigmoid kernel. The KDE-based estimator of $P_{E_f}(0)$ is defined as:

$$\begin{array}{c}\hfill {\widehat{P}}_{E_f}(0)=\frac{1}{n\sigma }\sum _{i=1}^n{K}_{\sigma }(y_i-f(x_i),0)=\frac{1}{n\sigma }\sum _{i=1}^n\varphi (\frac{y_i-f(x_i)}{\sigma }):={\widehat{\mathcal{R}}}^{\sigma }(f).\end{array}$$

Learning models for modal regression are usually formulated by Tikhonov regularization schemes associated with the empirical metric ${\widehat{\mathcal{R}}}^{\sigma }(f)$; see, e.g., [15,35].

Naturally, the data-free modal regression metric, $w.r.t.$${\widehat{\mathcal{R}}}^{\sigma }(f)$, can be defined as:

$${\mathcal{R}}^{\sigma }(f)=\frac{1}{\sigma }{\int }_{\mathcal{X}\times \mathcal{Y}}\varphi (\frac{y-f(x)}{\sigma })d\rho (x,y).$$

In theory, the learning performance of estimator $f:\mathcal{X}\to \mathbb{R}$ can be evaluated in terms of $\mathcal{R}(f)-\mathcal{R}(f^{*})$, which can be further bounded via ${\mathcal{R}}^{\sigma }(f)-{\mathcal{R}}^{\sigma }(f^{*})$ (see Theorem 10 in [17]).

Remark 1.

As illustrated in [17], when taking $K_{\sigma }$ as a Gaussian kernel, the modal regression for maximizing ${\mathcal{R}}^{\sigma }(f)$ is consistent with learning under the maximum correntropy criterion (MCC). By employing different kernels, we can provide rich evaluated metrics for better robust estimation.

2.2. Mode-Based Sparse Additive Models

The additive model is formulated as follows,

$$Y=\sum _{j=1}^p{f}_j^{*}(X_j)+ϵ,$$ (4)

where $X_j\in X$, $(j=1,2,···,p)$, $Y\in \mathcal{Y}$, and $f_j^{*}$ are unknown component functions. By employing nonlinear hypothesis function spaces with an additive structure, the additive model provides better flexibility for regression estimation and variable selection [19]. In [28], the theoretical properties of the sparse additive model with the quantile loss function were discussed. We introduce some basic notation and assumptions in a similar way.

Suppose that $E{f}_j^{*}(X_j)=0$ and $\parallel f_j^{*}{\parallel }_{K_j}\le 1$ for each $f_j^{*}$ in (4) with $j\in \mathcal{S}$. Here, $f_j^{*}:{\mathcal{X}}_j\to \mathbb{R}$ is an unknown univariate function in a reproducing kernel Hilbert space (RKHS) ${\mathcal{H}}_j:={\mathcal{H}}_{K_j}$ associated with kernel $K_j$ and norm ${\parallel ·\parallel }_{K_j}$ [30,31], and $\mathcal{S}\subseteq \{1,\dots ,p\}$ is an intrinsic subset with cardinality $|\mathcal{S}|<p$. This means each observation $(x_j,y_j)$ is generated according to:

$$y_i=\sum _{j\in \mathcal{S}}{f}_j^{*}(x_{ij})+{ϵ}_i,i=1,\dots ,n,$$

where $x_i={(x_{i1},\dots ,x_{ip})}^T\in {\mathbb{R}}^p$, $f_j^{*}\in {\mathcal{H}}_j$ and $ϵ$ satisfies the condition (2).

For any given $j\in \{1,\dots ,p\}$, denote ${\mathcal{B}}_r({\mathcal{H}}_j)=\{g\in {\mathcal{H}}_j{:\parallel g\parallel }_{K_j}\le r\}$. The hypothesis space considered here is defined by:

$$\mathcal{F}=\{f=\sum _{j=1}^p{f}_j:f_j\in {\mathcal{B}}_r({\mathcal{H}}_j),i=1,\dots ,p\},$$ (5)

which is a subset of the RKHS $\mathcal{H}=\{f={\sum }_{j=1}^p{f}_j:f_j\in {\mathcal{H}}_j\}$ with the norm:

$${\parallel f\parallel }_K^2=inf\{\sum _{j=1}^p\parallel f_j{\parallel }_{K_j}^2:f=\sum _{j=1}^p{f}_j\}.$$

For each ${\mathcal{X}}_j$ and the corresponding marginal distribution ${\rho }_{{\mathcal{X}}_j}$, we denote $\parallel f_j{\parallel }_2^2:={\int }_{{\mathcal{X}}_j}{|f_j(u)|}^{2}d{\rho }_{{\mathcal{X}}_j}(u)$. Given inputs ${\{x_i\}}_{i=1}^n$, define the empirical norm of each $f_j$ as:

$$\parallel f_j{\parallel }_n^2:=\frac{1}{n}\sum _{i=1}^n{f}_j^2(x_{ij}),\forall f_j\in {\mathcal{H}}_j,j\in \{1,\dots ,p\}.$$

With the help of the mode-based metric (3) and the hypothesis space (5), we formulated the mode-based sparse additive model as:

$$\widehat{f}=arg\underset{f\in \mathcal{F}}{max}\{{\widehat{\mathcal{R}}}^{\sigma }(f)-{\lambda }_1\sum _{j=1}^p\parallel f_j{\parallel }_n-{\lambda }_2\sum _{j=1}^p\parallel f_j{\parallel }_{K_j}\},$$ (6)

where $({\lambda }_1,{\lambda }_2)$ is a pair of positive regularization. The first regularization term is sparsity-promoting [11,36], and the second one guarantees smoothness in the solution.

By the representer theorem of kernel methods (e.g., [37]), the solution of (6) admits the following form:

$$\widehat{f}(u)=\sum _{i=1}^n\sum _{j=1}^p{\widehat{\alpha }}_{ij}K(u_j,x_{ij}),u={(u_1,\dots ,u_p)}^T$$

with a collection of coefficients $\{{\widehat{\alpha }}_j={({\alpha }_{1j},\dots ,{\alpha }_{nj})}^T\in {\mathbb{R}}^n:j=1,\dots ,p\}$.

The optimal coefficients with respect to (6) are the solution to the following non-convex optimization:

$$\begin{array}{c}\hfill \underset{{\alpha }_j\in {\mathbb{R}}^n,{\alpha }_j^T{K}_j{\alpha }_j\le 1}{max}\{\frac{1}{n}\sum _{i=1}^n\varphi (\frac{y_i-{\sum }_{j=1}^p{K}_{ji}^T{\alpha }_j}{\sigma })-\frac{{\lambda }_1}{\sqrt{n}}\sum _{j=1}^p\parallel K_j{\alpha }_j{\parallel }_2-{\lambda }_2\sum _{j=1}^p\sqrt{{\alpha }_j^T{K}_j{\alpha }_j}\}\end{array}$$

where $K_{ji}={(K_j(x_{1j},x_{ij}),\dots ,K_j(x_{nj},x_{ij}))}^T\in {\mathbb{R}}^n$ and $K_j={(K_j(x_{ij},x_{lj}))}_{i,l}^n=(K_{j1},\dots ,K_{jn})\in {\mathbb{R}}^{n\times n}$.

Remark 2.

There are various combinations of sparsity and smoothness regularization for additive models [2,3,29,30,31]. The regularization in this paper adopting a two-fold group Lasso scheme, which was employed in [28], but in quantile regression settings, is also different from the coefficient-based regularized modal regression in [19].

Remark 3.

From the lens of computation, the proposed algorithm (6) can be transformed into a regularized least-squares regression problem by HQ optimization [32]. Then, the transformed problem can be tackled with the SOCP [33] easily.

3. Error Analysis

This section states the upper bounds of the excess quantity $\mathcal{R}(f^{*})-\mathcal{R}(\widehat{f})$. For the ease of presentation, we only considered the special setting where ${\mathcal{H}}_j\equiv {\mathcal{H}}_{j^{\prime }},\forall j,j^{\prime }\in \{1,\dots ,p\}$, and we denote ${\oplus }_{j=1}^p{\mathcal{H}}_j$ as ${\mathcal{H}}_K$ with $supK(x,x)\le 1$.

Recall that the Mercer kernel $K:\mathcal{X}\times \mathcal{X}\to \mathbb{R}$ admits the following spectral expansion [38]:

$$K(x,x^{\prime })=\sum _{\ell \ge 1}{b}_{\ell }{\psi }_{\ell }(x){\psi }_{\ell }(x^{\prime }),x,x^{\prime }\in \mathcal{X},$$

where ${\{(b_{\ell },{\psi }_{\ell })\}}_{\ell \ge 1}$ are the pairs of eigenvalue-eigenfunctions of integral operator $\mathcal{T}f:\int K(·,x)f(x)d{\rho }_{\mathcal{X}}(x)$ with $b_1\ge b_2\ge \dots \ge 0$.

To evaluate the complexity of ${\mathcal{H}}_K$ in terms of the decay rate of eigenvalues ${\{b_{\ell }\}}_{\ell \ge 1}$ [27,28], we refer to Assumption 1 in [28] as the basis of our analysis.

Assumption 1.

There exist $s\in (0,1)$ and constant $c_1>0$ such that $b_{\ell }\le c_1{\ell }^{-\frac{1}{s}}$, $\forall \ell \ge 1$.

As illustrated in [27,28], the requirement $s<1$ is a weak condition since ${\sum }_{\ell }{b}_{\ell }=EK(x,x)\le 1$. In particular, it holds $b_{\ell }\asymp {\ell }^{-2h}$ for the Sobolev space ${\mathcal{H}}_K=W_2^h(h>\frac{1}{2})$ with the Lebesgue measure on $[0,1]$.

To describe the hypothesis in RKHS, we refer to Assumption 2 in [28].

Assumption 2.

For some $s\in (0,1)$ given in Assumption 1, there exists a positive constant $c_2$ such that ${\parallel f\parallel }_{\infty }\le c_2{\parallel f\parallel }_2^{1-s}{\parallel f\parallel }_K^s$, $\forall f\in {\mathcal{H}}_K$.

Remark 4.

To understand the statistical performance of the proposed estimator without any “correlatedness” conditions on covariates, Rademacher complexity [39] was used to measure functional complexity in [28]. We drew on the experience of [28].

In general, Assumption 2 is stronger than Assumption 1 and is satisfied when the RKHS is continuously embeddable in a Sobolev space. For the uniformly bounded ${\{{\psi }_{\ell }\}}_{\ell \ge 1}$, this sub-norm condition is consistent with Assumption 1.

For any given independent input variables ${\{x_i\}}_{i=1}^n\subset \mathcal{X}$, define the Rademacher complexity:

$${\mathcal{R}}_n(f):=\frac{1}{n}\sum _{i=1}^n{\sigma }_{i}f(x_i),\forall f\in {\mathcal{H}}_K,$$

where ${\{{\sigma }_i\}}_{i=1}^n$ is an $i.i.d.$ sequence of Rademacher variables that take $\{\pm 1\}$ with probability $1/2$. As shown in [40], it holds:

$$E{\mathcal{R}}_n\{f\in {\mathcal{H}}_K{\{\parallel f\parallel }_K={1,\parallel f\parallel }_2\le t\}\}\asymp \frac{1}{\sqrt{n}}{[\sum _{\ell }^{\infty }min\{t^2,b_{\ell }\}]}^{\frac{1}{2}}.$$

Moreover, from Assumption 1, define:

$$\begin{array}{ccc}\hfill {\gamma }_n& :=& inf\{\gamma \ge \sqrt{\frac{Alog\tilde{p}}{n}},E[\underset{{\parallel f\parallel }_K=1,{\parallel f\parallel }_2\le t}{sup}|{\mathcal{R}}_n(f)|]\le \gamma t+{\gamma }^2,\forall t\in (0,1)\}\hfill \\ & \asymp & max\{\sqrt{\frac{Alog\tilde{p}}{n}},{(\frac{1}{n})}^{\frac{1}{2(1+\alpha )}}\}.\hfill \end{array}$$

The main idea of our error analysis is to first state a theory result for a defined event and then investigate the behavior of $\widehat{f}$ in (6) conditional on that event.

Define $\eta (t):=max\{1,\sqrt{t},t/\sqrt{n}\}$ for any $t>0$ and ${\xi }_n:={\xi }_n(\lambda )=max\{{\lambda }^{-\frac{\alpha }{2}}{n}^{-\frac{1}{2}},{\lambda }^{-\frac{1}{2}}{n}^{-\frac{1}{1+\alpha }},\sqrt{\frac{logp}{n}}\}$, and consider the event:

$$\begin{array}{ccc}\hfill \theta (t)& =& \{|\frac{1}{n}\sum _{i=1}^n{ϵ}_{i}f(x_i)|\le c_{\alpha }\eta (t){\xi }_n{(\parallel f\parallel }_2+{\lambda }^{\frac{1}{2}}{\parallel f\parallel }_K),\forall f\in {\mathcal{H}}_K\},\hfill \end{array}$$

where ${\{{ϵ}_i\}}_{i=1}^n$ are zero-mean $i.i.d.$ random variables with $|{ϵ}_i|\le L$ and $c_{\alpha }$ is a constant depending on $\alpha$ and L.

Remark 5.

To analyze the behavior of the regularized estimator conditioned on the event, several basic facts of the empirical processes were introduced in [28]. Our work can be boiled down to this framework. We introduced the relevant lemmas in [28] as a stepping stone.

Lemma 1.

Let Assumptions 1 and 2 be true. If $\frac{logp}{\sqrt{n}}\le 1$, it holds:

$$P(\theta (t))\ge 1-exp\{-t\},\forall \lambda >0,t\ge 1.$$

The following lemma (see also Theorem 4 in [41]) demonstrates the relationship between the empirical norm ${\parallel ·\parallel }_n$ and ${\parallel ·\parallel }_2$ for functions in ${\mathcal{H}}_K$.

Lemma 2.

For $A\ge 1$ and any given $\tilde{p}\ge p$ with $log\tilde{p}\ge 2loglogn$, there exists a constant c such that:

$${\parallel f\parallel }_2\le {c(\parallel f\parallel }_n+{\gamma }_n{\parallel f\parallel }_K)$$

and:

$${\parallel f\parallel }_n\le {c(\parallel f\parallel }_2+{\gamma }_n{\parallel f\parallel }_K)$$

with confidence at least $1-{\tilde{p}}^{-A}$, where ${\gamma }_n\asymp max(\sqrt{\frac{Alog\tilde{p}}{n}},{(\frac{1}{n})}^{\frac{1}{2(1+\alpha )}})$.

Lemma 3.

Let ${\{z_i\}}_{i=1}^n\subset \mathcal{Z}$ be independent random variables, and let Γ be a class of real-valued functions on $\mathcal{Z}$ satisfying:

$$\parallel \gamma \parallel \le {\eta }_n,\forall \gamma \in \Gamma ,and\frac{1}{n}\sum _{i=1}^{n}var(\gamma (z_i))\le {\iota }_n^2,$$

for some positive constants ${\eta }_n$ and ${\iota }_n$. Define $\zeta :={sup}_{\gamma \in \Gamma }|\frac{1}{n}{\sum }_{i=1}^n\gamma (z_i)-E\gamma (z)|$. Then,

$$\begin{array}{c}\hfill P\{\zeta \ge E\zeta +t\sqrt{2({\iota }_n^2+2{\eta }_{n}Ez)}+\frac{2{\eta }_n{t}^2}{3}\le exp\{-n{t}^2\}\end{array}$$

For any given ${\Delta }_{-}$ and ${\Delta }_{+}$, define:

$$\begin{array}{ccc}\hfill \mathcal{F}({\Delta }_{-},{\Delta }_{+})& =& \{f=\sum _{j=1}^p{f}_j\in {\mathcal{H}}_K:{\gamma }_n\sum _{j=1}^p\parallel f_j-f_j^{*}{\parallel }_2\le {\Delta }_{-},{\gamma }_n^2\sum _{j=1}^p\parallel f_j-f_j^{*}{\parallel }_K\le {\Delta }_{+}\},\hfill \end{array}$$

Lemma 4.

Let Assumptions 1 and 2 be true for each ${\mathcal{H}}_j$. For any given $A\ge 2$, with confidence at least $1-{\tilde{p}}^{-A}$, it holds:

$$\begin{array}{cc}& {\mathcal{R}}^{\sigma }(f^{*})-{\mathcal{R}}^{\sigma }(f)-({\widehat{\mathcal{R}}}^{\sigma }(f^{*})-{\widehat{\mathcal{R}}}^{\sigma }(f))\le c_{*}\eta (t_0)({\Delta }_{-}+{\Delta }_{+})+exp\{-\tilde{p}\},\end{array}$$

for any $f\in \mathcal{F}({\Delta }_{-},{\Delta }_{+})$ with $max\{{\Delta }_{-},{\Delta }_{+}\}\le e^{\tilde{p}}$, where $t_0=2log(\frac{2\sqrt{3}}{log2})+Alog\tilde{p}+2log\tilde{p}$, $\lambda =n^{-\frac{1}{1+\alpha }}$, and $c_{*}$ is a positive constant.

Proof. 

Denote $\Gamma =\{\gamma (z):\gamma (z)=\frac{1}{\sigma }\varphi (\frac{y-f^{*}(x)}{\sigma })-\frac{1}{\sigma }\varphi (\frac{y-f(x)}{\sigma }),f\in \mathcal{F}({\Delta }_{-},{\Delta }_{+})\}$. It is easy to verify that:

$$E\gamma (z)-\frac{1}{n}\sum _{i=1}^n\gamma (z_i)=\mathcal{R}(f^{*})-\mathcal{R}(f)-(\widehat{\mathcal{R}}(f^{*})-\widehat{\mathcal{R}}(f)),\gamma \in \Gamma .$$

Let $\zeta :={sup}_{\gamma \in \Gamma }|\frac{1}{n}{\sum }_{i=1}^n\gamma (z_i)-E\gamma (z)|$. From Lemma 3, we have:

$$\zeta \le E\zeta +\sqrt{\frac{2t({\iota }_n^2+2{\eta }_{n}E\zeta )}{n}}+\frac{2{\eta }_{n}t}{3n},$$ (7)

with probability at least $1-exp\{-t\}$, where constants ${sup}_{\gamma \in \Gamma }{\parallel \gamma \parallel }_{\infty }={\eta }_n$ and ${sup}_{\gamma \in \Gamma }\sqrt{\frac{1}{n}{\sum }_{i=1}^{n}var(\gamma (z_i))}={\iota }_n$. Observing that:

$$\begin{array}{c}\hfill \sqrt{\frac{2t({\iota }_n^2+2{\eta }_{n}E\zeta )}{n}}\le \sqrt{\frac{2t{\iota }_n^2}{n}}+2\sqrt{\frac{{\eta }_{n}E\zeta }{n}}\le \sqrt{\frac{2t}{n}}{\iota }_n+E\zeta +\frac{{\eta }_n}{n},\end{array}$$ (8)

we can take:

$$\begin{array}{c}\hfill {\iota }_n^2\le 2E{(\gamma (z))}^2=2E{(\frac{1}{\sigma }\varphi (\frac{y-f^{*}(x)}{\sigma })-\frac{1}{\sigma }\varphi (\frac{y-f(x)}{\sigma }))}^2\le \frac{2\parallel {\varphi }^{\prime }{\parallel }_{\infty }^2}{{\sigma }^4}{\parallel f-f^{*}\parallel }_2^2\le \frac{2\parallel {\varphi }^{\prime }{\parallel }_{\infty }^2}{{\sigma }^4}\frac{{\Delta }_{-}^2}{{\gamma }^2},\end{array}$$ (9)

and:

$$\begin{array}{ccc}\hfill {\eta }_n& =& \underset{\gamma \in \Gamma }{sup}{\parallel \gamma \parallel }_{\infty }\le \frac{\parallel {\varphi }^{\prime }{\parallel }_{\infty }}{{\sigma }^2}\parallel f^{*}{-f\parallel }_{\infty }\le \frac{\parallel {\varphi }^{\prime }{\parallel }_{\infty }}{{\sigma }^2}{\parallel f^{*}-f\parallel }_K\le \frac{\parallel {\varphi }^{\prime }{\parallel }_{\infty }}{{\sigma }^2}\frac{{\Delta }_{+}}{{\gamma }_n^2}.\hfill \end{array}$$ (10)

Combining (7)–(10), we obtain with confidence at least $1-exp\{-t\}$

$$\zeta \le 2E\zeta +\frac{2\parallel {\varphi }^{\prime }{\parallel }_{\infty }}{{\gamma }_n{\sigma }^2}\sqrt{\frac{t}{n}}+\frac{\kappa \parallel {\varphi }^{\prime }{\parallel }_{\infty }{\Delta }_{+}}{{\sigma }^2{\gamma }_n^2}\frac{1+t}{n}.$$

By a symmetrization technique in [42], we have:

$$E\zeta \le 2E{\mathcal{R}}_n(\Gamma )\le \frac{2\parallel {\varphi }^{\prime }{\parallel }_{\infty }}{{\sigma }^2}E{\mathcal{R}}_n(\mathcal{F}-f^{*}).$$

Applying Lemma 3 for ${\mathcal{R}}_n(\mathcal{F}-f^{*})$, we obtain that:

$$E\left[{\mathcal{R}}_n(\mathcal{F}-f^{*})\right]\le {\mathcal{R}}_n(\mathcal{F}-f^{*})+4\frac{{\Delta }_{-}}{{\gamma }_n}\sqrt{\frac{2t}{n}}+\frac{{\Delta }_{+}}{{\gamma }_n^2}\frac{1+t}{n},$$

with probability at least $1-2exp\{-t\}$. Moreover, with probability at least $1-2exp\{-t\}$, it holds:

$$\begin{array}{ccc}\hfill \zeta & \le & \frac{8\parallel {\varphi }^{\prime }{\parallel }_{\infty }}{{\sigma }^2}{\mathcal{R}}_n(\mathcal{F}-f^{*})+\frac{6\parallel {\varphi }^{\prime }{\parallel }_{\infty }{\Delta }_{-}}{{\gamma }_n{\sigma }^2}\sqrt{\frac{t}{n}}+\frac{5\parallel {\varphi }^{\prime }{\parallel }_{\infty }{\Delta }_{+}}{{\gamma }_n^2{\sigma }^2}\frac{1+t}{n}\hfill \\ & \le & \frac{8\parallel {\varphi }^{\prime }{\parallel }_{\infty }}{{\sigma }^2}\sum _{j=1}^p{\mathcal{R}}_n({\mathcal{H}}_j-f_j^{*})+\frac{6\parallel {\varphi }^{\prime }{\parallel }_{\infty }{\Delta }_{-}}{{\gamma }_n{\sigma }^2}\sqrt{\frac{t}{n}}+\frac{5\parallel {\varphi }^{\prime }{\parallel }_{\infty }{\Delta }_{+}}{{\gamma }_n^2{\sigma }^2}\frac{1+t}{n}.\hfill \end{array}$$

For the event $\theta (t)$, Lemma 1 demonstrates that:

$$|{\mathcal{R}}_n(f)|\le c_{\alpha }\eta (t){\xi }_n{(\parallel f\parallel }_2+{\lambda }^{\frac{1}{2}}{\parallel f\parallel }_K),\forall f\in {\mathcal{H}}_K,\forall \lambda >0,$$

with confidence $1-exp\{-t\}$. Then,

$$\begin{array}{ccc}\hfill \zeta & \le & \frac{8\parallel {\varphi }^{\prime }{\parallel }_{\infty }{c}_{\alpha }\eta (t){\xi }_n}{{\sigma }^2}\underset{f\in \mathcal{F}}{sup}\{\sum _{j=1}^p\parallel f-f_j^{*}{\parallel }_2+{\lambda }^{\frac{1}{2}}\sum _{j=1}^p\parallel f_j-f_j^{*}{\parallel }_K\}+\frac{6\parallel {\varphi }^{\prime }{\parallel }_{\infty }{\Delta }_{-}}{{\gamma }_n{\sigma }^2}\sqrt{\frac{t}{n}}+\frac{5\parallel {\varphi }^{\prime }{\parallel }_{\infty }{\Delta }_{+}}{{\gamma }_n^2{\sigma }^2}\frac{1+t}{n}.\hfill \end{array}$$

Taking $\lambda =n^{-\frac{1}{1+\alpha }}$, we can verify that $c{\gamma }_n\ge {\xi }_n$ and ${\xi }_n{\lambda }^{\frac{1}{2}}\ge c{\gamma }_n^2$. Then,

$$\begin{array}{ccc}\hfill \zeta & \le & \frac{8c_{\alpha }\eta (t){\parallel {\varphi }^{\prime }\parallel }_{\infty }}{{\sigma }^2}({\Delta }_{+}+{\Delta }_{-})+\frac{6{\Delta }_{-}{\parallel {\varphi }^{\prime }\parallel }_{\infty }}{{\sigma }^2}\sqrt{\frac{t}{Alog\tilde{p}}}+\frac{5{\Delta }_{+}{\parallel {\varphi }^{\prime }\parallel }_{\infty }t}{{\sigma }^{2}Alog\tilde{p}},\hfill \end{array}$$

for some event $\theta ({\Delta }_{-},{\Delta }_{+})$.

For $t=2log(2\sqrt{3}/log2)+Alog\tilde{p}+2log\tilde{p}$, we deduce that $e^{-\tilde{p}}\le {\Delta }_{-}\le e^{\tilde{p}}$ and $e^{-\tilde{p}}\le {\Delta }_{+}\le e^{\tilde{p}}$ considering ${(2\tilde{p}+1)}^2$ different discrete pairs ${\Delta }_{-}^j={\Delta }_{+}^j:=2^{-j},j=-\tilde{p},\dots ,\tilde{p}$, and we deduce that:

$$\begin{array}{ccc}\hfill P(\bigcap _{k,j}\theta ({\Delta }_{-}^j,{\Delta }_{+}^j))& \le & 1-3({\frac{2}{log2}}^2{\tilde{p}}^{2}exp\{-2log(\frac{2\sqrt{3}}{log2}-Alog\tilde{p}-2log\tilde{p}\}\le 1-{\tilde{p}}^{-A}.\hfill \end{array}$$

When ${\Delta }_{-}\le e^{-\tilde{p}}$ or ${\Delta }_{+}\le e^{-\tilde{p}}$, it is trivial to obtain the desired result. □

The proof of Lemma 4 is derived from the proof of Proposition 1 in [28] for the quantile regression. We state our main result on the error bound.

Theorem 1.

Let the regularization parameters of $\widehat{f}$ defined in (6) be ${\lambda }_1=\sqrt{\xi }{\gamma }_n$ and ${\lambda }_2=\xi {\gamma }_n^2$, where $\xi =max\{2c\eta (t_0),4\}$ with $\eta (t)=max\{1,\sqrt{t},t/\sqrt{n}\}$, $t_0=2log(2\sqrt{3}/log2)+Alog\tilde{p}+2log\tilde{p}$, and ${\gamma }_n\asymp max(\sqrt{\frac{Alog\tilde{p}}{n}},{(\frac{1}{n})}^{\frac{1}{2(1+\alpha )}})$. Under the conditions of Assumptions 1 and 2, for any $\tilde{p}\ge p$ such that $logp\le \sqrt{n}$ and $log\tilde{p}\ge 2loglogn$, then for some constant $A\ge 2$, such that with probability at least $1-2{\tilde{d}}^{-A}$:

$$\begin{array}{ccc}\hfill \mathcal{R}(f^{*})-\mathcal{R}(\widehat{f})& \le & cs\parallel {\varphi }^{\prime }{\parallel }_{\infty }\eta (t_0){(\eta (t_0))}^{\frac{1}{4}}\sqrt{{\gamma }_n}\le c{(\eta (t_0))}^{\frac{5}{4}}max\{{(\frac{Alog\tilde{p}}{c})}^{\frac{1}{4}},{(\frac{1}{n})}^{\frac{1}{4(1+\alpha )}}\}\hfill \\ \hfill & \le & cmax{\{\sqrt{Alog\tilde{p}},\frac{Alog\tilde{p}}{\sqrt{n}}\}}^{\frac{5}{4}}·max\{{(\frac{Alog\tilde{p}}{n})}^{\frac{1}{4}},{(\frac{1}{n})}^{\frac{1}{4+4\alpha }}\}\hfill \\ & \le & cmax\{\frac{{(Alog\tilde{p})}^{\frac{7}{8}}}{n^{\frac{1}{4}}},\frac{{(Alog\tilde{p})}^{\frac{1}{2}}}{n^{\frac{1}{4+4\alpha }}},\frac{{(Alog\tilde{p})}^{\frac{3}{2}}}{n^{\frac{3}{4}}},\frac{{(Alog\tilde{p})}^{\frac{5}{4}}}{n^{\frac{3+2\alpha }{4+4\alpha }}}\}.\hfill \end{array}$$

Proof. 

By the definition of $\widehat{f}$ in (6), we know that:

$$\begin{array}{cc}\hfill & {\widehat{\mathcal{R}}}^{\sigma }(\widehat{f})-{\lambda }_1\sum _{j=1}^p\parallel {\widehat{f}}_j{\parallel }_n-{\lambda }_2\sum _{j=1}^p\parallel {\widehat{f}}_j{\parallel }_{K_j}\ge {\widehat{\mathcal{R}}}^{\sigma }(f^{*})-{\lambda }_1\sum _{j=1}^p\parallel f_j^{*}{\parallel }_n-{\lambda }_2\sum _{j=1}^p{\parallel f_j^{*}\parallel }_{K_j}.\end{array}$$

This implies that:

$$\begin{array}{ccc}\hfill & & {\widehat{\mathcal{R}}}^{\sigma }(\widehat{f})-{\mathcal{R}}^{\sigma }(f^{*})-{\lambda }_1\sum _{j=1}^p\parallel {\widehat{f}}_j{\parallel }_n-{\lambda }_2\sum _{j=1}^p{\parallel {\widehat{f}}_j\parallel }_{K_j}\hfill \\ & \ge & [{\mathcal{R}}^{\sigma }(\widehat{f})-{\mathcal{R}}^{\sigma }(f^{*})]-[{\widehat{\mathcal{R}}}^{\sigma }(\widehat{f})-{\widehat{\mathcal{R}}}^{\sigma }(f^{*})]-{\lambda }_1\sum _{j=1}^p\parallel f_j^{*}{\parallel }_n-{\lambda }_2\sum _{j=1}^p{\parallel f_j^{*}\parallel }_{K_j}.\hfill \end{array}$$

Moreover,

$$\begin{array}{ccc}\hfill & & {\mathcal{R}}^{\sigma }(f^{*})-{\mathcal{R}}^{\sigma }(\widehat{f})\le {\mathcal{R}}^{\sigma }(f^{*})-{\mathcal{R}}^{\sigma }(\widehat{f})+{\lambda }_1\sum _{j\notin \mathcal{S}}\parallel {\widehat{f}}_j{\parallel }_n+{\lambda }_2\sum _{j\notin \mathcal{S}}{\parallel {\widehat{f}}_j\parallel }_K\hfill \\ \hfill & \le & [{\mathcal{R}}^{\sigma }(f^{*})-{\mathcal{R}}^{\sigma }(\widehat{f})]-[{\widehat{\mathcal{R}}}^{\sigma }(f^{*})-{\widehat{\mathcal{R}}}^{\sigma }(\widehat{f})]+{\lambda }_1\sum _{j\in \mathcal{S}}(\parallel f_j^{*}{\parallel }_n-\parallel {\widehat{f}}_j{\parallel }_n)+{\lambda }_2\sum _{j\in \mathcal{S}}(\parallel f_j^{*}{\parallel }_K-\parallel {\widehat{f}}_j{\parallel }_K)\hfill \\ & \le & [{\mathcal{R}}^{\sigma }(f^{*})-{\mathcal{R}}^{\sigma }(\widehat{f})]-[{\widehat{\mathcal{R}}}^{\sigma }(f^{*})-{\widehat{\mathcal{R}}}^{\sigma }(\widehat{f})]+{\lambda }_1\sum _{j\in \mathcal{S}}\parallel {\widehat{f}}_j-f_j^{*}{\parallel }_n+{\lambda }_2\sum _{j\in \mathcal{S}}{\parallel {\widehat{f}}_j-f_j^{*}\parallel }_K.\hfill \end{array}$$ (11)

Taking ${\lambda }_1=\sqrt{\xi }{\gamma }_n$, ${\lambda }_2=\xi {\gamma }_n^2$ with ${\gamma }_n=max\{\sqrt{\frac{Alog\tilde{p}}{n}},{(\frac{1}{n})}^{\frac{1}{2+2\alpha }}\}$, $\alpha \in (0,1)$, we deduce that:

$$\begin{array}{c}\hfill {\gamma }_n\sum _{j=1}^p{\parallel {\widehat{f}}_j-f_j^{*}\parallel }_2\le 2p{(\frac{1}{n})}^{\frac{1}{2+2\alpha }}\le 2\tilde{p}(\frac{1}{4})\le e^{\tilde{p}},\forall n\ge 1,\tilde{p}\ge p,\end{array}$$

and:

$${\gamma }_n^2\sum _{j=1}^p\parallel f_j-f_j^{*}{\parallel }_{K_j}\le {\gamma }_n{\gamma }_n\sum _{j=1}^p{\parallel \widehat{f}-f^{*}\parallel }_{K_j}\le e^{-\tilde{p}}.$$

Therefore, we verify that $\widehat{f}\in \mathcal{F}({\Delta }_{-},{\Delta }_{+})$ with ${\Delta }_{-}\le e^{\tilde{p}}$ and ${\Delta }_{+}\le e^{\tilde{p}}$. With the choices ${\lambda }_2={\lambda }_1^2=\xi {\gamma }_n^2$, it holds:

$${\lambda }_1\parallel {\widehat{f}}_j-f_j^{*}{\parallel }_n+{\lambda }_2{\parallel {\widehat{f}}_j-f_j^{*}\parallel }_K\le 2({\lambda }_1+{\lambda }_2)=4\sqrt{\xi }{\gamma }_n,j\in \mathcal{S}.$$

due to the fact $\parallel f_j{\parallel }_n\le {\parallel f_j\parallel }_K\le 1$, for any $f_j\in {\mathcal{H}}_{K_j}$.

According to Lemma 4 and (11), we obtain:

$$\begin{array}{ccc}& & {\mathcal{R}}^{\sigma }(f^{*})-{\mathcal{R}}^{\sigma }(\widehat{f})\hfill \\ & \le & \frac{c\eta t_0{\parallel {\varphi }^{\prime }\parallel }_{\infty }}{{\sigma }^2}({\gamma }_n\sum _{j=1}^p\parallel {\widehat{f}}_j-f_j^{*}{\parallel }_2+{\gamma }_n^2\sum _{j=1}^p\parallel {\widehat{f}}_j-f_j^{*}{\parallel }_K)+{\lambda }_1\sum _{j\in \mathcal{S}}\parallel {\widehat{f}}_j-f_j^{*}{\parallel }_n+{\lambda }_2\sum _{j\in \mathcal{S}}{\parallel {\widehat{f}}_j-f_j^{*}\parallel }_K+e^{-\tilde{p}}\hfill \\ & \le & \frac{c\eta (t_0){\parallel {\varphi }^{\prime }\parallel }_{\infty }}{{\sigma }^2}\sqrt{\xi }{\gamma }_n+e^{-\tilde{p}},\hfill \end{array}$$

with probability at least $1-2{\tilde{p}}^{-A}$.

Notice that $log\tilde{p}\ge 2loglogn$ implies that $e^{-\tilde{p}}\le n^{-2}\le {\gamma }_n$. Then:

$${\mathcal{R}}^{\sigma }(f^{*})-\mathcal{R}(\widehat{f})\le \frac{c\eta (t_0){\parallel {\varphi }^{\prime }\parallel }_{\infty }}{{\sigma }^2}\sqrt{\xi }{\gamma }_n.$$

Combining this with Theorem 9 in [17] and setting $\sigma =(\parallel {\varphi }^{\prime }{{\parallel }_{\infty }\eta (t_0)\sqrt{\xi }{\gamma }_n)}^{\frac{1}{4}}$, we obtain the desired result. □

The proof of Theorem 1 is inspired by that of Theorem 1 in [28]; see [28] for more details. According to Theorem 1, we can conclude that the mode-based SpAM can achieve the learning rate with polynomial decay $\mathcal{O}(n^{-\frac{1}{4+4\alpha }})$ since $ϵ\in [0,1]$ and $A,\tilde{p}$ are positive constants.

4. Experimental Evaluation

To demonstrate the efficiency of our method, in this section, we evaluated our model on some synthetic datasets. The data in ${\mathbb{R}}^p$ with dimension $p=5$ and $p=10$ were generated randomly according to the uniform distribution on the interval $[0,1]$. Then, we computed the MSE of our estimator $\widehat{f}$. Figure 1, Figure 2 and Figure 3 depict the MSE of $\widehat{f}$ when the parameter pair $({\lambda }_1,{\lambda }_2)=(0,1),(1,0)$ and $(1,1)$, respectively, while the number of samples n varies from 50/60 to 80/90. This paper used Yalmip [43] modeling in the MATLAB environment and called fmincon to solve the problem. From the figures, we can notice that the MSEs tended to decrease with the increase of the number of samples n under three kinds of parameter settings, which verified that our method was effective in the regression of high-dimensional data.

Figure 1.

MSE of $\widehat{f}$ when $({\lambda }_1,{\lambda }_2)=(0,1)$.

Figure 2.

MSE of $\widehat{f}$ when $({\lambda }_1,{\lambda }_2)=(1,0)$.

Figure 3.

MSE of $\widehat{f}$ when $({\lambda }_1,{\lambda }_2)=(1,1)$.

5. Conclusions

In this work, we proposed a mode-based sparse additive model and established its generalization error bound. The theoretical results extended the previous mean-based analysis to the mode-based approach. We demonstrated that the mode-based SpAM can achieve the learning rate with polynomial decay $\mathcal{O}(n^{-\frac{1}{4+4\alpha }})$, which is comparable to the previous result in [15] with $\mathcal{O}(n^{-\frac{1}{7}})$. In the future, it will be important to further explore the variable selection consistency of the proposed model.

Author Contributions

Conceptualization, H.D., B.S., J.C. and Z.P.; methodology, H.D. and Z.P.; validation, B.S. and Z.P.; formal analysis, H.D., B.S. and Z.P.; investigation, H.D. and J.C.; resources, Z.P.; data curation, H.D. and J.C.; writing—original draft preparation, H.D. and J.C.; writing—review and editing, H.D. and J.C.; visualization, H.D. and J.C.; supervision, B.S. and Z.P.; project administration, B.S. and Z.P.; funding acquisition, H.D., B.S. and Z.P. All authors have read and agreed to the published version of the manuscript.

Funding

This work was supported in part by the Fundamental Research Funds for the Central Universities of China (Grant Nos. 2662019FW003 and 2662020LXQD001) and the National Natural Science Foundation of China (Grant No. 12001217).

Data Availability Statement

The synthetic data generation method of the simulation experiment has been introduced in the experimental part.

Conflicts of Interest

The authors declare no conflict of interest.