Gradient-type penalty method with inertial effects for solving constrained convex optimization problems with smooth data

Abstract

We consider the problem of minimizing a smooth convex objective function subject to the set of minima of another differentiable convex function. In order to solve this problem, we propose an algorithm which combines the gradient method with a penalization technique. Moreover, we insert in our algorithm an inertial term, which is able to take advantage of the history of the iterates. We show weak convergence of the generated sequence of iterates to an optimal solution of the optimization problem, provided a condition expressed via the Fenchel conjugate of the constraint function is fulfilled. We also prove convergence for the objective function values to the optimal objective value. The convergence analysis carried out in this paper relies on the celebrated Opial Lemma and generalized Fejér monotonicity techniques. We illustrate the functionality of the method via a numerical experiment addressing image classification via support vector machines.

Introduction and preliminaries

Let H be a real Hilbert space with the norm and inner product given by $\Vert ·\Vert$ and $⟨·,·⟩$, respectively, and f and g be convex functions acting on H, which we assume for simplicity to be everywhere defined and (Fréchet) differentiable. The object of our investigation is the optimization problem

$$\begin{array}{c}\hfill \underset{x\in \mathrm{argmin}g}{min}f(x).\end{array}$$ 1

We assume that

$$\begin{array}{c}\hfill \mathcal{S}:=\mathrm{argmin}\left\{f(x):x\in \mathrm{argmin}g\right\}\ne \mathrm{\varnothing }\end{array}$$

and that the gradients $\mathrm{\nabla }f$ and $\mathrm{\nabla }g$ are Lipschitz continuous operators with constants $L_f$ and $L_g$, respectively.

The work [5] of Attouch and Czarnecki has attracted since its appearance a huge interest from the research community, since it undertakes a qualitative analysis of the optimal solutions of (1) from the perspective of a penalty-term based dynamical system. This represented the starting point for the design and development of numerical algorithms for solving the minimization problem (1), several variants of it involving also nonsmooth data up to monotone inclusions that are related to optimality systems of constrained optimization problems. We refer the reader to [4–8, 10, 11, 13–15, 20–23, 33, 35] and the references therein for more insights into this research topic.

A key assumption used in this context in order to guarantee the convergence properties of the numerical algorithms is the condition

$$\begin{array}{c}\hfill \sum _{n=1}^{\infty }{\mathit{\lambda }}_n{\mathit{\beta }}_n\left[g^{\ast }\left(\frac{p}{{\mathit{\beta }}_n}\right)-{\mathit{\sigma }}_{\mathrm{argmin}g}\left(\frac{p}{{\mathit{\beta }}_n}\right)\right]<+\infty \forall p\in \mathrm{ran}(N_{\mathrm{argmin}g}),\end{array}$$

where ${\{{\mathit{\lambda }}_n\}}_{n=1}^{\infty }$ and ${\{{\mathit{\beta }}_n\}}_{n=1}^{\infty }$ are positive sequences, $g^{\ast }:\mathcal{H}\to \mathbb{R}\cup \{+\infty \}$ is the Fenchel conjugate of g:

$$\begin{array}{c}\hfill g^{\ast }(p)=\underset{x\in \mathcal{H}}{sup}\{⟨p,x⟩-g(x)\}\forall p\in \mathcal{H};\end{array}$$

${\mathit{\sigma }}_{\mathrm{argmin}g}:\mathcal{H}\to \mathbb{R}\cup \{+\infty \}$ is the support function of the set $\mathrm{argmin}g$:

$$\begin{array}{c}\hfill {\mathit{\sigma }}_{\mathrm{argmin}g}(p)=\underset{x\in \mathrm{argmin}g}{sup}⟨p,x⟩\forall p\in \mathcal{H};\end{array}$$

and $N_{\mathrm{argmin}g}$ is the normal cone to the set $\mathrm{argmin}g$, defined by

$$\begin{array}{c}\hfill N_{\mathrm{argmin}g}(x)=\{p\in \mathcal{H}:⟨p,y-x⟩\le 0\forall y\in \mathrm{argmin}g\}\end{array}$$

for $x\in \mathrm{argmin}g$ and $N_{\mathrm{argmin}g}(x)=\mathrm{\varnothing }$ for $x\notin \mathrm{argmin}g$. Finally, $\mathrm{ran}(N_{\mathrm{argmin}g})$ denotes the range of the normal cone $N_{\mathrm{argmin}g}$, that is, $p\in \mathrm{ran}(N_{\mathrm{argmin}g})$ if and only if there exists $x\in \mathrm{argmin}g$ such that $p\in N_{\mathrm{argmin}g}(x)$. Let us notice that for $x\in \mathrm{argmin}g$ one has $p\in N_{\mathrm{argmin}g}(x)$ if and only if ${\mathit{\sigma }}_{\mathrm{argmin}g}(p)=⟨p,x⟩$. We also assume without loss of generality that $ming=0$.

In this paper we propose a numerical algorithm for solving (1) that combines the gradient method with penalization strategies also by employing inertial and memory effects. Algorithms of inertial type result from the time discretization of differential inclusions of second order type (see [1, 3]) and were first investigated in the context of the minimization of a differentiable function by Polyak [36] and Bertsekas [12]. The resulting iterative schemes share the feature that the next iterate is defined by means of the last two iterates, a fact which induces the inertial effect in the algorithm. Since the works [1, 3], one can notice an increasing number of research efforts dedicated to algorithms of inertial type (see [1–3, 9, 16–19, 24–28, 30–32, 34]).

In this paper we consider the following inertial algorithm for solving (1):

Algorithm 1

Initialization: Choose the positive sequences ${\{{\mathit{\lambda }}_n\}}_{n=1}^{\infty }$ and ${\{{\mathit{\beta }}_n\}}_{n=1}^{\infty }$, and a positive constant parameter $\mathit{\alpha }\in (0,1)$. Take arbitrary $x_0,x_1\in H$.

Iterative step: For given current iterates $x_{n-1},x_n\in H$ ($n\ge 1$), define $x_{n+1}\in H$ by

$$\begin{array}{c}\hfill x_{n+1}:=x_n+\mathit{\alpha }(x_n-x_{n-1})-{\mathit{\lambda }}_n\mathrm{\nabla }f(x_n)-{\mathit{\lambda }}_n{\mathit{\beta }}_n\mathrm{\nabla }g(x_n).\end{array}$$

We notice that in the above iterative scheme ${\{{\mathit{\lambda }}_n\}}_{n=1}^{\infty }$ represents the sequence of step sizes, ${\{{\mathit{\beta }}_n\}}_{n=1}^{\infty }$ the sequence of penalty parameters, while $\mathit{\alpha }$ controls the influence of the inertial term.

For every $n\ge 1$ we denote by ${\mathrm{\Omega }}_n:=f+{\mathit{\beta }}_{n}g$, which is also a (Fréchet) differentiable function, and notice that $\mathrm{\nabla }{\mathrm{\Omega }}_n$ is $L_n:=L_f+{\mathit{\beta }}_n{L}_g$-Lipschitz continuous.

In case $\mathit{\alpha }=0$, Algorithm 1 collapses in the algorithm considered in [35] for solving (1). We prove weak convergence for the generated iterates to an optimal solution of (1), by making use of generalized Fejér monotonicity techniques and the Opial Lemma and by imposing the key assumption mentioned above as well as some mild conditions on the involved parameters. Moreover, the performed analysis allows us also to show the convergence of the objective function values to the optimal objective value of (1). As an illustration of the theoretical results, we present in the last section an application addressing image classification via support vector machines.

Convergence analysis

This section is devoted to the asymptotic analysis of Algorithm 1.

Assumption 2

Assume that the following statements hold:

• (I)The function f is bounded from below;
• (II)There exist positive constants $c>1$ and $K>0$ such that $\frac{L_n}{2}+\frac{\mathit{\alpha }-1}{{\mathit{\lambda }}_n}\le -\left(c+(1+\mathit{\alpha })K\right)$ and ${\mathit{\beta }}_{n+1}-{\mathit{\beta }}_n\le K{\mathit{\lambda }}_{n+1}{\mathit{\beta }}_{n+1}$ for all $n\ge 1$;
• (III)For every $p\in \mathrm{ran}(N_{\mathrm{argmin}g})$, we have ${\sum }_{n=1}^{\infty }{\mathit{\lambda }}_n{\mathit{\beta }}_n\left[g^{\ast }\left(\frac{p}{{\mathit{\beta }}_n}\right)-{\mathit{\sigma }}_{\mathrm{argmin}g}\left(\frac{p}{{\mathit{\beta }}_n}\right)\right]<+\infty$;
• (IV) ${lim\; inf}_{n\to +\infty }{\mathit{\lambda }}_n{\mathit{\beta }}_n>0$, $\left(\frac{1}{{\mathit{\lambda }}_{n+1}}-\frac{1}{{\mathit{\lambda }}_n}\right)\le \frac{2}{\mathit{\alpha }}$ for all $n\ge 1$ and ${\sum }_{n=1}^{\infty }{\mathit{\lambda }}_n=+\infty$.

We would like to mention that in [21] we proposed a forward-backward-forward algorithm of penalty-type, endowed with inertial and memory effects, for solving monotone inclusion problems, which gave rise to a primal-dual iterative scheme for solving convex optimization problems with complex structures. However, we succeeded in proving only weak ergodic convergence for the generated iterates, while with the specific choice of the sequences ${\{{\mathit{\lambda }}_n\}}_{n=1}^{\infty }$ and ${\{{\mathit{\beta }}_n\}}_{n=1}^{\infty }$ in Assumption 2 we will be able to prove weak convergence of the iterates generated in Algorithm 1 to an optimal solution of (1).

Remark 3

The conditions in Assumption 2 slightly extend the ones considered in [35] in the noninertial case. The only differences are given by the first inequality in (II), which here involves the constant $\mathit{\alpha }$ which controls the inertial terms (for the corresponding condition in [35] one only has to take $\mathit{\alpha }=0$), and by the inequality $\left(\frac{1}{{\mathit{\lambda }}_{n+1}}-\frac{1}{{\mathit{\lambda }}_n}\right)\le \frac{2}{\mathit{\alpha }}$ for all $n\ge 1$.

We refer to Remark 12 for situations where the fulfillment of the conditions in Assumption 2 is guaranteed.

We start the convergence analysis with three technical results.

Lemma 4

Let $\overline{x}\in \mathcal{S}$ and set $\overline{p}:=-\mathrm{\nabla }f(\overline{x})$. We have for all $n\ge 1$

$$\begin{array}{cc}& {\mathit{\phi }}_{n+1}-{\mathit{\phi }}_n-\mathit{\alpha }\left({\mathit{\phi }}_n-{\mathit{\phi }}_{n-1}\right)+{\mathit{\lambda }}_n{\mathit{\beta }}_{n}g(x_n)\hfill \\ \hfill & \le \Vert x_{n+1}-x_n{\Vert }^2+\mathit{\alpha }{\Vert x_n-x_{n-1}\Vert }^2\hfill \\ \hfill & +{\mathit{\lambda }}_n{\mathit{\beta }}_n\left[g^{\ast }\left(\frac{2\overline{p}}{{\mathit{\beta }}_n}\right)-{\mathit{\sigma }}_{\mathrm{argmin}g}\left(\frac{2\overline{p}}{{\mathit{\beta }}_n}\right)\right],\hfill \end{array}$$ 2

where ${\mathit{\phi }}_n:={\Vert x_n-\overline{x}\Vert }^2$.

Proof

Since $\overline{x}\in \mathcal{S}$, we have according to the first-order optimality conditions that $0\in \mathrm{\nabla }f(\overline{x})+N_{\mathrm{argmin}g}(\overline{x})$, thus $\overline{p}=-\mathrm{\nabla }f(\overline{x})\in N_{\mathrm{argmin}g}(\overline{x})$. Notice that for all $n\ge 1$

$$\begin{array}{c}\hfill \mathrm{\nabla }f(x_n)=\frac{y_n-x_{n+1}}{{\mathit{\lambda }}_n}-{\mathit{\beta }}_n\mathrm{\nabla }g(x_n),\end{array}$$

where $y_n:=x_n+\mathit{\alpha }(x_n-x_{n-1})$. This, together with the monotonicity of $\mathrm{\nabla }f$, imply that

$$\begin{array}{cc}& 〈\frac{y_n-x_{n+1}}{{\mathit{\lambda }}_n}-{\mathit{\beta }}_n\mathrm{\nabla }g(x_n)+\overline{p},x_n-\overline{x}〉\hfill \\ \hfill & =〈\mathrm{\nabla }f(x_n)-\mathrm{\nabla }f(\overline{x}),x_n-\overline{x}〉\ge 0\forall n\ge 1,\hfill \end{array}$$ 3

so

$$\begin{array}{c}\hfill 2〈y_n-x_{n+1},x_n-\overline{x}〉\ge 2{\mathit{\lambda }}_n{\mathit{\beta }}_n⟨\mathrm{\nabla }g(x_n),x_n-\overline{x}⟩-2{\mathit{\lambda }}_n⟨\overline{p},x_n-\overline{x}⟩\forall n\ge 1.\end{array}$$ 4

On the other hand, since g is convex and differentiable, we have for all $n\ge 1$

$$\begin{array}{c}\hfill 0=g(\overline{x})\ge g(x_n)+⟨\mathrm{\nabla }g(x_n),\overline{x}-x_n⟩,\end{array}$$

which means that

$$\begin{array}{c}\hfill 2{\mathit{\lambda }}_n{\mathit{\beta }}_n⟨\mathrm{\nabla }g(x_n),x_n-\overline{x}⟩\ge 2{\mathit{\lambda }}_n{\mathit{\beta }}_{n}g(x_n).\end{array}$$ 5

As for all $n\ge 1$

$$\begin{array}{c}\hfill 2⟨x_n-x_{n+1},x_n-\overline{x}⟩={\Vert x_{n+1}-x_n\Vert }^2+{\mathit{\phi }}_n-{\mathit{\phi }}_{n+1}\end{array}$$

and

$$\begin{array}{c}\hfill 2\mathit{\alpha }⟨x_n-x_{n-1},x_n-\overline{x}⟩=\mathit{\alpha }{\Vert x_n-x_{n-1}\Vert }^2+\mathit{\alpha }\left({\mathit{\phi }}_n-{\mathit{\phi }}_{n-1}\right),\end{array}$$

it follows

$$\begin{array}{cc}\hfill 2⟨y_n-x_{n+1},x_n-\overline{x}⟩=& 2⟨x_n-x_{n+1},x_n-\overline{x}⟩+2\mathit{\alpha }⟨x_n-x_{n-1},x_n-\overline{x}⟩\hfill \\ \hfill =& \Vert x_{n+1}-x_n{\Vert }^2+\mathit{\alpha }{\Vert x_n-x_{n-1}\Vert }^2\hfill \\ \hfill & +{\mathit{\phi }}_n-{\mathit{\phi }}_{n+1}+\mathit{\alpha }\left({\mathit{\phi }}_n-{\mathit{\phi }}_{n-1}\right).\hfill \end{array}$$ 6

Combining (4), (5) and (6), we obtain that for each $n\ge 1$

$$\begin{array}{cc}& {\mathit{\phi }}_{n+1}-{\mathit{\phi }}_n-\mathit{\alpha }\left({\mathit{\phi }}_n-{\mathit{\phi }}_{n-1}\right)+{\mathit{\lambda }}_n{\mathit{\beta }}_{n}g(x_n)\hfill \\ \hfill & \le \Vert x_{n+1}-x_n{\Vert }^2+\mathit{\alpha }{\Vert x_n-x_{n-1}\Vert }^2-{\mathit{\lambda }}_n{\mathit{\beta }}_{n}g(x_n)\hfill \\ \hfill & +2{\mathit{\lambda }}_n⟨\overline{p},x_n⟩-2{\mathit{\lambda }}_n⟨\overline{p},\overline{x}⟩.\hfill \end{array}$$ 7

Finally, since $\overline{x}\in \mathrm{argmin}g$, we have that for all $n\ge 1$

$$\begin{array}{cc}\hfill 2{\mathit{\lambda }}_n⟨\overline{p},x_n⟩-{\mathit{\lambda }}_n{\mathit{\beta }}_{n}g(x_n)-2{\mathit{\lambda }}_n⟨\overline{p},\overline{x}⟩=& {\mathit{\lambda }}_n{\mathit{\beta }}_n\left[〈\frac{2\overline{p}}{{\mathit{\beta }}_n},x_n〉-g(x_n)-〈\frac{2\overline{p}}{{\mathit{\beta }}_n},\overline{x}〉\right]\hfill \\ \hfill \le & {\mathit{\lambda }}_n{\mathit{\beta }}_n\left[g^{\ast }\left(\frac{2\overline{p}}{{\mathit{\beta }}_n}\right)-〈\frac{2\overline{p}}{{\mathit{\beta }}_n},\overline{x}〉\right]\hfill \\ \hfill =& {\mathit{\lambda }}_n{\mathit{\beta }}_n\left[g^{\ast }\left(\frac{2\overline{p}}{{\mathit{\beta }}_n}\right)-{\mathit{\sigma }}_{\mathrm{argmin}g}\left(\frac{2\overline{p}}{{\mathit{\beta }}_n}\right)\right],\hfill \end{array}$$

which completes the proof. $\square$

Lemma 5

We have for all $n\ge 1$

$$\begin{array}{cc}\hfill {\mathrm{\Omega }}_{n+1}(x_{n+1})\le & {\mathrm{\Omega }}_n(x_n)+({\mathit{\beta }}_{n+1}-{\mathit{\beta }}_n)g(x_{n+1})\hfill \\ \hfill & +\left[\frac{L_n}{2}+\frac{\mathit{\alpha }}{2{\mathit{\lambda }}_n}-\frac{1}{{\mathit{\lambda }}_n}\right]\Vert x_{n+1}-x_n{\Vert }^2+\frac{\mathit{\alpha }}{2{\mathit{\lambda }}_n}{\Vert x_n-x_{n-1}\Vert }^2.\hfill \end{array}$$ 8

Proof

From the descent Lemma and the fact that $\mathrm{\nabla }{\mathrm{\Omega }}_n$ is $L_n$-Lipschitz continuous, we get that

$$\begin{array}{c}\hfill {\mathrm{\Omega }}_n(x_{n+1})\le {\mathrm{\Omega }}_n(x_n)+〈\mathrm{\nabla }{\mathrm{\Omega }}_n(x_n),x_{n+1}-x_n〉+\frac{L_n}{2}{\Vert x_{n+1}-x_n\Vert }^2\forall n\ge 1.\end{array}$$

Since $\mathrm{\nabla }{\mathrm{\Omega }}_n(x_n)=-\frac{x_{n+1}-y_n}{{\mathit{\lambda }}_n}$, it holds for all $n\ge 1$

$$\begin{array}{cc}\hfill f(x_{n+1})+{\mathit{\beta }}_{n}g(x_{n+1})\le & f(x_n)+{\mathit{\beta }}_{n}g(x_n)\hfill \\ \hfill & -〈\frac{x_{n+1}-y_n}{{\mathit{\lambda }}_n},x_{n+1}-x_n〉+\frac{L_n}{2}{\Vert x_{n+1}-x_n\Vert }^2\hfill \end{array}$$

and then

$$\begin{array}{cc}\hfill f(x_{n+1})+{\mathit{\beta }}_{n+1}g(x_{n+1})\le & f(x_n)+{\mathit{\beta }}_{n}g(x_n)+({\mathit{\beta }}_{n+1}-{\mathit{\beta }}_n)g(x_{n+1})\hfill \\ \hfill & -\frac{1}{{\mathit{\lambda }}_n}{\Vert x_{n+1}-x_n\Vert }^2+\frac{\mathit{\alpha }}{{\mathit{\lambda }}_n}〈x_n-x_{n-1},x_{n+1}-x_n〉\hfill \\ \hfill & +\frac{L_n}{2}{\Vert x_{n+1}-x_n\Vert }^2,\hfill \end{array}$$

which is nothing else than

$$\begin{array}{cc}\hfill {\mathrm{\Omega }}_{n+1}(x_{n+1})\le & {\mathrm{\Omega }}_n(x_n)+({\mathit{\beta }}_{n+1}-{\mathit{\beta }}_n)g(x_{n+1})+\left[\frac{L_n}{2}-\frac{1}{{\mathit{\lambda }}_n}\right]{\Vert x_{n+1}-x_n\Vert }^2\hfill \\ \hfill & +\frac{\mathit{\alpha }}{{\mathit{\lambda }}_n}〈x_n-x_{n-1},x_{n+1}-x_n〉.\hfill \end{array}$$ 9

By the Cauchy–Schwarz inequalty it holds that

$$\begin{array}{cc}\hfill ⟨x_n-x_{n-1},x_{n+1}-x_n⟩\le & \frac{1}{2}\Vert x_{n-1}-x_n{\Vert }^2+\frac{1}{2}{\Vert x_{n+1}-x_n\Vert }^2,\hfill \end{array}$$

hence, (9) becomes

$$\begin{array}{cc}\hfill {\mathrm{\Omega }}_{n+1}(x_{n+1})\le & {\mathrm{\Omega }}_n(x_n)+({\mathit{\beta }}_{n+1}-{\mathit{\beta }}_n)g(x_{n+1})+\frac{\mathit{\alpha }}{2{\mathit{\lambda }}_n}{\Vert x_{n-1}-x_n\Vert }^2\hfill \\ \hfill & +\left[\frac{L_n}{2}-\frac{1}{{\mathit{\lambda }}_n}+\frac{\mathit{\alpha }}{2{\mathit{\lambda }}_n}\right]{\Vert x_{n+1}-x_n\Vert }^2\forall n\ge 1.\hfill \end{array}$$

For $\overline{x}\in \mathcal{S}$ and all $n\ge 1$, we set

$$\begin{array}{cc}\hfill {\mathrm{\Gamma }}_n:=& f(x_n)+(1-K{\mathit{\lambda }}_n){\mathit{\beta }}_{n}g(x_n)+K{\mathit{\phi }}_n\hfill \\ \hfill =& {\mathrm{\Omega }}_n(x_n)-K{\mathit{\lambda }}_n{\mathit{\beta }}_{n}g(x_n)+K{\mathit{\phi }}_n,\hfill \end{array}$$

and, for simplicity, we denote

$$\begin{array}{c}\hfill {\mathit{\delta }}_n:=\left(\frac{1}{2{\mathit{\lambda }}_n}+K\right)\mathit{\alpha }+c.\end{array}$$

Lemma 6

Let $\overline{x}\in \mathcal{S}$ and set $\overline{p}:=-\mathrm{\nabla }f(\overline{x})$. We have for all $n\ge 2$

$$\begin{array}{cc}\hfill {\mathrm{\Gamma }}_{n+1}-{\mathrm{\Gamma }}_n-\mathit{\alpha }({\mathrm{\Gamma }}_n-{\mathrm{\Gamma }}_{n-1})\le & -{\mathit{\delta }}_n\Vert x_{n+1}-x_n{\Vert }^2+\mathit{\alpha }\left(\frac{1}{2{\mathit{\lambda }}_n}+K\right){\Vert x_n-x_{n-1}\Vert }^2\hfill \\ \hfill & +K{\mathit{\lambda }}_n{\mathit{\beta }}_n\left[g^{\ast }\left(\frac{2\overline{p}}{{\mathit{\beta }}_n}\right)-{\mathit{\sigma }}_{\mathrm{argmin}g}\left(\frac{2\overline{p}}{{\mathit{\beta }}_n}\right)\right]\hfill \\ \hfill & +\mathit{\alpha }\left({\mathrm{\Omega }}_{n-1}(x_{n-1})-{\mathrm{\Omega }}_n(x_n)\right)\hfill \\ \hfill & +\mathit{\alpha }K\left({\mathit{\lambda }}_n{\mathit{\beta }}_{n}g(x_n)-{\mathit{\lambda }}_{n-1}{\mathit{\beta }}_{n-1}g(x_{n-1})\right).\hfill \end{array}$$ 10

Proof

According to Lemma 5 and Assumption 2(II), (8) becomes for all $n\ge 1$

$$\begin{array}{cc}\hfill {\mathrm{\Omega }}_{n+1}(x_{n+1})-{\mathrm{\Omega }}_n(x_n)-K{\mathit{\lambda }}_{n+1}{\mathit{\beta }}_{n+1}g(x_{n+1})\le & -(K+{\mathit{\delta }}_n){\Vert x_{n+1}-x_n\Vert }^2\hfill \\ \hfill & +\frac{\mathit{\alpha }}{2{\mathit{\lambda }}_n}{\Vert x_n-x_{n-1}\Vert }^2.\hfill \end{array}$$ 11

On the other hand, after multiplying (2) by K, we obtain for all $n\ge 1$

$$\begin{array}{cc}& K{\mathit{\phi }}_{n+1}-K{\mathit{\phi }}_n-\mathit{\alpha }\left(K{\mathit{\phi }}_n-K{\mathit{\phi }}_{n-1}\right)+K{\mathit{\lambda }}_n{\mathit{\beta }}_{n}g(x_n)\hfill \\ \hfill & \le K\Vert x_{n+1}-x_n{\Vert }^2+K\mathit{\alpha }{\Vert x_n-x_{n-1}\Vert }^2+K{\mathit{\lambda }}_n{\mathit{\beta }}_n\left[g^{\ast }\left(\frac{2\overline{p}}{{\mathit{\beta }}_n}\right)-{\mathit{\sigma }}_{\mathrm{argmin}g}\left(\frac{2\overline{p}}{{\mathit{\beta }}_n}\right)\right].\hfill \end{array}$$ 12

After summing up the relations (11) and (12) and adding on both sides of the resulting inequality the expressions $\mathit{\alpha }\left({\mathrm{\Omega }}_{n-1}(x_{n-1})-{\mathrm{\Omega }}_n(x_n)\right)$ and $\mathit{\alpha }(K{\mathit{\lambda }}_n{\mathit{\beta }}_{n}g(x_n)-K{\mathit{\lambda }}_{n-1}{\mathit{\beta }}_{n-1}g(x_{n-1}))$ for all $n\ge 2$, we obtain the required statement. $\square$

The following proposition will play an essential role in the convergence analysis (see also [1–3, 16]).

Proposition 7

Let ${\{a_n\}}_{n=1}^{\infty },{\{b_n\}}_{n=1}^{\infty }$ and ${\{c_n\}}_{n=1}^{\infty }$ be real sequences and $\mathit{\alpha }\in [0,1)$ be given. Assume that ${\{a_n\}}_{n=1}^{\infty }$ is bounded from below, ${\{b_n\}}_{n=1}^{\infty }$ is nonnegative and ${\sum }_{n=1}^{\infty }{c}_n<+\infty$ such that

$$\begin{array}{c}\hfill a_{n+1}-a_n-\mathit{\alpha }(a_n-a_{n-1})+b_n\le c_n\forall n\ge 1.\end{array}$$

Then the following statements hold:

• (i) ${\sum }_{n=1}^{\infty }{[a_n-a_{n-1}]}_{+}<+\infty$, where ${[t]}_{+}:=max\{t,0\}$;
• (ii) ${\{a_n\}}_{n=1}^{\infty }$ converges and ${\sum }_{n=1}^{\infty }{b}_n<+\infty$.

The following lemma collects some convergence properties of the sequences involved in our analysis.

Lemma 8

Let $\overline{x}\in \mathcal{S}$. Then the following statements are true:

• (i)The sequence ${\{{\mathrm{\Gamma }}_n\}}_{n=1}^{\infty }$ is bounded from below.
• (ii) ${\sum }_{n=1}^{\infty }{\Vert x_{n+1}-x_n\Vert }^2<+\infty$ and ${lim}_{n\to +\infty }{\mathrm{\Gamma }}_n$ exists.
• (iii) ${lim}_{n\to +\infty }\Vert x_n-\overline{x}\Vert$ exists and ${\sum }_{n=1}^{\infty }{\mathit{\lambda }}_n{\mathit{\beta }}_{n}g(x_n)<+\infty$.
• (iv) ${lim}_{n\to +\infty }{\mathrm{\Omega }}_n(x_n)$ exists.
• (v) ${lim}_{n\to +\infty }g(x_n)=0$ and every sequential weak cluster point of the sequence ${\{x_n\}}_{n=1}^{\infty }$ lies in $\mathrm{argmin}g$.

Proof

We set $\overline{p}:=-\mathrm{\nabla }f(\overline{x})$ and recall that $g(\overline{x})=ming=0$.

(i) Since f is convex and differentiable, it holds for all $n\ge 1$

$$\begin{array}{cc}\hfill {\mathrm{\Gamma }}_n=& f(x_n)+(1-K{\mathit{\lambda }}_n){\mathit{\beta }}_{n}g(x_n)+K{\mathit{\phi }}_n\hfill \\ \hfill \ge & f(x_n)+K{\Vert x_n-\overline{x}\Vert }^2\hfill \\ \hfill \ge & f(\overline{x})+⟨\mathrm{\nabla }f(\overline{x}),x_n-\overline{x}⟩+K{\Vert x_n-\overline{x}\Vert }^2\ge f(\overline{x})-\frac{\Vert \overline{p}{\Vert }^2}{4K},\hfill \end{array}$$

which means that ${\{{\mathrm{\Gamma }}_n\}}_{n=1}^{\infty }$ is bounded from below. Notice that the first inequality in the above relation is a consequence of Assumption 2(II), since $\frac{1-\mathit{\alpha }}{{\mathit{\lambda }}_n}\ge c+(1+\mathit{\alpha })K\ge K$, thus ${\mathit{\lambda }}_{n}K\le 1-\mathit{\alpha }\le 1$ for all $n\ge 1$.

(ii) For all $n\ge 2$, we may set

$$\begin{array}{c}\hfill {\mathit{\mu }}_n:={\mathrm{\Gamma }}_n-\mathit{\alpha }{\mathrm{\Gamma }}_{n-1}+\mathit{\alpha }\left(\frac{1}{2{\mathit{\lambda }}_n}+K\right){\Vert x_n-x_{n-1}\Vert }^2\end{array}$$

and

$$\begin{array}{c}\hfill u_n:={\mathrm{\Omega }}_{n-1}(x_{n-1})-{\mathrm{\Omega }}_n(x_n)+K{\mathit{\lambda }}_n{\mathit{\beta }}_{n}g(x_n)-K{\mathit{\lambda }}_{n-1}{\mathit{\beta }}_{n-1}g(x_{n-1}).\end{array}$$

We fix a natural number $N_0\ge 2$. Then

$$\begin{array}{c}\hfill \sum _{n=2}^{N_0}{u}_n=f(x_1)+(1-K{\mathit{\lambda }}_1){\mathit{\beta }}_{1}g(x_1)-f(x_{N_0})-(1-K{\mathit{\lambda }}_{N_0}){\mathit{\beta }}_{N_0}g(x_{N_0}).\end{array}$$

Since f is bounded from below and $g(x_{N_0})\ge g(\overline{x})=0$, it follows that ${\sum }_{n=2}^{\infty }{u}_n<+\infty$.

We notice that $-{\mathit{\delta }}_n+\mathit{\alpha }\left(\frac{1}{2{\mathit{\lambda }}_{n+1}}+K\right)=\frac{\mathit{\alpha }}{2}\left(\frac{1}{{\mathit{\lambda }}_{n+1}}-\frac{1}{{\mathit{\lambda }}_n}\right)-c$ and, since $\left(\frac{1}{{\mathit{\lambda }}_{n+1}}-\frac{1}{{\mathit{\lambda }}_n}\right)\le \frac{2}{\mathit{\alpha }}$, we have for all $n\ge 1$

$$\begin{array}{c}\hfill -{\mathit{\delta }}_n+\mathit{\alpha }\left(\frac{1}{2{\mathit{\lambda }}_{n+1}}+K\right)\le 1-c.\end{array}$$ 13

Thus, according Lemma 6, we get for all $n\ge 2$

$$\begin{array}{cc}\hfill {\mathit{\mu }}_{n+1}-{\mathit{\mu }}_n=& {\mathrm{\Gamma }}_{n+1}-{\mathrm{\Gamma }}_n-\mathit{\alpha }({\mathrm{\Gamma }}_n-{\mathrm{\Gamma }}_{n-1})+\mathit{\alpha }\left(\frac{1}{2{\mathit{\lambda }}_{n+1}}+K\right){\Vert x_{n+1}-x_n\Vert }^2\hfill \\ \hfill & -\mathit{\alpha }\left(\frac{1}{2{\mathit{\lambda }}_n}+K\right){\Vert x_n-x_{n-1}\Vert }^2\hfill \\ \hfill \le & -{\mathit{\delta }}_n{\Vert x_{n+1}-x_n\Vert }^2+K{\mathit{\lambda }}_n{\mathit{\beta }}_n\left[g^{\ast }\left(\frac{2\overline{p}}{{\mathit{\beta }}_n}\right)-{\mathit{\sigma }}_{\mathrm{argmin}g}\left(\frac{2\overline{p}}{{\mathit{\beta }}_n}\right)\right]\hfill \\ \hfill & +\mathit{\alpha }{u}_n+\mathit{\alpha }\left(\frac{1}{2{\mathit{\lambda }}_{n+1}}+K\right){\Vert x_{n+1}-x_n\Vert }^2\hfill \\ \hfill \le & (1-c)\Vert x_{n+1}-x_n{\Vert }^2+K{\mathit{\lambda }}_n{\mathit{\beta }}_n\left[g^{\ast }\left(\frac{2\overline{p}}{{\mathit{\beta }}_n}\right)-{\mathit{\sigma }}_{\mathrm{argmin}g}\left(\frac{2\overline{p}}{{\mathit{\beta }}_n}\right)\right]+\mathit{\alpha }{u}_n.\hfill \end{array}$$

We fix another natural number $N_1\ge 2$ and sum up the last inequality for $n=2,\dots ,N_1$. We obtain

$$\begin{array}{cc}\hfill {\mathit{\mu }}_{N_1+1}-{\mathit{\mu }}_2\le & (1-c)\sum _{n=2}^{N_1}{\Vert x_{n+1}-x_n\Vert }^2\hfill \\ \hfill & +K\sum _{n=2}^{N_1}{\mathit{\lambda }}_n{\mathit{\beta }}_n\left[g^{\ast }\left(\frac{2\overline{p}}{{\mathit{\beta }}_n}\right)-{\mathit{\sigma }}_{\mathrm{argmin}g}\left(\frac{2\overline{p}}{{\mathit{\beta }}_n}\right)\right]\hfill \\ \hfill & +\mathit{\alpha }\sum _{n=2}^{N_1}{u}_n,\hfill \end{array}$$ 14

which, by taking into account Assumption 2(III), means that ${\{{\mathit{\mu }}_n\}}_{n=2}^{\infty }$ is bounded from above by a positive number that we denote by M. Consequently, for all $n\ge 2$ we have

$$\begin{array}{c}\hfill {\mathrm{\Gamma }}_{n+1}-\mathit{\alpha }{\mathrm{\Gamma }}_n\le {\mathit{\mu }}_{n+1}\le M,\end{array}$$

so

$$\begin{array}{c}\hfill {\mathrm{\Gamma }}_{n+1}\le \mathit{\alpha }{\mathrm{\Gamma }}_n+M,\end{array}$$

which further implies that

$$\begin{array}{c}\hfill {\mathrm{\Gamma }}_n\le {\mathit{\alpha }}^{n-2}{\mathrm{\Gamma }}_2+M\sum _{k=1}^{n-2}{\mathit{\alpha }}^{k-1}\le {\mathit{\alpha }}^{n-2}{\mathrm{\Gamma }}_2+\frac{M}{1-\mathit{\alpha }}\forall n\ge 3.\end{array}$$

We have for all $n\ge 2$

$$\begin{array}{c}\hfill {\mathit{\mu }}_{n+1}\ge f(\overline{x})-\frac{\Vert \overline{p}{\Vert }^2}{4K}-\mathit{\alpha }{\mathrm{\Gamma }}_n,\end{array}$$

hence

$$\begin{array}{c}\hfill -{\mathit{\mu }}_{n+1}\le \mathit{\alpha }{\mathrm{\Gamma }}_n-f(\overline{x})+\frac{\Vert \overline{p}{\Vert }^2}{4K}\le {\mathit{\alpha }}^{n-1}{\mathrm{\Gamma }}_2+\frac{\mathit{\alpha }M}{1-\mathit{\alpha }}-f(\overline{x})+\frac{\Vert \overline{p}{\Vert }^2}{4K}.\end{array}$$ 15

Consequently, for the arbitrarily chosen natural number $N_1\ge 2$, we have [see (14)]

$$\begin{array}{cc}\hfill (c-1)\sum _{n=2}^{N_1}{\Vert x_{n+1}-x_n\Vert }^2\le & -{\mathit{\mu }}_{N_1+1}+{\mathit{\mu }}_2\hfill \\ \hfill & +K\sum _{n=2}^{N_1}{\mathit{\lambda }}_n{\mathit{\beta }}_n\left[g^{\ast }\left(\frac{2\overline{p}}{{\mathit{\beta }}_n}\right)-{\mathit{\sigma }}_{\mathrm{argmin}g}\left(\frac{2\overline{p}}{{\mathit{\beta }}_n}\right)\right]+\mathit{\alpha }\sum _{n=2}^{N_1}{u}_n,\hfill \end{array}$$

which together with (15) and the fact that $c>1$ implies that

$$\begin{array}{c}\hfill \sum _{n=1}^{\infty }{\Vert x_{n+1}-x_n\Vert }^2<+\infty .\end{array}$$

On the other hand, due to (13) we have ${\mathit{\delta }}_{n+1}\le {\mathit{\delta }}_n+1$ for all $n\ge 1$. Consequently, using also that $c>1$, (10) implies that

$$\begin{array}{cc}\hfill {\mathrm{\Gamma }}_{n+1}-{\mathrm{\Gamma }}_n-\mathit{\alpha }({\mathrm{\Gamma }}_n-{\mathrm{\Gamma }}_{n-1})\le & -{\mathit{\delta }}_n\Vert x_{n+1}-x_n{\Vert }^2+({\mathit{\delta }}_n-c){\Vert x_n-x_{n-1}\Vert }^2\hfill \\ \hfill & +K{\mathit{\lambda }}_n{\mathit{\beta }}_n\left[g^{\ast }\left(\frac{2\overline{p}}{{\mathit{\beta }}_n}\right)-{\mathit{\sigma }}_{\mathrm{argmin}g}\left(\frac{2\overline{p}}{{\mathit{\beta }}_n}\right)\right]+\mathit{\alpha }{u}_n\hfill \\ \hfill \le & -{\mathit{\delta }}_n\Vert x_{n+1}-x_n{\Vert }^2+{\mathit{\delta }}_{n-1}{\Vert x_n-x_{n-1}\Vert }^2\hfill \\ \hfill & +K{\mathit{\lambda }}_n{\mathit{\beta }}_n\left[g^{\ast }\left(\frac{2\overline{p}}{{\mathit{\beta }}_n}\right)-{\mathit{\sigma }}_{\mathrm{argmin}g}\left(\frac{2\overline{p}}{{\mathit{\beta }}_n}\right)\right]+\mathit{\alpha }{u}_n\forall n\ge 1.\hfill \end{array}$$

According to Proposition 7 and by taking into account that ${\{{\mathrm{\Gamma }}_n\}}_{n=1}^{\infty }$ is bounded from below, we obtain that ${lim}_{n\to +\infty }{\mathrm{\Gamma }}_n$ exists.

(iii) By Lemma 4 and Proposition 7, ${lim}_{n\to +\infty }{\mathit{\phi }}_n$ exists and ${\sum }_{n=1}^{\infty }{\mathit{\lambda }}_n{\mathit{\beta }}_{n}g(x_n)<+\infty$.

(iv) Since ${\mathrm{\Omega }}_n(x_n)={\mathrm{\Gamma }}_n-K{\mathit{\phi }}_n+K{\mathit{\lambda }}_n{\mathit{\beta }}_{n}g(x_n)$ for all $n\ge 1$, by using (ii) and (iii), we get that ${lim}_{n\to +\infty }{\mathrm{\Omega }}_n(x_n)$ exists.

(v) Since ${lim\; inf}_{n\to +\infty }{\mathit{\lambda }}_n{\mathit{\beta }}_n>0$, we also obtain that ${lim}_{n\to +\infty }g(x_n)=0.$ Let w be a sequential weak cluster point of ${\{x_n\}}_{n=1}^{\infty }$ and assume that the subsequence ${\{x_{n_j}\}}_{j=1}^{\infty }$ converges weakly to w. Since g is weak lower semicontinuous, we have

$$\begin{array}{c}\hfill g(w)\le \underset{j\to +\infty }{lim\; inf}g(x_{n_j})=\underset{n\to +\infty }{lim}g(x_n)=0,\end{array}$$

which implies that $w\in \mathrm{argmin}g$. This completes the proof. $\square$

In order to show also the convergence of the sequence ${(f(x_n))}_{n=1}^{\infty }$, we prove first the following result.

Lemma 9

Let $\overline{x}\in \mathcal{S}$ be given. We have

$$\begin{array}{c}\hfill \sum _{n=1}^{\infty }{\mathit{\lambda }}_n\left[{\mathrm{\Omega }}_n(x_n)-f(\overline{x})\right]<+\infty .\end{array}$$

Proof

Since f is convex and differentiable, we have for all $n\ge 1$

$$\begin{array}{c}\hfill f(\overline{x})\ge f(x_n)+⟨\mathrm{\nabla }f(x_n),\overline{x}-x_n⟩.\end{array}$$

Since g is convex and differentiable, we have for all $n\ge 1$

$$\begin{array}{c}\hfill 0\ge {\mathit{\beta }}_{n}g(x_n)+⟨{\mathit{\beta }}_n\mathrm{\nabla }g(x_n),\overline{x}-x_n⟩,\end{array}$$

which together imply that

$$\begin{array}{cc}\hfill f(\overline{x})\ge & {\mathrm{\Omega }}_n(x_n)+⟨\mathrm{\nabla }{\mathrm{\Omega }}_n(x_n),\overline{x}-x_n⟩\hfill \\ \hfill =& {\mathrm{\Omega }}_n(x_n)+〈\frac{y_n-x_{n+1}}{{\mathit{\lambda }}_n},\overline{x}-x_n〉\forall n\ge 1.\hfill \end{array}$$

From here we obtain for all $n\ge 1$ [see (6)]

$$\begin{array}{cc}\hfill 2{\mathit{\lambda }}_n\left[{\mathrm{\Omega }}_n(x_n)-f(\overline{x})\right]\le & 2⟨y_n-x_{n+1},x_n-\overline{x}⟩\hfill \\ \hfill =& \Vert x_{n+1}-x_n{\Vert }^2+{\mathit{\phi }}_n-{\mathit{\phi }}_{n+1}+\mathit{\alpha }({\mathit{\phi }}_n-{\mathit{\phi }}_{n-1})+\mathit{\alpha }{\Vert x_n-x_{n-1}\Vert }^2.\hfill \end{array}$$

Hence, by using the previous lemma, the required result holds. $\square$

The Opial Lemma that we recall below will play an important role in the proof of the main result of this paper.

Proposition 10

(Opial Lemma) Let H be a real Hilbert space, $C\subseteq H$ a nonempty set and ${\{x_n\}}_{n=1}^{\infty }$ a given sequence such that:

• (i)For every $z\in C,{lim}_{n\to +\infty }\Vert x_n-z\Vert$ exists.
• (ii)Every sequential weak cluster point of ${\{x_n\}}_{n=1}^{\infty }$ lies in C.

Then the sequence ${\{x_n\}}_{n=1}^{\infty }$ converges weakly to a point in C.

Theorem 11

• (i)The sequence ${\{x_n\}}_{n=1}^{\infty }$ converges weakly to a point in $\mathcal{S}$.
• (ii)The sequence ${(f(x_n))}_{n=1}^{\infty }$ converges to the optimal objective value of the optimization problem (1).

Proof

(i) According to Lemma 8, ${lim}_{n\to +\infty }\Vert x_n-\overline{x}\Vert$ exists for all $\overline{x}\in \mathcal{S}$. Let w be a sequential weak cluster point of ${\{x_n\}}_{n=1}^{\infty }$. Then there exists a subsequence ${\{x_{n_j}\}}_{j=1}^{\infty }$ of ${\{x_n\}}_{n=1}^{\infty }$ such that $x_{n_j}$ converges weakly to w as $j\to +\infty$. By Lemma 8, we have that $w\in \mathrm{argmin}g$. This means that in order to come to the conclusion it suffices to show that $f(w)\le f(x)$ for all $x\in \mathrm{argmin}g$. From Lemma 9, Lemma 8 and the fact that ${\sum }_{n=1}^{\infty }{\mathit{\lambda }}_n=+\infty$, it follows that ${lim}_{n\to \infty }[{\mathrm{\Omega }}_n(x_n)-f(\overline{x})]\le 0$ for all $\overline{x}\in \mathcal{S}$. Thus,

$$\begin{array}{c}\hfill f(w)\le \underset{j\to +\infty }{lim\; inf}f(x_{n_j})\le \underset{n\to +\infty }{lim}{\mathrm{\Omega }}_n(x_n)\le f(\overline{x})\forall \overline{x}\in \mathcal{S},\end{array}$$

which shows that $w\in \mathcal{S}$. Hence, thanks to Opial Lemma, ${\{x_n\}}_{n=1}^{\infty }$ converges weakly to a point in $\mathcal{S}$.

(ii) The statement follows easily from the above considerations. $\square$

In the end of this section we present some situations where Assumption 2 is verified.

Remark 12

Let $\mathit{\alpha }\in (0,1),c\in (1,+\infty ),q\in (0,1)$ and $\mathit{\gamma }\in \left(0,\frac{2}{L_g}\right)$ be arbitrarily chosen. We set

$$\begin{array}{cc}\hfill K:=& \frac{2}{\mathit{\alpha }}>0,\hfill \\ \hfill {\mathit{\beta }}_n:=& \frac{\mathit{\gamma }[L_f+2((1+\mathit{\alpha })K+c)]}{2-\mathit{\gamma }{L}_g}+(1-\mathit{\alpha })\mathit{\gamma }K{n}^q,\hfill \end{array}$$

and

$$\begin{array}{c}\hfill {\mathit{\lambda }}_n:=\frac{(1-\mathit{\alpha })\mathit{\gamma }}{{\mathit{\beta }}_n},\end{array}$$

for all $n\ge 1$.

• (i)Since ${\mathit{\beta }}_n\ge \frac{\mathit{\gamma }[L_f+2((1+\mathit{\alpha })K+c)]}{2-\mathit{\gamma }{L}_g}$, we have ${\mathit{\beta }}_n(2-\mathit{\gamma }{L}_g)\ge \mathit{\gamma }[L_f+2((1+\mathit{\alpha })K+c)]$, which implies that $\frac{L_n}{2}+\frac{\mathit{\alpha }-1}{{\mathit{\lambda }}_n}\le -\left(c+(1+\mathit{\alpha })K\right)$ for all $n\ge 1$.
• (ii)
  For all $n\ge 1$ it holds

  $$\begin{array}{c}\hfill {\mathit{\beta }}_{n+1}-{\mathit{\beta }}_n=(1-\mathit{\alpha })\mathit{\gamma }K[{(n+1)}^q-n^q]\le (1-\mathit{\alpha })\mathit{\gamma }K=K{\mathit{\lambda }}_{n+1}{\mathit{\beta }}_{n+1}.\end{array}$$
• (iii)It holds ${lim\; inf}_{n\to +\infty }{\mathit{\lambda }}_n{\mathit{\beta }}_n={lim\; inf}_{n\to +\infty }(1-\mathit{\alpha })\mathit{\gamma }>0$.
• (iv)
  For all $n\ge 1$ we have

  $$\begin{array}{c}\hfill \frac{1}{{\mathit{\lambda }}_{n+1}}-\frac{1}{{\mathit{\lambda }}_n}=\frac{1}{(1-\mathit{\alpha })\mathit{\gamma }}\left({\mathit{\beta }}_{n+1}-{\mathit{\beta }}_n\right)=K\left({(n+1)}^q-n^q\right)\le K=\frac{2}{\mathit{\alpha }}.\end{array}$$
• (v)Since $q\in (0,1)$, we have ${\sum }_{n=1}^{\infty }\frac{1}{{\mathit{\beta }}_n}=+\infty$, which implies that ${\sum }_{n=1}^{\infty }{\mathit{\lambda }}_n=+\infty$.
• (vi)
  Finally, as $g\le {\mathit{\delta }}_{\mathrm{argmin}g}$, we have $g^{\ast }\ge {({\mathit{\delta }}_{\mathrm{argmin}g})}^{\ast }={\mathit{\sigma }}_{\mathrm{argmin}g}$ and this implies that $g^{\ast }-{\mathit{\sigma }}_{\mathrm{argmin}g}\ge 0$. We present a situation where Assumption 2(III) holds and refer to [10] for further examples. For instance, if $g(x)\ge \frac{a}{2}{\mathrm{dist}}^2(x,\mathrm{argmin}g)$ where $a>0$, then $g^{\ast }(x)-{\mathit{\sigma }}_{\mathrm{argmin}g}(x)\le \frac{1}{2a}{\Vert x\Vert }^2$ for every $x\in H$. Thus, for $p\in \mathrm{ran}(N_{\mathrm{argmin}g})$, we have

  $$\begin{array}{c}\hfill {\mathit{\lambda }}_n{\mathit{\beta }}_n\left[g^{\ast }\left(\frac{p}{{\mathit{\beta }}_n}\right)-{\mathit{\sigma }}_{\mathrm{argmin}g}\left(\frac{p}{{\mathit{\beta }}_n}\right)\right]\le \frac{{\mathit{\lambda }}_n}{2a{\mathit{\beta }}_n}{\Vert p\Vert }^2.\end{array}$$

  Hence ${\sum }_{n=1}^{\infty }{\mathit{\lambda }}_n{\mathit{\beta }}_n\left[g^{\ast }\left(\frac{p}{{\mathit{\beta }}_n}\right)-{\mathit{\sigma }}_{\mathrm{argmin}g}\left(\frac{p}{{\mathit{\beta }}_n}\right)\right]$ converges, if ${\sum }_{n=1}^{\infty }\frac{{\mathit{\lambda }}_n}{{\mathit{\beta }}_n}$ converges or, equivalently, if ${\sum }_{n=1}^{\infty }\frac{1}{{\mathit{\beta }}_n^2}$ converges. This holds for the above choices of ${\{{\mathit{\beta }}_n\}}_{n=1}^{\infty }$ and ${\{{\mathit{\lambda }}_n\}}_{n=1}^{\infty }$ when $q\in \left(\frac{1}{2},1\right)$.

Numerical example: image classification via support vector machines

In this section we employ the algorithm proposed in this paper in the context of image classification via support vector machines.

Having a set of training data $a_i\in {\mathbb{R}}^n,i=1,\dots ,k,$ belonging to one of two given classes denoted by “$-1$” and “$+1$”, the aim is to construct by using this information a decision function given in the form of a separating hyperplane, which assigns every new data to one of the two classes with a misclassification rate as low as possible. In order to be able to handle the situation when a full separation is not possible, we make use of non-negative slack variables ${\mathit{\xi }}_i\ge 0,i=1,\dots ,k$; thus the goal will be to find $(s,r,\mathit{\xi })\in {\mathbb{R}}^n\times \mathbb{R}\times {\mathbb{R}}_{+}^k$ as optimal solution of the following optimization problem

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill \text{minimize}& \frac{1}{2}{\Vert s\Vert }^2+\frac{C}{2}{\Vert \mathit{\xi }\Vert }^2\hfill \\ \hfill \text{subject to}& d_i(a_i^{\top }s+r)\ge 1-{\mathit{\xi }}_i,\forall i=1,\dots ,k\hfill \\ \hfill & {\mathit{\xi }}_i\ge 0,\forall i=1,\dots ,k,\hfill \end{array}\end{array}$$

where for $i=1,\dots ,k,d_i$ is equal to $-1$ if $a_i$ belongs to the class “$-1$” and it is equal to $+1$, otherwise. Each new data $a\in {\mathbb{R}}^n$ will by assigned to one of the two classes by means of the resulting decision function $z(a)=a^{\top }s+r$, namely, a will be assigned to the class “$-1$”, if $z(a)<0$, and to the class “$+1$”, otherwise. For more theoretical insights in support vector machines we refer the reader to [29].

By making use of the matrix

$$\begin{array}{c}\hfill \mathbf{A}=\left[\begin{array}{cccccc}{d}_1{a}_1^{\top }& d_1& 1& 0& \cdots & 0\\ d_2{a}_2^{\top }& d_2& 0& 1& \cdots & 0\\ ⋮& ⋮& ⋮& ⋮& \ddots & ⋮\\ d_k{a}_k^{\top }& d_k& 0& 0& \cdots & 1\\ {\mathbf{0}}_{{\mathbb{R}}^n}^{\top }& 0& 1& 0& \cdots & 0\\ {\mathbf{0}}_{{\mathbb{R}}^n}^{\top }& 0& 0& 1& \cdots & 0\\ ⋮& ⋮& ⋮& ⋮& \ddots & ⋮\\ {\mathbf{0}}_{{\mathbb{R}}^n}^{\top }& 0& 0& 0& \cdots & 1\\ \end{array}\right]\in {\mathbb{R}}^{2k\times (n+1+k)}\end{array}$$

the problem under investigation can be written as

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill \text{minimize}& \frac{1}{2}{\Vert s\Vert }^2+\frac{C}{2}{\Vert \mathit{\xi }\Vert }^2\hfill \\ \hfill \text{subject to}& \mathbf{A}\left(\begin{array}{c}s\\ r\\ \mathit{\xi }\end{array}\right)-\left(\begin{array}{c}{\mathbf{1}}_{{\mathbb{R}}^k}\\ {\mathbf{0}}_{{\mathbb{R}}^k}\end{array}\right)\in {\mathbb{R}}_{+}^{2k}\hfill \end{array}\end{array}$$

or, equivalently,

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill \text{minimize}& \frac{1}{2}{\Vert s\Vert }^2+\frac{C}{2}{\Vert \mathit{\xi }\Vert }^2\hfill \\ \hfill \text{subject to}& \left(\begin{array}{c}s\\ r\\ \mathit{\xi }\end{array}\right)\in argmin\frac{1}{2}{\mathrm{dist}}^2\left(\mathbf{A}(·)-\left(\begin{array}{c}{\mathbf{1}}_{{\mathbb{R}}^k}\\ {\mathbf{0}}_{{\mathbb{R}}^k}\end{array}\right),{\mathbb{R}}_{+}^{2k}\right).\hfill \end{array}\end{array}$$

By considering $f:{\mathbb{R}}^n\times \mathbb{R}\times {\mathbb{R}}^k\to \mathbb{R}$ as $f\left(\begin{array}{c}s\\ r\\ \mathit{\xi }\end{array}\right):=\frac{1}{2}{\Vert s\Vert }^2+\frac{C}{2}{\Vert \mathit{\xi }\Vert }^2$, we have $\mathrm{\nabla }f\left(\begin{array}{c}s\\ r\\ \mathit{\xi }\end{array}\right)=\left(\begin{array}{c}s\\ 0\\ C\mathit{\xi }\end{array}\right)$ and notice that $\mathrm{\nabla }f$ is $max\{1,C\}$-Lipschitz continuous.

Further, for $g\left(\begin{array}{c}s\\ r\\ \mathit{\xi }\end{array}\right):=\frac{1}{2}{\mathrm{dist}}^2\left(\mathbf{A}\left(\begin{array}{c}s\\ r\\ \mathit{\xi }\end{array}\right)-\left(\begin{array}{c}{\mathbf{1}}_{{\mathbb{R}}^k}\\ {\mathbf{0}}_{{\mathbb{R}}^k}\end{array}\right),{\mathbb{R}}_{+}^{2k}\right)$, we have $\mathrm{\nabla }g\left(\begin{array}{c}s\hfill \\ r\hfill \\ \mathit{\xi }\hfill \end{array}\right)={\mathbf{A}}^{\top }\left(I-{\mathrm{proj}}_{{\mathbb{R}}_{+}^{2k}}\right)\left(\mathbf{A}\left(\begin{array}{c}s\\ r\\ \mathit{\xi }\end{array}\right)-\left(\begin{array}{c}{\mathbf{1}}_{{\mathbb{R}}^k}\\ {\mathbf{0}}_{{\mathbb{R}}^k}\end{array}\right)\right)$ and notice that $\mathrm{\nabla }g$ is ${\Vert \mathbf{A}\Vert }^2$-Lipschitz continuous, where ${\mathrm{proj}}_{{\mathbb{R}}_{+}^{2k}}$ denotes the projection operator on the set ${\mathbb{R}}_{+}^{2k}$.

For the numerical experiments we used a data set consisting of 6.000 training images and 2.060 test images of size $28\times 28$ taken from the website http://www.cs.nyu.edu/~roweis/data.html representing the handwritten digits 2 and 7, labeled by $-1$ and $+1$, respectively (see Fig. 1). We evaluated the quality of the resulting decision function on test data set by computing the percentage of misclassified images.

Fig. 1.

A sample of images belonging to the classes $-1$ and $+1$, respectively

We denote by $\mathcal{D}=\{(X_i,Y_i),i=1,\dots ,6.000\}\subset {\mathbb{R}}^{784}\times \{-1,+1\}$ the set of available training data consisting of 3.000 images in the class $-1$ and 3.000 images in the class $+1$. Due to numerical reasons each image has been vectorized and normalized. We tested in MATLAB different combinations of parameters chosen as in Remark 12 by running the algorithm for 3.000 iterations. A sample of misclassified images is shown in Fig. 2.

Fig. 2.

A sample of misclassified images

In Table 1 we present the misclassification rate in percentage for different choices for the parameters $\mathit{\alpha }\in (0,1)$ (we recall that in this case we take $K:=2/\mathit{\alpha }$) and $C>0$, while for $\mathit{\alpha }=0$ which corresponds to the noninertial version of the algorithm we consider different choices of the parameter $K>0$ and $C>0$. We observe that when combining $\mathit{\alpha }=0.1$ with each regularization parameters $C=5,10,100$ leads to the lowest misclassification rate with 2.1845%.

Table 1.

Misclassification rate in percentage for different choices for the parameters $\mathit{\alpha }$ and C when $c=2$ and $q=0.9$

$\mathit{\alpha }$	$C=0.1$	$C=1$	$C=2$	$C=5$	$C=10$	$C=100$
0.1	2.2330	2.2330	2.2330	2.1845	2.1845	2.1845
0.3	2.2330	2.2816	2.2816	2.2816	2.2816	2.2816
0.5	2.2330	2.2330	2.2330	2.2816	2.2816	2.3301
0.7	2.3786	2.3786	2.3786	2.3786	2.3786	2.3786
0.9	2.9126	2.9126	2.9126	2.9126	2.8641	2.8155
0 ($\text{K}=0.1$)	3.1068	3.0583	3.0583	2.9612	2.9612	2.7184
0 ($\text{K}=1$)	2.2816	2.2330	2.2330	2.2330	2.2330	2.2330
0 ($\text{K}=10$)	2.2816	2.2330	2.2330	2.2330	2.2330	2.2330
0 ($\text{K}=100$)	2.2330	2.2330	2.2330	2.2330	2.2330	2.2330
0 ($\text{K}=1000$)	2.2330	2.2330	2.2330	2.2330	2.2330	2.2330

In Table 2 we present the misclassification rate in percentage for different choices of the parameters $C>0$ and $c>1$. The lowest classification rate of $2.1845\%$ is obtained for each regularization parameter $C=5,10,100$.

Table 2.

Misclassification rate in percentage for different choices for the parameters C and $c>1$ when $\mathit{\alpha }=0.1$ and $q=0.9$

C	$c=1.1$	$c=2$	$c=5$	$c=10$	$c=100$
0.1	2.2330	2.2330	2.2330	2.2330	2.2330
1	2.2330	2.2330	2.2330	2.2330	2.2330
2	2.2330	2.2330	2.2330	2.2330	2.2330
5	2.1845	2.1845	2.1845	2.1845	2.1845
10	2.1845	2.1845	2.1845	2.1845	2.1845
100	2.1845	2.1845	2.1845	2.1845	2.1845

Finally, Table 3 shows the misclassification rate in percentage for different choices for the parameters $C>0$ and $q\in (1/2,1)$. The lowest classification rate of $2.1845\%$ is obtained when combining the value $q=0.9$ with each regularization parameter $C=5,10,100$.

Table 3.

Misclassification rate in percentage for different choices for the parameters C and $q\in (1/2,1)$ when $\mathit{\alpha }=0.1$ and $c=2$

C	$q=0.6$	$q=0.75$	$q=0.9$
0.1	2.2816	2.3301	2.2330
1	2.2330	2.2816	2.2330
2	2.2816	2.2816	2.2330
5	2.2330	2.2816	2.1845
10	2.2330	2.2816	2.1845
100	2.2330	2.2330	2.1845