Variable Smoothing for Convex Optimization Problems Using Stochastic Gradients

Abstract

We aim to solve a structured convex optimization problem, where a nonsmooth function is composed with a linear operator. When opting for full splitting schemes, usually, primal–dual type methods are employed as they are effective and also well studied. However, under the additional assumption of Lipschitz continuity of the nonsmooth function which is composed with the linear operator we can derive novel algorithms through regularization via the Moreau envelope. Furthermore, we tackle large scale problems by means of stochastic oracle calls, very similar to stochastic gradient techniques. Applications to total variational denoising and deblurring, and matrix factorization are provided.

Introduction

The problem at hand is the following structured convex optimization problem

$$\begin{array}{c}\hfill \underset{x\in \mathcal{H}}{min}f(x)+g(Kx),\end{array}$$ 1

for real Hilbert spaces $\mathcal{H}$ and $\mathcal{G}$, $f:\mathcal{H}\to \overline{\mathbb{R}}:=\mathbb{R}\cup \{\pm \infty \}$ a proper, convex and lower semicontinuous function, $g:\mathcal{G}\to \mathbb{R}$ a, possibly nonsmooth, convex and Lipschitz continuous function, and $K:\mathcal{H}\to \mathcal{G}$ a linear continuous operator.

Our aim will be to devise an algorithm for solving (1) following the full splitting paradigm (see [5, 6, 8, 9, 15, 17, 29]). In other words, we allow only proximal evaluations for simple nonsmooth functions, but no proximal evaluations for compositions with linear continuous operators, like, for instance, for $g\circ K$.

We will accomplish this feat by the means of a smoothing strategy, which, for the purpose of this paper, means, making use of the Moreau-Yosida approximation. The approach can be described as follows: we “smooth” g, i.e. we replace it by its Moreau envelope, and solve the resulting optimization problem by an accelerated proximal-gradient algorithm (see [3, 13, 21]). This approach is similar to those in [7, 10, 11, 20, 22], where a convergence rate of $\mathcal{O}(\frac{log(k)}{k})$ is proved. However, our techniques (for the deterministic case) resemble more the ones in [28], where an improved rate of $\mathcal{O}(\frac{1}{k})$ is shown, with the most notable difference to our work is that we use a simpler stepsize and treat the stochastic case.

The only other family of methods able to solve problems of type (1) are the so called primal–dual algorithms, first and foremost the primal–dual hybrid gradient (PDHG) introduced in [15]. In comparison, this method does not need the Lipschitz continuity of g in order to prove convergence. However, in this very general case, convergence rates can only be shown for the so-called restricted primal–dual gap function. In order to derive from here convergence rates for the primal objective function, either Lipschitz continuity of g or finite dimensionality of the problem plus the condition that g must have full domain are necessary (see, for instance, [5, Theorem 9]). This means, that for infinite dimensional problems the assumptions required by both, PDHG and our method, for deriving convergence rates for the primal objective function are in fact equal, but for finite dimensional problems the assumption of PDHG are weaker. In either case, however, we are able to prove these rates for the sequence of iterates ${(x_k)}_{k\ge 1}$ itself whereas PDHG only has them for the sequence of so-called ergodic iterates, i.e. ${(\frac{1}{k}{\sum }_{i=1}^k{x}_i)}_{k\ge 1}$, which is naturally undesirable as the averaging slows the convergence down. Furthermore, we do not show any convergence for the iterates as these are notoriously hard to obtain for accelerated method whereas PDHG gets these in the strongly convex setting via standard fixed point arguments (see e.g. [29]).

Furthermore, we will also consider the case where only a stochastic oracle of the proximal operator of g is available to us. This setup corresponds e.g. to the case where the objective function is given as

$$\begin{array}{c}\hfill \underset{x\in \mathcal{H}}{min}f(x)+\sum _{i=1}^m{g}_i(K_{i}x),\end{array}$$ 2

where, for $i=1,\cdots ,m$, ${\mathcal{G}}_i$ are real Hilbert spaces, $g_i:{\mathcal{G}}_i\to \mathbb{R}$ are convex and Lipschitz continuous functions and $K_i:\mathcal{H}\to {\mathcal{G}}_i$ are linear continuous operators, but the number of summands being large we wish to not compute all proximal operators of all $g_i,i=1,\cdots ,m$, for purpose of making iterations cheaper to compute.

For the finite sum case (2), there exist algorithms of similar spirit such as those in [14, 24]. Some algorithms do in fact deal with a similar setup of stochastic gradient like evaluations, see [26], but only for smooth terms in the objective function.

In Sect. 2 we will cover the preliminaries about the Moreau-Yosida envelope as well as useful identities and estimates connected to it. In Sect. 3 we will deal with the deterministic case and prove a convergence rate of $\mathcal{O}(\frac{1}{k})$ for the function values at the iterates. Next up, in Sect. 4, we will consider the stochastic case as described above and prove a convergence rate of $\mathcal{O}\left(\frac{log(k)}{\sqrt{k}}\right)$. Last but not least, we will look at some numerical examples in image processing in Sect. 5.

It is important to note that the proof for the deterministic setting differs surprisingly from the one for the stochastic setting. The technique for the stochastic setting is less refined in the sense that there is no coupling between the smoothing parameter and the extrapolation parameter. Where as this technique works also works for the deterministic setting it gives a worse convergence rate of $\mathcal{O}\left(\frac{logk}{k}\right)$. The tight coupling of the two sequences of parameters, however does not work in the proof of the stochastic algorithm as it does not allow for the particular choice of the smoothing parameters needed there.

Preliminaries

In the main problem (1), the nonsmooth function regularizer g is supposed to be Lipschitz continuous. This assumption is necessary to ensure our main convergence results, however, many of the preliminary lemmata of this section hold true similarly if the function is only assumed to be lower semicontinuous. We will point this out in every statement of this section individually.

Definition 2.1

For a proper, convex and lower semicontinuous function $g:\mathcal{H}\to \overline{\mathbb{R}}$, its convex conjugate is denoted by $g^{\ast }$ defined as a function from $\mathcal{H}$ to $\overline{\mathbb{R}}$, given by

$$\begin{array}{c}\hfill g^{\ast }(x):=\underset{p\in \mathcal{H}}{sup}\left\{〈x,,,p〉,-,g,(p)\right\}\forall x\in \mathcal{H}.\end{array}$$

As mentioned in the introduction, we want to smooth a nonsmooth function by considering its Moreau envelope. The next definition will clarify exactly what object we are talking about.

Definition 2.2

For a proper, convex and lower semicontinuous function $g:\mathcal{H}\to \overline{\mathbb{R}}$, its Moreau envelope with the parameter $\mu \ge 0$ is defined as a function from $\mathcal{H}$ to $\mathbb{R}$, given by

$$\begin{array}{c}\hfill {}^{{\mu }_{}}g(·):={\left(g^{\ast },+,\frac{\mu }{2},{\Vert ·\Vert }^2\right)}^{\ast }(·)=\underset{p\in \mathcal{H}}{sup}\left\{〈·,,,p〉,-,g^{\ast },(p),-,\frac{\mu }{2},{\Vert p\Vert }^2\right\}.\end{array}$$

From this definition, however, it is not completely evident that the Moreau envelope indeed fulfills its purpose in being a smooth representation of the original function. The next lemma will remedy this fact.

Lemma 2.1

(see [2, Proposition 12.29]) Let $g:\mathcal{H}\to \overline{\mathbb{R}}$ be a proper, convex and lower semicontinuous function and $\mu >0$. Then its Moreau envelope is Fréchet differentiable on $\mathcal{H}$. In particular, the gradient itself is given by

$$\begin{array}{c}\hfill \mathrm{\nabla }({}^{{\mu }_{}}g)(x)=\frac{1}{\mu }\left(x,-,{\text{prox}}_{\mu g}^{},\left(x\right)\right)={\text{prox}}_{\frac{1}{\mu }{g}^{\ast }}^{}\left(\frac{x}{\mu }\right)\forall x\in \mathcal{H}\end{array}$$

and is ${\mu }^{-1}$-Lipschitz continuous.

In particular, for all $\mu >0$, a gradient step with respect to the Moreau envelope corresponds to a proximal step

$$\begin{array}{c}\hfill x-\mu \mathrm{\nabla }({}^{{\mu }_{}}g)(x)={\text{prox}}_{\mu g}^{}\left(x\right)\forall x\in \mathcal{H}.\end{array}$$

The previous lemma establishes two things. Not only does it clarify the smoothness of the Moreau envelope, but it also gives a way of computing its gradient. Obviously, a smooth representation whose gradient we would not be able to compute would not be any good.

As mentioned in the introduction, we want to smooth the nonsmooth summand of the objective function which is composed with the linear operator as this can be considered the crux of problem (1). The function $g\circ K$ will be smoothed via considering instead ${}^{{\mu }_{}}g\circ K:\mathcal{H}\to \mathbb{R}$. Clearly, by the chain rule, this function is continuously differentiable with gradient given for every $x\in \mathcal{H}$ by

$$\begin{array}{c}\hfill \mathrm{\nabla }\left({}^{{\mu }_{}},g,\circ ,K\right)(x)=K^{\ast }\mathrm{\nabla }\left({}^{{\mu }_{}},g\right)(Kx)=\frac{1}{\mu }{K}^{\ast }\left(K,x,-,{\text{prox}}_{\mu g}^{},\left(K,x\right)\right)=K^{\ast }{\text{prox}}_{\frac{1}{\mu }{g}^{\ast }}^{}\left(\frac{\mathit{Kx}}{\mu }\right),\end{array}$$

and is thus Lipschitz continuous with Lipschitz constant $\frac{{\Vert K\Vert }^2}{\mu }$, where $\Vert K\Vert$ denotes the operator norm of K.

Lipschitz continuity will play an integral role in our investigations, as can be seen by the following lemmas.

Lemma 2.2

(see [4, Proposition 4.4.6]) Let $g:\mathcal{H}\to \mathbb{R}$ be a convex and $L_g$-Lipschitz continuous function. Then, the domain of its Fenchel conjugate is bounded, i.e.

$$\begin{array}{c}\hfill dom{g}^{\ast }\subseteq B(0,L_g),\end{array}$$

where $B(0,L_g)$ denotes the open ball with radius $L_g$ around the origin.

The Moreau envelope even preserves the Lipschitzness of the original function.

Lemma 2.3

(see [18, Lemma 2.1]) Let $g:\mathcal{H}\to \mathbb{R}$ be a convex and $L_g$-Lipschitz continuous function. Then its Moreau envelope ${}^{{\mu }_{}}g$ is $L_g$-Lipschitz as well, i.e.

$$\begin{array}{c}\hfill |{}^{{\mu }_{}}g(x)-{}^{{\mu }_{}}g(y)|\le L_g\Vert x-y\Vert \forall x,y\in \mathcal{H}.\end{array}$$

Proof

We observe that for all $x\in \mathcal{H}$

$$\begin{array}{c}\hfill \mathrm{\nabla }{}^{{\mu }_{}}g(x)\in \partial g({\text{prox}}_{\mu g}^{}\left(x\right)).\end{array}$$

Therefore we can bound the gradient norm

$$\begin{array}{c}\hfill \Vert \mathrm{\nabla }{}^{{\mu }_{}}g(x)\Vert \le sup\{\Vert v\Vert :y\in \mathcal{H},v\in \partial g(y)\}\le L_g\forall x\in \mathcal{H},\end{array}$$ 3

where we used in the last step that the Lipschitz continuity of g. The statement follows from the mean-value theorem. $\square$

The following lemmata are not new, but we provide proofs anyways in order to remain self-contained and to shed insight on how to use the Moreau envelope for the interested reader.

Lemma 2.4

(see [28, Lemma 10 (a)]) Let $g:\mathcal{H}\to \overline{\mathbb{R}}$ be proper, convex and lower semicontinuous. The maximizing argument in the definition of the Moreau-Yosida envelope is given by its gradient, i.e. for $\mu >0$ it holds that

$$\begin{array}{c}\hfill \underset{p\in \mathcal{H}}{\arg \max }\left\{〈·,,,p〉,-,g^{\ast },(p),-,\frac{\mu }{2},{\Vert p\Vert }^2\right\}=\mathrm{\nabla }{}^{{\mu }_{}}g(·).\end{array}$$

Proof

Let $x\in \mathcal{H}$ be fixed. It holds

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill \underset{p\in \mathcal{H}}{\arg \max }\left\{⟨x,p⟩,-,g^{\ast },(p),-,\frac{\mu }{2},{\Vert p\Vert }^2\right\}=& \underset{p\in \mathcal{H}}{\arg \max }\left\{-,\frac{1}{2\mu },{\Vert x\Vert }^2,+,⟨x,p⟩,-,\frac{\mu }{2},{\Vert p\Vert }^2,-,g^{\ast },(p)\right\}\hfill \\ \hfill =& \underset{p\in \mathcal{H}}{\arg \max }\left\{-,\frac{\mu }{2},{∥\frac{x}{\mu },-,p∥}^2,-,g^{\ast },(p)\right\}\hfill \\ \hfill =& \underset{p\in \mathcal{H}}{\arg \min }\left\{g^{\ast },(p),+,\frac{\mu }{2},{∥\frac{x}{\mu },-,p∥}^2\right\}\hfill \\ \hfill =& {\text{prox}}_{\frac{1}{\mu }{g}^{\ast }}^{}\left(\frac{x}{\mu }\right)\hfill \end{array}\end{array}$$

and the conclusion follows by using Lemma 2.1. $\square$

Lemma 2.5

(see [28, Lemma 10 (a)]) For a proper, convex and lower semicontinuous function $g:\mathcal{H}\to \overline{\mathbb{R}}$ and every $x\in \mathcal{H}$ we can consider the mapping from $(0,+\infty )$ to $\mathbb{R}$ given by

$$\begin{array}{c}\hfill \mu \mapsto {}^{{\mu }_{}}g(x).\end{array}$$ 4

This mapping is convex and differentiable and its derivative is given by

$$\begin{array}{c}\hfill \frac{\partial }{\partial \mu }{}^{{\mu }_{}}g(x)=-\frac{1}{2}{\Vert \mathrm{\nabla }{}^{{\mu }_{}}g(x)\Vert }^2\forall x\in \mathcal{H}\forall \mu \in (0,+\infty ).\end{array}$$

Proof

Let $x\in \mathcal{H}$ be fixed. From the definition of the Moreau-Yosida envelope we can see that the mapping given in (4) is a pointwise supremum of functions which are linear in $\mu$. It is therefore convex. Furthermore, since the objective function is strongly concave, this supremum is uniquely attained at $\mathrm{\nabla }{}^{{\mu }_{}}g(x)=\underset{p\in \mathcal{H}}{\arg \max }\left\{〈x,,,p〉,-,g^{\ast },(p),-,\frac{\mu }{2},{\Vert p\Vert }^2\right\}$. According to the Danskin Theorem, the function $\mu \mapsto {}^{{\mu }_{}}g(x)$ is differentiable and its gradient is given by

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill \frac{\partial }{\partial \mu }{}^{{\mu }_{}}g(x)=& \frac{\partial }{\partial \mu }\underset{p\in \mathcal{H}}{sup}\left\{⟨x,p⟩,-,g^{\ast },(p),-,\frac{\mu }{2},{\Vert p\Vert }^2\right\}\hfill \\ \hfill =& -\frac{1}{2}{\Vert \mathrm{\nabla }{}^{{\mu }_{}}g(x)\Vert }^2\forall \mu \in (0,+\infty ).\hfill \end{array}\end{array}$$

$\square$

Lemma 2.6

( [28, Lemma 10 (b)]) Let $g:\mathcal{H}\to \overline{\mathbb{R}}$ be proper, convex and lower semicontinuous. For ${\mu }_1,{\mu }_2>0$ and every $x\in \mathcal{H}$ it holds

$$\begin{array}{c}\hfill {}^{{\mu }_1}g(x)\le {}^{{\mu }_2}g(x)+({\mu }_2-{\mu }_1)\frac{1}{2}{\Vert \mathrm{\nabla }{}^{{\mu }_1}g(x)\Vert }^2.\end{array}$$ 5

If g is additionally $L_g$-Lipschitz and if ${\mu }_2\ge {\mu }_1>0$, then

$$\begin{array}{c}\hfill {}^{{\mu }_2}g(x)\le {}^{{\mu }_1}g(x)\le {}^{{\mu }_2}g(x)+({\mu }_2-{\mu }_1)\frac{L_g^2}{2}.\end{array}$$ 6

Proof

Let $x\in \mathcal{H}$ be fixed. Via Lemma 2.5 we know that the map $\mu \mapsto {}^{{\mu }_{}}g(x)$ is convex and differentiable. We can therefore use the gradient inequality to deduce that

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill {}^{{\mu }_2}g(x)\ge & {}^{{\mu }_1}g(x)+({\mu }_2-{\mu }_1)\left(\frac{\partial }{\partial \mu },{}^{{\mu }_{}},g,(x),{|}_{\mu ={\mu }_1}\right)\hfill \\ \hfill =& {}^{{\mu }_1}g(x)-({\mu }_2-{\mu }_1)\frac{1}{2}{\Vert \mathrm{\nabla }{}^{{\mu }_1}g(x)\Vert }^2,\hfill \end{array}\end{array}$$

which is exactly the first statement of the lemma. The first inequality of (6) follows directly from the definition of the Moreau envelope and the second one from (5) and (3). $\square$

By applying a limiting argument it is easy to see that (6) implies that for any $\mu >0$

$$\begin{array}{c}\hfill {}^{{\mu }_{}}g(x)\le g(x)\le {}^{{\mu }_{}}g(x)+\mu \frac{L_g^2}{2},\end{array}$$ 7

which shows that the Moreau envelope is always a lower approximation the original function.

Lemma 2.7

(see [28, Lemma 10 (c)]) Let $g:\mathcal{H}\to \overline{\mathbb{R}}$ be proper, convex and lower semicontinuous. Then, for $\mu >0$ and every $x,y\in \mathcal{H}$ we have that

$$\begin{array}{c}\hfill {}^{{\mu }_{}}g(x)+〈\mathrm{\nabla },{}^{{\mu }_{}},g,(x),,,y,-,x〉\le g(y)-\frac{\mu }{2}{\Vert \mathrm{\nabla }{}^{{\mu }_{}}g(x)\Vert }^2.\end{array}$$

Proof

Using Lemma 2.4 and the definition of the Moreau-Yosida envelope we get that

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill {}^{{\mu }_{}}g(x)+〈\mathrm{\nabla },{}^{{\mu }_{}},g,(x),,,y,-,x〉=& 〈x,,,\mathrm{\nabla },{}^{{\mu }_{}},g,(x)〉-g^{\ast }(\mathrm{\nabla }{}^{{\mu }_{}}g(x))\hfill \\ \hfill & -\frac{\mu }{2}{\Vert \mathrm{\nabla }{}^{{\mu }_{}}g(x)\Vert }^2+〈\mathrm{\nabla },{}^{{\mu }_{}},g,(x),,,y,-,x〉\hfill \\ \hfill =& 〈\mathrm{\nabla },{}^{{\mu }_{}},g,(x),,,y〉-g^{\ast }(\mathrm{\nabla }{}^{{\mu }_{}}g(x))-\frac{\mu }{2}{\Vert \mathrm{\nabla }{}^{{\mu }_{}}g(x)\Vert }^2\hfill \\ \hfill \le & \underset{p\in \mathcal{H}}{sup}\{〈p,,,y〉-g^{\ast }(p)\}-\frac{\mu }{2}{\Vert \mathrm{\nabla }{}^{{\mu }_{}}g(x)\Vert }^2\hfill \\ \hfill =& g(y)-\frac{\mu }{2}{\Vert \mathrm{\nabla }{}^{{\mu }_{}}g(x)\Vert }^2.\hfill \end{array}\end{array}$$

$\square$

In the convergence proof of Lemma 3.3 we will need the inequality in the above lemma at the points Kx and Ky, namely

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill g(Ky)-\frac{\mu }{2}{\Vert \mathrm{\nabla }{}^{{\mu }_{}}g(Kx)\Vert }^2& \ge {}^{{\mu }_{}}g(Kx)+〈\mathrm{\nabla },{}^{{\mu }_{}},g,(Kx),,,K,y,-,K,x〉\hfill \\ \hfill & ={}^{{\mu }_{}}g(Kx)+〈K^{\ast },\mathrm{\nabla },{}^{{\mu }_{}},g,(Kx),,,y,-,x〉\hfill \\ \hfill & ={}^{{\mu }_{}}g(Kx)+〈\mathrm{\nabla },({}^{{\mu }_{}}g\circ K),(x),,,y,-,x〉\forall x,y\in \mathcal{H}.\hfill \end{array}\end{array}$$ 8

The following lemma is a standard result for convex and Fréchet differentiable functions.

Lemma 2.8

(see [23]) For a convex and Fréchet differentiable function $h:\mathcal{H}\to \mathbb{R}$ with $L_h$-Lipschitz continuous gradient we have that

$$\begin{array}{c}\hfill h(x)+〈\mathrm{\nabla },h,(,x,),,,y,-,x〉\le h(y)-\frac{1}{2L_h}{\Vert \mathrm{\nabla }h(x)-\mathrm{\nabla }h(y)\Vert }^2\forall x,y\in \mathcal{H}.\end{array}$$

By applying Lemma 2.8 with ${}^{{\mu }_{}}g$, Kx and Ky instead of h, x and y respectively, we obtain

$$\begin{array}{c}\hfill {}^{{\mu }_{}}g(Kx)+〈\mathrm{\nabla },({}^{{\mu }_{}}g\circ K),(x),,,y,-,x〉\le {}^{{\mu }_{}}g(Ky)-\frac{\mu }{2}{\Vert \mathrm{\nabla }{}^{{\mu }_{}}g(Kx)-\mathrm{\nabla }{}^{{\mu }_{}}g(Ky)\Vert }^2\forall x,y\in \mathcal{H}.\end{array}$$ 9

The following technical result will be used in the proof of the convergence statement.

Lemma 2.9

For $\alpha \in (0,1)$ and every $x,y\in \mathcal{H}$ we have that

$$\begin{array}{c}\hfill {(1-\alpha )\Vert x-y\Vert }^2+{\alpha \Vert y\Vert }^2\ge \alpha (1-\alpha ){\Vert x\Vert }^2.\end{array}$$

Deterministic Method

Problem 3.1

The problem at hand reads

$$\begin{array}{c}\hfill \underset{x\in \mathcal{H}}{min}F(x):=f(x)+g(Kx),\end{array}$$

for a proper, convex and lower semicontinuous function $f:\mathcal{H}\to \overline{\mathbb{R}}$, a convex and $L_g$-Lipschitz continuous $(L_g>0)$ function $g:\mathcal{G}\to \mathbb{R}$, and a nonzero linear continuous operator $K:\mathcal{H}\to \mathcal{G}$.

The idea of the algorithm which we propose to solve (1) is to smooth g and then to solve the resulting problem by means of an accelerated proximal-gradient method.

Algorithm 3.1

(Variable Accelerated SmooThing (VAST)) Let $y_0=x_0\in \mathcal{H},{({\mu }_k)}_{k\ge 0}\subseteq (0,+\infty )$, and ${(t_k)}_{k\ge 1}$ a sequence of real numbers with $t_1=1$ and $t_k\ge 1$ for every $k\ge 2$. Consider the following iterative scheme

$$\begin{array}{c}\hfill (\forall k\ge 1)\left.⌊\begin{array}{c}{L}_k=\frac{{\Vert K\Vert }^2}{{\mu }_k}\hfill \\ {\gamma }_k=\frac{1}{L_k}\hfill \\ x_k={\text{prox}}_{{\gamma }_{k}f}^{}\left(y_{k-1},-,{\gamma }_k,K^{\ast },{\text{prox}}_{\frac{1}{{\mu }_k}{g}^{\ast }}^{},\left(\frac{K{y}_{k-1}}{{\mu }_k}\right)\right)\hfill \\ y_k=x_k+\frac{t_k-1}{t_{k+1}}(x_k-x_{k-1}).\hfill \end{array}\right)\end{array}$$

Remark 3.1

The assumption $t_1=1$ can be removed but guarantees easier computation and is also in line with classical choices of ${(t_k)}_{k\ge 1}$ in [13, 21].

Remark 3.2

The sequence ${(u_k)}_{k\ge 1}$ given by

$$\begin{array}{c}\hfill u_k:=x_{k-1}+t_k(x_k-x_{k-1})\forall k\ge 1,\end{array}$$

despite not appearing in the algorithm, will feature a prominent role in the convergence proof. Due to the convention $t_1=1$ we have that

$$\begin{array}{c}\hfill u_1:=x_0+t_1(x_1-x_0)=x_1.\end{array}$$

We also denote

$$\begin{array}{c}\hfill F^k=f+{}^{{\mu }_k}g\circ K\forall k\ge 0.\end{array}$$

The next theorem is the main result of this section and it will play a fundamental role when proving a convergence rate of $\mathcal{O}(\frac{1}{k})$ for the sequence ${(F(x_k))}_{k\ge 0}$.

Theorem 3.1

Consider the setup of Problem 3.1 and let ${(x_k)}_{k\ge 0}$ and ${(y_k)}_{k\ge 0}$ be the sequences generated by Algorithm 3.1. Assume that for every $k\ge 1$

$$\begin{array}{c}\hfill {\mu }_k-{\mu }_{k+1}-\frac{{\mu }_{k+1}}{t_{k+1}}\le 0\end{array}$$

and

$$\begin{array}{c}\hfill \left(1,-,\frac{1}{t_{k+1}}\right){\gamma }_{k+1}{t}_{k+1}^2={\gamma }_k{t}_k^2.\end{array}$$

Then, for every optimal solution $x^{\ast }$ of Problem 3.1, it holds

$$\begin{array}{c}\hfill F(x_N)-F(x^{\ast })\le \frac{\Vert x_0-x^{\ast }{\Vert }^2}{2{\gamma }_N{t}_N^2}+{\mu }_N\frac{L_g^2}{2}\forall N\ge 1.\end{array}$$

The proof of this result relies on several partial results which we will prove as follows.

Lemma 3.1

The following statement holds for every $z\in \mathcal{H}$ and every $k\ge 0$

$$\begin{array}{cc}& F^{k+1}(x_{k+1})+\frac{1}{2{\gamma }_{k+1}}{\Vert x_{k+1}-z\Vert }^2\hfill \\ \hfill & \le f(z)+{}^{{\mu }_{k+1}}g(K{y}_k)+〈\mathrm{\nabla },({}^{{\mu }_{k+1}}g\circ K),(y_k),,,z,-,y_k〉+\frac{1}{2{\gamma }_{k+1}}{\Vert z-y_k\Vert }^2.\hfill \end{array}$$

Proof

Let $k\ge 0$ be fixed. Since, by the definition of the proximal map, $x_{k+1}$ is the minimizer of a $\frac{1}{{\gamma }_{k+1}}$-strongly convex function we know that for every $z\in \mathcal{H}$

$$\begin{array}{cc}& f(x_{k+1})+{}^{{\mu }_{k+1}}g(K{y}_k)+〈\mathrm{\nabla },({}^{{\mu }_{k+1}}g\circ K),(y_k),,,x_{k+1},-,y_k〉+\frac{1}{2{\gamma }_{k+1}}{\Vert x_{k+1}-y_k\Vert }^2+\hfill \\ \hfill & \frac{1}{2{\gamma }_{k+1}}{\Vert x_{k+1}-z\Vert }^2\hfill \\ \hfill & \le f(z)+{}^{{\mu }_{k+1}}g(K{y}_k)+〈\mathrm{\nabla },({}^{{\mu }_{k+1}}g\circ K),(y_k),,,z,-,y_k〉+\frac{1}{2{\gamma }_{k+1}}{\Vert z-y_k\Vert }^2.\hfill \end{array}$$

Next we use the $L_{k+1}$-smoothness of ${}^{{\mu }_{k+1}}g\circ K$ and the fact that $\frac{1}{{\gamma }_{k+1}}=L_{k+1}$ to deduce

$$\begin{array}{cc}& f(x_{k+1})+{}^{{\mu }_{k+1}}g(K{x}_{k+1})+\frac{1}{2{\gamma }_{k+1}}{\Vert x_{k+1}-z\Vert }^2\hfill \\ \hfill & \le f(z)+{}^{{\mu }_{k+1}}g(K{y}_k)+〈\mathrm{\nabla },({}^{{\mu }_{k+1}}g\circ K),(y_k),,,z,-,y_k〉+\frac{1}{2{\gamma }_{k+1}}{\Vert z-y_k\Vert }^2.\hfill \end{array}$$

$\square$

Lemma 3.2

Let $x^{\ast }$ be an optimal solution of Problem 3.1. Then it holds

$$\begin{array}{c}\hfill {\gamma }_1(F^1(x_1)-F(x^{\ast }))+\frac{1}{2}\Vert u_1-x^{\ast }{\Vert }^2\le \frac{1}{2}{\Vert x^{\ast }-x_0\Vert }^2.\end{array}$$

Proof

We use the gradient inequality to deduce that for every $z\in \mathcal{H}$ and every $k\ge 0$

$$\begin{array}{c}\hfill {}^{{\mu }_{k+1}}g(K{y}_k)+〈\mathrm{\nabla },({}^{{\mu }_{k+1}}g\circ K),(y_k),,,z,-,y_k〉\le {}^{{\mu }_{k+1}}g(Kz)\le g(Kz)\end{array}$$

and plug this into the statement of Lemma 3.1 to conclude that

$$\begin{array}{c}\hfill F^{k+1}(x_{k+1})+\frac{1}{2{\gamma }_{k+1}}\Vert x_{k+1}{-z\Vert }^2\le F(z)+\frac{1}{2{\gamma }_{k+1}}{\Vert z-y_k\Vert }^2.\end{array}$$

For $k=0$ we get that

$$\begin{array}{c}\hfill F^1(x_1)+\frac{1}{2{\gamma }_1}\Vert x_1-x^{\ast }{\Vert }^2\le F(x^{\ast })+\frac{1}{2{\gamma }_1}{\Vert x^{\ast }-y_0\Vert }^2.\end{array}$$

Now we us the fact that $u_1=x_1$ and $y_0=x_0$ to obtain the conclusion. $\square$

Lemma 3.3

Let $x^{\ast }$ be an optimal solution of Problem 3.1. The following descent-type inequality holds for every $k\ge 0$

$$\begin{array}{c}\hfill \begin{array}{cc}& F^{k+1}(x_{k+1})-F(x^{\ast })+\frac{\Vert u_{k+1}-x^{\ast }{\Vert }^2}{2{\gamma }_{k+1}{t}_{k+1}^2}\le \left(1,-,\frac{1}{t_{k+1}}\right)\left(F^k,(x_k),-,F,(x^{\ast })\right)+\frac{\Vert u_k-x^{\ast }{\Vert }^2}{2{\gamma }_{k+1}{t}_{k+1}^2}\hfill \\ \hfill & +\left(1,-,\frac{1}{t_{k+1}}\right)\left({\mu }_k,-,{\mu }_{k+1},-,\frac{{\mu }_{k+1}}{t_{k+1}}\right){\Vert \mathrm{\nabla }{}^{{\mu }_{k+1}}g(K{x}_k)\Vert }^2.\hfill \end{array}\end{array}$$

Proof

Let $k\ge 0$ be fixed. We apply Lemma 3.1 with $z:=\left(1,-,\frac{1}{t_{k+1}}\right)x_k+\frac{1}{t_{k+1}}{x}^{\ast }$ to deduce that

$$\begin{array}{c}\hfill \begin{array}{cc}& F^{k+1}(x_{k+1})+\frac{\Vert u_{k+1}-x^{\ast }{\Vert }^2}{2{\gamma }_{k+1}{t}_{k+1}^2}\le f\left(\left(1,-,\frac{1}{t_{k+1}}\right),x_k,+,\frac{1}{t_{k+1}},x^{\ast }\right)+{}^{{\mu }_{k+1}}g(K{y}_k)\hfill \\ \hfill & +\left(1,-,\frac{1}{t_{k+1}}\right)〈\mathrm{\nabla },({}^{{\mu }_{k+1}}g\circ K),(y_k),,,x_k,-,y_k〉\hfill \\ \hfill & +\frac{1}{t_{k+1}}〈\mathrm{\nabla },({}^{{\mu }_{k+1}}g\circ K),(y_k),,,x^{\ast },-,y_k〉+\frac{1}{2{\gamma }_{k+1}{t}_{k+1}^2}{\Vert u_k-x^{\ast }\Vert }^2.\hfill \end{array}\end{array}$$

Using the convexity of f gives

$$\begin{array}{c}\hfill \begin{array}{cc}& F^{k+1}(x_{k+1})+\frac{\Vert u_{k+1}-x^{\ast }{\Vert }^2}{2{\gamma }_{k+1}{t}_{k+1}^2}\le \left(1,-,\frac{1}{t_{k+1}}\right)f(x_k)+\frac{1}{t_{k+1}}f(x^{\ast })\hfill \\ \hfill & +\left(1,-,\frac{1}{t_{k+1}}\right){}^{{\mu }_{k+1}}g(K{y}_k)+\left(1,-,\frac{1}{t_{k+1}}\right)〈\mathrm{\nabla },({}^{{\mu }_{k+1}}g\circ K),(y_k),,,x_k,-,y_k〉\hfill \\ \hfill & +\frac{1}{t_{k+1}}{}^{{\mu }_{k+1}}g(K{y}_k)+\frac{1}{t_{k+1}}〈\mathrm{\nabla },({}^{{\mu }_{k+1}}g\circ K),(y_k),,,x^{\ast },-,y_k〉+\frac{\Vert u_k-x^{\ast }{\Vert }^2}{2{\gamma }_{k+1}{t}_{k+1}^2}.\hfill \end{array}\end{array}$$ 10

Now, we use (8) to deduce that

$$\begin{array}{c}\hfill \begin{array}{cc}& \frac{1}{t_{k+1}}{}^{{\mu }_{k+1}}g(K{y}_k)+\frac{1}{t_{k+1}}〈\mathrm{\nabla },({}^{{\mu }_{k+1}}g\circ K),(y_k),,,x^{\ast },-,y_k〉\hfill \\ \hfill & \le \frac{1}{t_{k+1}}g(K{x}^{\ast })-\frac{1}{t_{k+1}}\frac{{\mu }_{k+1}}{2}{\Vert \mathrm{\nabla }{}^{{\mu }_{k+1}}g(K{y}_k)\Vert }^2\hfill \end{array}\end{array}$$ 11

and (9) to conclude that

$$\begin{array}{c}\hfill \begin{array}{cc}& \left(1,-,\frac{1}{t_{k+1}}\right){}^{{\mu }_{k+1}}g(K{y}_k)+\left(1,-,\frac{1}{t_{k+1}}\right)〈\mathrm{\nabla },({}^{{\mu }_{k+1}}g\circ K),(y_k),,,x_k,-,y_k〉\hfill \\ \hfill & \le \left(1,-,\frac{1}{t_{k+1}}\right){}^{{\mu }_{k+1}}g(K{x}_k)-\left(1,-,\frac{1}{t_{k+1}}\right)\frac{{\mu }_{k+1}}{2}{\Vert \mathrm{\nabla }{}^{{\mu }_{k+1}}g(K{y}_k)-\mathrm{\nabla }{}^{{\mu }_{k+1}}g(K{x}_k)\Vert }^2.\hfill \end{array}\end{array}$$ 12

Combining (10), (11) and (12) gives

$$\begin{array}{c}\hfill \begin{array}{cc}& F^{k+1}(x_{k+1})+\frac{\Vert u_{k+1}-x^{\ast }{\Vert }^2}{2{\gamma }_{k+1}{t}_{k+1}^2}\le \left(1,-,\frac{1}{t_{k+1}}\right){}^{{\mu }_{k+1}}g(K{x}_k)+\left(1,-,\frac{1}{t_{k+1}}\right)f(x_k)\hfill \\ \hfill & +\frac{1}{t_{k+1}}g(K{x}^{\ast })+\frac{1}{t_{k+1}}f(x^{\ast })-\left(1,-,\frac{1}{t_{k+1}}\right)\frac{{\mu }_{k+1}}{2}{\Vert \mathrm{\nabla }{}^{{\mu }_{k+1}}g(K{y}_k)-\mathrm{\nabla }{}^{{\mu }_{k+1}}g(K{x}_k)\Vert }^2\hfill \\ \hfill & -\frac{1}{t_{k+1}}\frac{{\mu }_{k+1}}{2}{\Vert \mathrm{\nabla }{}^{{\mu }_{k+1}}g(K{y}_k)\Vert }^2+\frac{\Vert u_k-x^{\ast }{\Vert }^2}{2{\gamma }_{k+1}{t}_{k+1}^2}.\hfill \end{array}\end{array}$$

The first term on the right hand side is ${}^{{\mu }_{k+1}}g(K{x}_k)$ but we would like it to be ${}^{{\mu }_k}g(K{x}_k)$. Therefore we use Lemma 2.6 to deduce that

$$\begin{array}{c}\hfill \begin{array}{cc}& F^{k+1}(x_{k+1})+\frac{\Vert u_{k+1}-x^{\ast }{\Vert }^2}{2{\gamma }_{k+1}{t}_{k+1}^2}\le \left(1,-,\frac{1}{t_{k+1}}\right){}^{{\mu }_k}g(K{x}_k)+\left(1,-,\frac{1}{t_{k+1}}\right)f(x_k)\hfill \\ \hfill & +\frac{1}{t_{k+1}}g(K{x}^{\ast })+\frac{1}{t_{k+1}}f(x^{\ast })+\left(1,-,\frac{1}{t_{k+1}}\right)({\mu }_k-{\mu }_{k+1})\frac{1}{2}{\Vert \mathrm{\nabla }{}^{{\mu }_{k+1}}g(K{x}_k)\Vert }^2\hfill \\ \hfill & -\left(1,-,\frac{1}{t_{k+1}}\right)\frac{{\mu }_{k+1}}{2}{\Vert \mathrm{\nabla }{}^{{\mu }_{k+1}}g(K{y}_k)-\mathrm{\nabla }{}^{{\mu }_{k+1}}g(K{x}_k)\Vert }^2\hfill \\ \hfill & -\frac{1}{t_{k+1}}\frac{{\mu }_{k+1}}{2}{\Vert \mathrm{\nabla }{}^{{\mu }_{k+1}}g(K{y}_k)\Vert }^2+\frac{\Vert u_k-x^{\ast }{\Vert }^2}{2{\gamma }_{k+1}{t}_{k+1}^2}.\hfill \end{array}\end{array}$$ 13

Next up we want to estimate all the norms of gradients by using Lemma 2.9 which says that

$$\begin{array}{c}\hfill \begin{array}{cc}& \left(1,-,\frac{1}{t_{k+1}}\right)\Vert \mathrm{\nabla }{}^{{\mu }_{k+1}}g(K{y}_k)-\mathrm{\nabla }{}^{{\mu }_{k+1}}g(K{x}_k){\Vert }^2+\frac{1}{t_{k+1}}{\Vert \mathrm{\nabla }{}^{{\mu }_{k+1}}g(K{y}_k)\Vert }^2\hfill \\ \hfill & \ge \left(1,-,\frac{1}{t_{k+1}}\right)\frac{1}{t_{k+1}}{\Vert \mathrm{\nabla }{}^{{\mu }_{k+1}}g(K{x}_k)\Vert }^2.\hfill \end{array}\end{array}$$ 14

Combining (13) and (14) gives

$$\begin{array}{c}\hfill \begin{array}{cc}& F^{k+1}(x_{k+1})+\frac{\Vert u_{k+1}-x^{\ast }{\Vert }^2}{2{\gamma }_{k+1}{t}_{k+1}^2}\le \left(1,-,\frac{1}{t_{k+1}}\right){}^{{\mu }_k}g(K{x}_k)+\left(1,-,\frac{1}{t_{k+1}}\right)f(x_k)\hfill \\ \hfill & +\frac{1}{t_{k+1}}g(K{x}^{\ast })+\frac{1}{t_{k+1}}f(x^{\ast })+\left(1,-,\frac{1}{t_{k+1}}\right)({\mu }_k-{\mu }_{k+1})\frac{1}{2}{\Vert \mathrm{\nabla }{}^{{\mu }_{k+1}}g(K{x}_k)\Vert }^2\hfill \\ \hfill & -\frac{{\mu }_{k+1}}{2}\left(1,-,\frac{1}{t_{k+1}}\right)\frac{1}{t_{k+1}}{\Vert \mathrm{\nabla }{}^{{\mu }_{k+1}}g(K{x}_k)\Vert }^2+\frac{\Vert u_k-x^{\ast }{\Vert }^2}{2{\gamma }_{k+1}{t}_{k+1}^2}.\hfill \end{array}\end{array}$$

Now we combine the two terms containing $\Vert \mathrm{\nabla }{}^{{\mu }_{k+1}}g(K{x}_k){\Vert }^2$ and get that

$$\begin{array}{c}\hfill \begin{array}{cc}& F^{k+1}(x_{k+1})+\frac{\Vert u_{k+1}-x^{\ast }{\Vert }^2}{2{\gamma }_{k+1}{t}_{k+1}^2}\le \left(1,-,\frac{1}{t_{k+1}}\right){}^{{\mu }_k}g(K{x}_k)+\left(1,-,\frac{1}{t_{k+1}}\right)f(x_k)\hfill \\ \hfill & +\frac{1}{t_{k+1}}g(K{x}^{\ast })+\frac{1}{t_{k+1}}f(x^{\ast })+\frac{\Vert u_k-x^{\ast }{\Vert }^2}{2{\gamma }_{k+1}{t}_{k+1}^2}\hfill \\ \hfill & +\left(1,-,\frac{1}{t_{k+1}}\right)\left({\mu }_k,-,{\mu }_{k+1},-,\frac{{\mu }_{k+1}}{t_{k+1}}\right)\frac{1}{2}{\Vert \mathrm{\nabla }{}^{{\mu }_{k+1}}g(K{x}_k)\Vert }^2.\hfill \end{array}\end{array}$$

By subtracting $F(x^{\ast })=f(x^{\ast })+g(K{x}^{\ast })$ on both sides we finally obtain

$$\begin{array}{c}\hfill \begin{array}{cc}& F^{k+1}(x_{k+1})-F(x^{\ast })+\frac{\Vert u_{k+1}-x^{\ast }{\Vert }^2}{2{\gamma }_{k+1}{t}_{k+1}^2}\le \left(1,-,\frac{1}{t_{k+1}}\right)\left(F^k,(x_k),-,F,(x^{\ast })\right)+\frac{\Vert u_k-x^{\ast }{\Vert }^2}{2{\gamma }_{k+1}{t}_{k+1}^2}\hfill \\ \hfill & +\left(1,-,\frac{1}{t_{k+1}}\right)\left({\mu }_k,-,{\mu }_{k+1},-,\frac{{\mu }_{k+1}}{t_{k+1}}\right)\frac{1}{2}{\Vert \mathrm{\nabla }{}^{{\mu }_{k+1}}g(K{x}_k)\Vert }^2.\hfill \end{array}\end{array}$$

$\square$

Now we are in the position to prove Theorem 3.1.

Proof of Theorem 3.1

We start with the statement of Lemma 3.3 and use the assumption that

$$\begin{array}{c}\hfill {\mu }_k-{\mu }_{k+1}-\frac{{\mu }_{k+1}}{t_{k+1}}\le 0\end{array}$$

to make the last term in the inequality disappear for every $k\ge 0$

$$\begin{array}{c}\hfill \begin{array}{cc}& F^{k+1}(x_{k+1})-F(x^{\ast })+\frac{\Vert u_{k+1}-x^{\ast }{\Vert }^2}{2{\gamma }_{k+1}{t}_{k+1}^2}\hfill \\ \hfill & \le \left(1,-,\frac{1}{t_{k+1}}\right)\left(F^k,(x_k),-,F,(x^{\ast })\right)+\frac{\Vert u_k-x^{\ast }{\Vert }^2}{2{\gamma }_{k+1}{t}_{k+1}^2}.\hfill \end{array}\end{array}$$

Now we use the assumption that

$$\begin{array}{c}\hfill \left(1,-,\frac{1}{t_{k+1}}\right){\gamma }_{k+1}{t}_{k+1}^2={\gamma }_k{t}_k^2\end{array}$$

to get that for every $k\ge 0$

$$\begin{array}{cc}& {\gamma }_{k+1}{t}_{k+1}^2(F^{k+1}(x_{k+1})-F(x^{\ast }))+\frac{\Vert u_{k+1}-x^{\ast }{\Vert }^2}{2}\hfill \\ \hfill & \le {\gamma }_k{t}_k^2(F^k(x_k)-F(x^{\ast }))+\frac{\Vert u_k-x^{\ast }{\Vert }^2}{2}.\hfill \end{array}$$ 15

Let $N\ge 2$. Summing (15) from $k=1$ to $N-1$ and getting rid of the nonnegative term $\Vert u_N-x^{\ast }{\Vert }^2$ gives

$$\begin{array}{c}\hfill {\gamma }_N{t}_N^2(F^N(x_N)-F(x^{\ast }))\le {\gamma }_1(F^1(x_1)-F(x^{\ast }))+\frac{\Vert u_1-x^{\ast }{\Vert }^2}{2}\forall N\ge 2.\end{array}$$

Since $t_1=1$, the above inequality is fulfilled also for $N=1$. Using Lemma 3.2 shows that

$$\begin{array}{c}\hfill F^N(x_N)-F(x^{\ast })\le \frac{\Vert x_0-x^{\ast }{\Vert }^2}{{\gamma }_N{t}_N^2}\forall N\ge 1.\end{array}$$

The above inequality, however, is still in terms of the smoothed objective function. In order to go to the actual objective function we apply (7) and deduce that

$$\begin{array}{c}\hfill F(x_N)-F(x^{\ast })\le F^N(x_N)-F(x^{\ast })+{\mu }_N\frac{L_g^2}{2}\le \frac{\Vert x_0-x^{\ast }{\Vert }^2}{2{\gamma }_N{t}_N^2}+{\mu }_N\frac{L_g^2}{2}\forall N\ge 1.\end{array}$$

$\square$

Corollary 3.1

By choosing the parameters ${({\mu }_k)}_{k\ge 1},{(t_k)}_{k\ge 1},{({\gamma }_k)}_{k\ge 1}$ in the following way,

$$\begin{array}{c}\hfill t_1=1,{\mu }_1=b{\Vert K\Vert }^2,\text{for}b>0,\end{array}$$

and for every $k\ge 1$

$$\begin{array}{c}\hfill t_{k+1}:=\sqrt{t_k^2+2t_k},{\mu }_{k+1}:={\mu }_k\frac{t_k^2}{t_{k+1}^2-t_{k+1}},{\gamma }_k:=\frac{{\mu }_k}{{\Vert K\Vert }^2},\end{array}$$ 16

they fulfill

$$\begin{array}{c}\hfill {\mu }_k-{\mu }_{k+1}-\frac{{\mu }_{k+1}}{t_{k+1}}\le 0\end{array}$$ 17

and

$$\begin{array}{c}\hfill \left(1,-,\frac{1}{t_{k+1}}\right){\gamma }_{k+1}{t}_{k+1}^2={\gamma }_k{t}_k^2\end{array}$$ 18

For this choice of the parameters we have that

$$\begin{array}{c}\hfill F(x_N)-F(x^{\ast })\le \frac{\Vert x_0-x^{\ast }{\Vert }^2}{b(N+1)}+\frac{b{L}_g^2{\Vert K\Vert }^2}{(N+1)}exp\left(\frac{4{\pi }^2}{6}\right)\forall N\ge 1.\end{array}$$

Proof

Since ${\gamma }_k$ and ${\mu }_k$ are a scalar multiple of each other (18) is equivalent to

$$\begin{array}{c}\hfill \left(1,-,\frac{1}{t_{k+1}}\right){\mu }_{k+1}{t}_{k+1}^2={\mu }_k{t}_k^2\forall k\ge 1\end{array}$$

and further to (by taking into account that $t_{k+1}>1$ for every $k\ge 1$)

$$\begin{array}{c}\hfill {\mu }_{k+1}={\mu }_k\frac{t_k^2}{t_{k+1}^2}\frac{t_{k+1}}{t_{k+1}-1}={\mu }_k\frac{t_k^2}{t_{k+1}^2-t_{k+1}}\forall k\ge 1.\end{array}$$ 19

Our update choice in (16) for the sequence ${({\mu }_k)}_{k\ge 1}$ is exactly chosen in such a way that it satisfies this. Plugging (19) into (17) gives for every $k\ge 1$ the condition

$$\begin{array}{c}\hfill 1\le \left(1,+,\frac{1}{t_{k+1}}\right)\frac{t_k^2}{t_{k+1}^2}\frac{t_{k+1}}{t_{k+1}-1}=\frac{t_k^2}{t_{k+1}^2}\frac{t_{k+1}+1}{t_{k+1}-1},\end{array}$$

which is equivalent to

$$\begin{array}{c}\hfill 0\ge t_{k+1}^3-t_{k+1}^2-t_k^2{t}_{k+1}-t_k^2\end{array}$$

and further to

$$\begin{array}{c}\hfill t_{k+1}^2+t_k^2\ge t_{k+1}\left(t_{k+1}^2,-,t_k^2\right).\end{array}$$

Plugging in $t_{k+1}=\sqrt{t_k^2+2t_k}$ we get that this equivalent to

$$\begin{array}{c}\hfill t_{k+1}^2+t_k^2\ge t_{k+1}2t_k\forall k\ge 1,\end{array}$$

which is evidently fulfilled. Thus, the choices in (16) are indeed feasible for our algorithm.

Now we want to prove the claimed convergence rates. Via induction we show that

$$\begin{array}{c}\hfill \frac{k+1}{2}\le t_k\le k\forall k\ge 1.\end{array}$$ 20

Evidently, this holds for $t_1=1$. Assuming that (20) holds for $k\ge 1$, we easily see that

$$\begin{array}{c}\hfill t_{k+1}=\sqrt{t_k^2+2t_k}\le \sqrt{k^2+2k}\le \sqrt{k^2+2k+1}=k+1\end{array}$$

and, on the other hand,

$$\begin{array}{c}\hfill t_{k+1}=\sqrt{t_k^2+2t_k}\ge \sqrt{\frac{{(k+1)}^2}{4}+k+1}=\frac{1}{2}\sqrt{k^2+6k+5}\ge \frac{1}{2}\sqrt{k^2+4k+4}=\frac{k+2}{2}.\end{array}$$

In the following we prove a similar estimate for the sequence ${({\mu }_k)}_{k\ge 1}$. To this end we show, again by induction, the following recursion for every $k\ge 2$

$$\begin{array}{c}\hfill {\mu }_k={\mu }_1\frac{{\prod }_{j=1}^{k-1}{t}_j}{{\prod }_{j=2}^k(t_j-1)}\frac{1}{t_k}.\end{array}$$ 21

For $k=2$ this follows from the definition (19). Assume now that (21) holds for $k\ge 2$. From here we have that

$$\begin{array}{c}\hfill {\mu }_{k+1}={\mu }_k\frac{t_k^2}{t_{k+1}(t_{k+1}-1)}={\mu }_1\frac{{\prod }_{j=1}^{k-1}{t}_j}{{\prod }_{j=2}^k(t_j-1)}\frac{1}{t_k}\frac{t_k^2}{t_{k+1}(t_{k+1}-1)}={\mu }_1\frac{{\prod }_{j=1}^k{t}_j}{{\prod }_{j=2}^{k+1}(t_j-1)}\frac{1}{t_{k+1}}.\end{array}$$

Using (21) together with (20) we can check that for every $k\ge 1$

$$\begin{array}{c}\hfill \begin{array}{c}\hfill {\mu }_{k+1}={\mu }_1\frac{{\prod }_{j=1}^k{t}_j}{{\prod }_{j=2}^{k+1}(t_j-1)}\frac{1}{t_{k+1}}=\frac{{\mu }_1}{t_{k+1}}\prod _{j=1}^k\frac{t_j}{(t_{j+1}-1)}\ge \frac{{\mu }_1}{t_{k+1}}=b{\Vert K\Vert }^2\frac{1}{t_{k+1}},\end{array}\end{array}$$ 22

where we used in the last step the fact that $t_{k+1}\le t_k+1$.

The last thing to check is the fact that ${\mu }_k$ goes to zero like $\frac{1}{k}$. First we check that for every $k\ge 1$

$$\begin{array}{c}\hfill \frac{t_k}{t_{k+1}-1}\le 1+\frac{1}{t_{k+1}(t_{k+1}-1)}.\end{array}$$ 23

This can be seen via

$$\begin{array}{c}\hfill (t_k+1)t_{k+1}\le {(t_k+1)}^2=t_{k+1}^2+1\forall k\ge 1.\end{array}$$

By bringing $t_{k+1}$ to the other side we get that

$$\begin{array}{c}\hfill t_{k+1}{t}_k\le t_{k+1}^2-t_{k+1}+1,\end{array}$$

from which we can deduce (23) by dividing by $t_{k+1}^2-t_{k+1}$.

We plug in the estimate (23) in (21) and get for every $k\ge 2$

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill {\mu }_k& ={\mu }_1\frac{{\prod }_{j=1}^{k-1}{t}_j}{{\prod }_{j=1}^{k-1}(t_{j+1}-1)}\frac{1}{t_k}\hfill \\ \hfill & \le {\mu }_1\prod _{j=1}^{k-1}\left(1,+,\frac{1}{t_{j+1}(t_{j+1}-1)}\right)\frac{1}{t_k}\le {\mu }_1\prod _{j=1}^{k-1}\left(1,+,\frac{4}{(j+2)j}\right)\frac{1}{t_k}\hfill \\ \hfill & \le {\mu }_1\prod _{j=1}^{k-1}\left(1,+,\frac{4}{j^2}\right)\frac{1}{t_k}\le {\mu }_{1}exp\left(\frac{{\pi }^{2}4}{6}\right)\frac{1}{t_k}=b{\Vert K\Vert }^{2}exp\left(\frac{{\pi }^{2}4}{6}\right)\frac{1}{t_k}.\hfill \end{array}\end{array}$$

With the above inequalities we can to deduce the claimed convergence rates. First note that from Theorem 3.1 we have

$$\begin{array}{c}\hfill F(x_N)-F(x^{\ast })\le \frac{\Vert x_0-x^{\ast }{\Vert }^2}{2{\gamma }_N{t}_N^2}+{\mu }_N\frac{L_g^2}{2}\forall N\ge 1.\end{array}$$

Now, in order to obtain the desired conclusion, we use the above estimates and deduce for every $N\ge 1$

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill \frac{\Vert x_0-x^{\ast }{\Vert }^2}{2{\gamma }_N{t}_N^2}+{\mu }_N\frac{L_g^2}{2}& \le \frac{\Vert x_0-x^{\ast }{\Vert }^2}{2b{t}_N}+\frac{b{L}_g^2{\Vert K\Vert }^2}{2t_N}exp\left(\frac{4{\pi }^2}{6}\right)\hfill \\ \hfill & \le \frac{\Vert x_0-x^{\ast }{\Vert }^2}{b(N+1)}+\frac{b{L}_g^2{\Vert K\Vert }^2}{(N+1)}exp\left(\frac{4{\pi }^2}{6}\right),\hfill \end{array}\end{array}$$

where we used that

$$\begin{array}{c}\hfill {\gamma }_N{t}_N=\frac{{\mu }_N{t}_N}{{\Vert K\Vert }^2}\ge b,\end{array}$$

as shown in (22). $\square$

Remark 3.3

Consider the choice (see [21])

$$\begin{array}{c}\hfill t_1=1,t_{k+1}=\frac{1+\sqrt{1+4t_k^2}}{2}\forall k\ge 1\end{array}$$

and

$$\begin{array}{c}\hfill {\mu }_1=b{\Vert K\Vert }^2,\text{for}b>0.\end{array}$$

Since

$$\begin{array}{c}\hfill t_k^2=t_{k+1}^2-t_{k+1}\forall k\ge 1,\end{array}$$

we see that in this setting we have to choose

$$\begin{array}{c}\hfill {\mu }_k=b{\Vert K\Vert }^2\text{and}{\gamma }_k=b\forall k\ge 1.\end{array}$$

Thus, the sequence of optimal function values ${(F(x_N))}_{N\ge 1}$ approaches a ${b\Vert K\Vert }^2\frac{L_g}{2}$-approximation of the optimal objective value $F(x^{\ast })$ with a convergence rate of $\mathcal{O}(\frac{1}{N^2})$, i.e.

$$\begin{array}{c}\hfill F(x_N)-F(x^{\ast })\le 2\frac{\Vert x_0-x^{\ast }{\Vert }^2}{b{(N+1)}^2}+b\frac{{\Vert K\Vert }^2{L}_g^2}{2}\forall N\ge 1.\end{array}$$

Stochastic Method

Problem 4.1

The problem is the same as in the deterministic case

$$\begin{array}{c}\hfill \underset{x\in \mathcal{H}}{min}f(x)+g(Kx)\end{array}$$

other than the fact that at each iteration we are only given a stochastic estimator of the quantity

$$\begin{array}{c}\hfill \mathrm{\nabla }({}^{{\mu }_k}g\circ K)(·)=K^{\ast }{\text{prox}}_{\frac{1}{{\mu }_k}{g}^{\ast }}^{}\left(\frac{1}{{\mu }_k},K,·\right)\forall k\ge 1.\end{array}$$

Remark 4.1

See Algorithm 4.3 for a setting where such an estimator is easily computed.

For the stochastic quantities arising in this section we will use the following notation. For every $k\ge 0$, we denote by $\sigma (x_0,\cdots ,x_k)$ the smallest $\sigma$-algebra generated by the family of random variables $\{x_0,\cdots ,x_k\}$ and by ${\mathbb{E}}_k(·):=\mathbb{E}(·|\sigma (x_0,\cdots ,x_k))$ the conditional expectation with respect to this $\sigma$-algebra.

Algorithm 4.1

(stochastic Variable Accelerated SmooThing (sVAST)) Let $y_0=x_0\in \mathcal{H},{({\mu }_k)}_{k\ge 1}$ a sequence of positive and nonincreasing real numbers, and ${(t_k)}_{k\ge 1}$ a sequence of real numbers with $t_1=1$ and $t_k\ge 1$ for every $k\ge 2$. Consider the following iterative scheme

$$\begin{array}{c}\hfill (\forall k\ge 1)\left.⌊\begin{array}{c}{L}_k=\frac{{\Vert K\Vert }^2}{{\mu }_k}\hfill \\ {\gamma }_k=\frac{1}{L_k}\hfill \\ x_k={\text{prox}}_{{\gamma }_{k}f}^{}\left(y_{k-1},-,{\gamma }_k,{\xi }_{k-1}\right)\hfill \\ y_k=x_k+\frac{t_k-1}{t_{k+1}}(x_k-x_{k-1}),\hfill \end{array}\right)\end{array}$$

where we make the standard assumptions about our gradient estimator of being unbiased, i.e.

$$\begin{array}{c}\hfill {\mathbb{E}}_k({\xi }_k)=\mathrm{\nabla }({}^{{\mu }_{k+1}}g\circ K)(y_k),\end{array}$$

and having bounded variance

$$\begin{array}{c}\hfill {\mathbb{E}}_k\left(\Vert ,{\xi }_k,-,\mathrm{\nabla },({}^{{\mu }_{k+1}}g\circ K),(y_k),{\Vert }^2\right)\le {\sigma }^2\end{array}$$

for every $k\ge 0$.

Note that we use the same notations as in the deterministic case

$$\begin{array}{c}\hfill u_k:=x_{k-1}+t_k(x_k-x_{k-1})\text{and}{F}^k(·):=f+{}^{{\mu }_k}g\circ K\forall k\ge 1.\end{array}$$

Lemma 4.1

The following statement holds for every (deterministic) $z\in \mathcal{H}$ and every $k\ge 0$

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill {\mathbb{E}}_k\left(F^{k+1},(x_{k+1}),+,\frac{\Vert x_{k+1}{-z\Vert }^2}{2{\gamma }_{k+1}}\right)\le & F^{k+1}(z)+\frac{\Vert z-y_k{\Vert }^2}{2{\gamma }_{k+1}}\hfill \\ \hfill & +{\gamma }_{k+1}\left({\sigma }^2,+,\frac{{\Vert K\Vert }^2{L}_g^2}{2}\right)\hfill \end{array}\end{array}$$

Proof

Here we have to proceed a little bit different from Lemma 3.1. Namely, we have to treat the gradient step and the proximal step differently. For this purpose we define the auxiliary variable

$$\begin{array}{c}\hfill z_k:=y_{k-1}-{\gamma }_k{\xi }_{k-1}\forall k\ge 1.\end{array}$$

Let $k\ge 1$ be fixed. From the gradient step we get

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill \Vert z-z_k{\Vert }^2& =\Vert y_{k-1}-{\gamma }_k{\xi }_{k-1}{-z\Vert }^2\hfill \\ \hfill & =\Vert y_{k-1}{-z\Vert }^2-2{\gamma }_k〈{\xi }_{k-1},,,y_{k-1},-,z〉+{\gamma }_k^2{\Vert {\xi }_{k-1}\Vert }^2.\hfill \end{array}\end{array}$$

Taking the conditional expectation gives

$$\begin{array}{c}\hfill \begin{array}{c}\hfill {\mathbb{E}}_{k-1}\left(\Vert z-,z_k,{\Vert }^2\right)={\Vert y_{k-1}-z\Vert }^2-2{\gamma }_k〈\mathrm{\nabla },({}^{{\mu }_k}g\circ K),(y_{k-1}),,,y_{k-1},-,z〉+{\gamma }_k^2{\mathbb{E}}_{k-1}\left(\Vert ,{\xi }_{k-1},{\Vert }^2\right).\end{array}\end{array}$$

Using the gradient inequality we deduce

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill {\mathbb{E}}_{k-1}\left(\Vert z-,z_k,{\Vert }^2\right)\le & \Vert y_{k-1}{-z\Vert }^2-2{\gamma }_k(({}^{{\mu }_k}g\circ K)(y_{k-1})-({}^{{\mu }_k}g\circ K)(z))\hfill \\ \hfill & +{\gamma }_k^2{\mathbb{E}}_{k-1}\left(\Vert ,{\xi }_{k-1},{\Vert }^2\right)\hfill \end{array}\end{array}$$

and therefore

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill {\gamma }_k({}^{{\mu }_k}g\circ K)(y_{k-1})+\frac{1}{2}{\mathbb{E}}_{k-1}\left(\Vert z-,z_k,{\Vert }^2\right)\le & \frac{1}{2}{\Vert y_{k-1}-z\Vert }^2+{\gamma }_k({}^{{\mu }_k}g\circ K)(z)\hfill \\ \hfill & +\frac{{\gamma }_k^2}{2}{\mathbb{E}}_{k-1}\left(\Vert ,{\xi }_{k-1},{\Vert }^2\right).\hfill \end{array}\end{array}$$ 24

Also from the smoothness of $({}^{{\mu }_k}g\circ K)$ we deduce via the Descent Lemma that

$$\begin{array}{c}\hfill \begin{array}{c}\hfill {}^{{\mu }_k}g(K{z}_k)\le {}^{{\mu }_k}g(K{y}_{k-1})+〈\mathrm{\nabla },({}^{{\mu }_k}g\circ K),(y_{k-1}),,,z_k,-,y_{k-1}〉+\frac{L_k}{2}{\Vert z_k-y_{k-1}\Vert }^2.\end{array}\end{array}$$

Plugging in the definition of $z_k$ and using the fact that $L_k=\frac{1}{{\gamma }_k}$ we get

$$\begin{array}{c}\hfill \begin{array}{c}\hfill {}^{{\mu }_k}g(K{z}_k)\le {}^{{\mu }_k}g(K{y}_{k-1})-{\gamma }_k〈\mathrm{\nabla },({}^{{\mu }_k}g\circ K),(y_{k-1}),,,{\xi }_{k-1}〉+\frac{{\gamma }_k}{2}{\Vert {\xi }_{k-1}\Vert }^2.\end{array}\end{array}$$

Now we take the conditional expectation to deduce that

$$\begin{array}{c}\hfill \begin{array}{c}\hfill {\mathbb{E}}_{k-1}({}^{{\mu }_k}g(K{z}_k))\le {}^{{\mu }_k}g(K{y}_{k-1})-{\gamma }_k{∥\mathrm{\nabla },({}^{{\mu }_k}g\circ K),(y_{k-1})∥}^2+\frac{{\gamma }_k}{2}{\mathbb{E}}_{k-1}\left(\Vert ,{\xi }_{k-1},{\Vert }^2\right).\end{array}\end{array}$$ 25

Multiplying (25) by ${\gamma }_k$ and adding it to (24) gives

$$\begin{array}{cc}& {\gamma }_k{\mathbb{E}}_{k-1}\left({}^{{\mu }_k},g,(K{z}_k)\right)+\frac{1}{2}{\mathbb{E}}_{k-1}\left(\Vert z-,z_k,{\Vert }^2\right)\hfill \\ \hfill & \le {\gamma }_k{}^{{\mu }_k}g(Kz)+\frac{1}{2}{\Vert y_{k-1}-z\Vert }^2-{\gamma }_k^2{∥\mathrm{\nabla },({}^{{\mu }_k}g\circ K),(y_{k-1})∥}^2+{\gamma }_k^2{\mathbb{E}}_{k-1}\left(\Vert ,{\xi }_{k-1},{\Vert }^2\right).\hfill \end{array}$$

Now we use the assumption about the bounded variance to deduce that

$$\begin{array}{c}\hfill \begin{array}{c}\hfill {\gamma }_k{\mathbb{E}}_{k-1}\left({}^{{\mu }_k},g,(K{z}_k)\right)+\frac{1}{2}{\mathbb{E}}_{k-1}\left(\Vert z-,z_k,{\Vert }^2\right)\le {\gamma }_k{}^{{\mu }_k}g(Kz)+\frac{1}{2}{\Vert y_{k-1}-z\Vert }^2+{\gamma }_k^2{\sigma }^2.\end{array}\end{array}$$ 26

Next up for the proximal step we deduce

$$\begin{array}{c}\hfill f(x_k)+\frac{1}{2{\gamma }_k}\Vert x_k-z_k{\Vert }^2+\frac{1}{2{\gamma }_k}\Vert x_k{-z\Vert }^2\le f(z)+\frac{1}{2{\gamma }_k}{\Vert z-z_k\Vert }^2.\end{array}$$ 27

Taking the conditional expectation and combining (26) and (27) we get

$$\begin{array}{c}\hfill \begin{array}{cc}& {\mathbb{E}}_{k-1}\left({\gamma }_k,({}^{{\mu }_k}g(K{z}_k)+f(x_k)),+,\frac{1}{2},\Vert ,x_k,-,z_k,{\Vert }^2,+,\frac{1}{2},{\Vert x_k-z\Vert }^2\right)\hfill \\ \hfill & \le {\gamma }_k{F}^k(z)+\frac{1}{2}{\Vert y_{k-1}-z\Vert }^2+{\gamma }_k^2{\sigma }^2.\hfill \end{array}\end{array}$$

From here, using now Lemma 2.3, we get that

$$\begin{array}{c}\hfill \begin{array}{cc}& {\mathbb{E}}_{k-1}\left({\gamma }_k,F^k,(x_k),-,{\gamma }_k,L_g,\Vert K\Vert \Vert ,x_k,-,z_k,\Vert +,\frac{1}{2},\Vert ,x_k,-,z_k,{\Vert }^2,+,\frac{1}{2},{\Vert x_k-z\Vert }^2\right)\hfill \\ \hfill & \le {\gamma }_k{F}^k(z)+\frac{1}{2}{\Vert y_{k-1}-z\Vert }^2+{\gamma }_k^2{\sigma }^2.\hfill \end{array}\end{array}$$

Now we use

$$\begin{array}{c}\hfill -\frac{1}{2}{\gamma }_k^2{L}_g^2{\Vert K\Vert }^2\le \frac{1}{2}\Vert x_k-z_k{\Vert }^2-{\gamma }_k{L}_g\Vert K\Vert \Vert x_k-z_k\Vert \end{array}$$

to obtain that

$$\begin{array}{c}\hfill \begin{array}{cc}& {\mathbb{E}}_{k-1}\left({\gamma }_k,F^k,(x_k),+,\frac{1}{2},{\Vert x_k-z\Vert }^2\right)\hfill \\ \hfill & \le {\gamma }_k{F}^k(z)+\frac{1}{2}{\Vert y_{k-1}-z\Vert }^2+{\gamma }_k^2{\sigma }^2+\frac{1}{2}{\gamma }_k^2{L}_g^2{\Vert K\Vert }^2.\hfill \end{array}\end{array}$$

$\square$

Lemma 4.2

Let $x^{\ast }$ be an optimal solution of Problem 4.1. Then it holds

$$\begin{array}{c}\hfill \begin{array}{c}\hfill \mathbb{E}\left({\gamma }_1,(F^1(x_1)-F^1(x^{\ast }))\right)+\frac{1}{2}\Vert u_1-x^{\ast }{\Vert }^2\le \frac{1}{2}{\Vert x_0-x^{\ast }\Vert }^2+{\gamma }_1^2{\sigma }^2+\frac{1}{2}{\gamma }_1^2{L}_g^2{\Vert K\Vert }^2.\end{array}\end{array}$$

Proof

Applying the previous lemma with $k=0$ and $z=x^{\ast }$, we get that

$$\begin{array}{c}\hfill \begin{array}{c}\hfill \mathbb{E}\left({\gamma }_1,F^1,(x_1),+,\frac{1}{2},{\Vert x_1-x^{\ast }\Vert }^2\right)\le {\gamma }_1{F}^1(x^{\ast })+\frac{1}{2}{\Vert y_0-x^{\ast }\Vert }^2+{\gamma }_1^2{\sigma }^2+\frac{1}{2}{\gamma }_1^2{L}_g^2{\Vert K\Vert }^2.\end{array}\end{array}$$

Therefore, using the fact that $y_0=x_0$ and $u_1=x_1$,

$$\begin{array}{c}\hfill \begin{array}{c}\hfill \mathbb{E}\left({\gamma }_1,(F^1(x_1)-F^1(x^{\ast })),+,\frac{1}{2},{\Vert u_1-x^{\ast }\Vert }^2\right)\le \frac{1}{2}{\Vert x_0-x^{\ast }\Vert }^2+{\gamma }_1^2{\sigma }^2+\frac{1}{2}{\gamma }_1^2{L}_g^2{\Vert K\Vert }^2,\end{array}\end{array}$$

which finishes the proof. $\square$

Theorem 4.1

Consider the setup of Problem 4.1 and let ${(x_k)}_{k\ge 0}$ and ${(y_k)}_{k\ge 0}$ denote the sequences generated by Algorithm 4.1. Assume that for all $k\ge 1$

$$\begin{array}{c}\hfill {\rho }_{k+1}:=t_k^2-t_{k+1}^2+t_{k+1}\ge 0.\end{array}$$

Then, for every optimal solution $x^{\ast }$ of Problem 4.1, it holds

$$\begin{array}{cc}\hfill \mathbb{E}\left(F,(x_N),-,F,(x^{\ast })\right)\le & \frac{1}{{\gamma }_N{t}_N^2}\frac{1}{2}{\Vert x_0-x^{\ast }\Vert }^2+\frac{1}{{\gamma }_N{t}_N^2}\frac{{\Vert K\Vert }^2{L}_g^2}{2}\sum _{k=1}^N{\gamma }_k^2(t_k+{\rho }_k)\hfill \\ \hfill & +\frac{1}{{\gamma }_N{t}_N^2}\left({\sigma }^2,+,\frac{{\Vert K\Vert }^2{L}_g^2}{2}\right)\sum _{k=1}^N{t}_k^2{\gamma }_k^2\forall N\ge 1.\hfill \end{array}$$

Proof of Theorem 4.1

Let $k\ge 0$ be fixed. Lemma 4.1 for $z:=\left(1,-,\frac{1}{t_{k+1}}\right)x_k+\frac{1}{t_{k+1}}{x}^{\ast }$ gives

$$\begin{array}{cc}& {\mathbb{E}}_k\left(F^{k+1},(x_{k+1}),+,\frac{1}{2{\gamma }_{k+1}},{∥\frac{1}{t_{k+1}},u_{k+1},-,\frac{1}{t_{k+1}},x^{\ast }∥}^2\right)\hfill \\ \hfill & \le F^{k+1}\left(\left(1,-,\frac{1}{t_{k+1}}\right),x_k,+,\frac{1}{t_{k+1}},x^{\ast }\right)+\frac{1}{2{\gamma }_{k+1}}{∥\frac{1}{t_{k+1}},x^{\ast },-,\frac{1}{t_{k+1}},u_k∥}^2\hfill \\ \hfill & +{\gamma }_{k+1}\left({\sigma }^2,+,\frac{{\Vert K\Vert }^2{L}_g^2}{2}\right).\hfill \end{array}$$

From here and from the convexity of $F^{k+1}$ follows

$$\begin{array}{cc}& {\mathbb{E}}_k\left(F^{k+1},(x_{k+1}),-,F^{k+1},(x^{\ast })\right)-\left(1,-,\frac{1}{t_{k+1}}\right)(F^{k+1}(x_k)-F^{k+1}(x^{\ast }))\hfill \\ \hfill & \le \frac{\Vert u_k-x^{\ast }{\Vert }^2}{2{\gamma }_{k+1}{t}_{k+1}^2}-{\mathbb{E}}_k\left(\frac{\Vert u_{k+1}-x^{\ast }{\Vert }^2}{2{\gamma }_{k+1}{t}_{k+1}^2}\right)+{\gamma }_{k+1}\left({\sigma }^2,+,\frac{{\Vert K\Vert }^2{L}_g^2}{2}\right).\hfill \end{array}$$

Now, by multiplying both sides with by $t_{k+1}^2$, we deduce

$$\begin{array}{c}\hfill \begin{array}{cc}& {\mathbb{E}}_k\left(t_{k+1}^2,(F^{k+1}(x_{k+1})-F^{k+1}(x^{\ast }))\right)+(t_{k+1}-t_{k+1}^2)(F^{k+1}(x_k)-F^{k+1}(x^{\ast }))\hfill \\ \hfill & \le \frac{1}{2{\gamma }_{k+1}}\left(\Vert ,u_k,-,x^{\ast },{\Vert }^2,-,{\mathbb{E}}_k,\left(\Vert ,u_{k+1},-,x^{\ast },{\Vert }^2\right)\right)+t_{k+1}^2{\gamma }_{k+1}\left({\sigma }^2,+,\frac{{\Vert K\Vert }^2{L}_g^2}{2}\right).\hfill \end{array}\end{array}$$ 28

Next, by adding $t_k^2(F^{k+1}(x_k)-F^{k+1}(x^{\ast }))$ on both sides of (28), gives

$$\begin{array}{cc}& {\mathbb{E}}_k\left(t_{k+1}^2,(F^{k+1}(x_{k+1})-F^{k+1}(x^{\ast }))\right)+{\rho }_{k+1}(F^{k+1}(x_k)-F^{k+1}(x^{\ast }))\le \hfill \\ \hfill & t_k^2(F^{k+1}(x_k)-F^{k+1}(x^{\ast }))+\frac{1}{2{\gamma }_{k+1}}\left(\Vert ,u_k,-,x^{\ast },{\Vert }^2,-,{\mathbb{E}}_k,\left(\Vert ,u_{k+1},-,x^{\ast },{\Vert }^2\right)\right)\hfill \\ \hfill & +t_{k+1}^2{\gamma }_{k+1}\left({\sigma }^2,+,\frac{{\Vert K\Vert }^2{L}_g^2}{2}\right).\hfill \end{array}$$

Utilizing (6) together with the assumption that ${({\mu }_k)}_{k\ge 1}$ is nonincreasing leads to

$$\begin{array}{cc}& {\mathbb{E}}_k\left(t_{k+1}^2,(F^{k+1}(x_{k+1})-F^{k+1}(x^{\ast }))\right)+{\rho }_{k+1}(F^{k+1}(x_k)-F^{k+1}(x^{\ast }))\hfill \\ \hfill & \le t_k^2(F^k(x_k)-F^k(x^{\ast }))+\frac{1}{2{\gamma }_{k+1}}\left(\Vert ,u_k,-,x^{\ast },{\Vert }^2,-,{\mathbb{E}}_k,\left(\Vert ,u_{k+1},-,x^{\ast },{\Vert }^2\right)\right)+t_k^2({\mu }_k-{\mu }_{k+1})\frac{L_g^2}{2}\hfill \\ \hfill & +t_{k+1}^2{\gamma }_{k+1}\left({\sigma }^2,+,\frac{{\Vert K\Vert }^2{L}_g^2}{2}\right).\hfill \end{array}$$

Now, using that $t_k^2\ge t_{k+1}^2-t_{k+1}$, we get

$$\begin{array}{cc}& {\mathbb{E}}_k\left(t_{k+1}^2,(F^{k+1}(x_{k+1})-F^{k+1}(x^{\ast }))\right)+{\rho }_{k+1}(F^{k+1}(x_k)-F^{k+1}(x^{\ast }))\hfill \\ \hfill & \le t_k^2(F^k(x_k)-F^k(x^{\ast }))+\frac{1}{2{\gamma }_{k+1}}(\Vert u_k-x^{\ast }{\Vert }^2-{\mathbb{E}}_k\left(\Vert ,u_{k+1},-,x^{\ast },{\Vert }^2)\right)\hfill \\ \hfill & +t_k^2{\mu }_k\frac{L_g^2}{2}-t_{k+1}^2{\mu }_{k+1}\frac{L_g^2}{2}+t_{k+1}{\mu }_{k+1}\frac{L_g^2}{2}\hfill \\ \hfill & +t_{k+1}^2{\gamma }_{k+1}\left({\sigma }^2,+,\frac{{\Vert K\Vert }^2{L}_g^2}{2}\right).\hfill \end{array}$$

Multiplying both sides with ${\gamma }_{k+1}$ and putting all terms on the correct sides yields

$$\begin{array}{c}\hfill \begin{array}{cc}& {\mathbb{E}}_k({\gamma }_{k+1}{t}_{k+1}^2\left(F^{k+1},(x_{k+1}),-,F^{k+1},(x^{\ast }),+,{\mu }_{k+1},\frac{L_g^2}{2}\right)+\frac{1}{2}{\Vert u_{k+1}-x^{\ast }\Vert }^2)+\hfill \\ \hfill & {\gamma }_{k+1}{\rho }_{k+1}(F^{k+1}(x_k)-F^{k+1}(x^{\ast }))\hfill \\ \hfill & \le {\gamma }_{k+1}{t}_k^2\left(F^k,(x_k),-,F^k,(x^{\ast }),+,{\mu }_k,\frac{L_g^2}{2}\right)+\frac{1}{2}{\Vert u_k-x^{\ast }\Vert }^2\hfill \\ \hfill & +{\gamma }_{k+1}{t}_{k+1}{\mu }_{k+1}\frac{L_g^2}{2}+t_{k+1}^2{\gamma }_{k+1}^2\left({\sigma }^2,+,\frac{{\Vert K\Vert }^2{L}_g^2}{2}\right).\hfill \end{array}\end{array}$$ 29

At this point we would like to discard the term ${\gamma }_{k+1}{\rho }_{k+1}(F^{k+1}(x_k)-F^{k+1}(x^{\ast }))$ which we currently cannot as the positivity of $F^{k+1}(x_k)-F^{k+1}(x^{\ast })$ is not ensured. So we add ${\gamma }_{k+1}{\rho }_{k+1}{\mu }_{k+1}\frac{L_g^2}{2}$ on both sides of (29) and get

$$\begin{array}{c}\hfill \begin{array}{cc}& {\mathbb{E}}_k({\gamma }_{k+1}{t}_{k+1}^2\left(F^{k+1},(x_{k+1}),-,F^{k+1},(x^{\ast }),+,{\mu }_{k+1},\frac{L_g^2}{2}\right)+\frac{1}{2}{\Vert u_{k+1}-x^{\ast }\Vert }^2)+\hfill \\ \hfill & {\gamma }_{k+1}{\rho }_{k+1}\left(F^{k+1},(x_k),-,F^{k+1},(x^{\ast }),+,{\mu }_{k+1},\frac{L_g^2}{2}\right)\le \hfill \\ \hfill & {\gamma }_{k+1}{t}_k^2\left(F^k,(x_k),-,F^k,(x^{\ast }),+,{\mu }_k,\frac{L_g^2}{2}\right)+\frac{1}{2}{\Vert u_k-x^{\ast }\Vert }^2+\hfill \\ \hfill & +{\gamma }_{k+1}{\mu }_{k+1}\frac{L_g^2}{2}(t_{k+1}+{\rho }_{k+1})+t_{k+1}^2{\gamma }_{k+1}^2\left({\sigma }^2,+,\frac{{\Vert K\Vert }^2{L}_g^2}{2}\right).\hfill \end{array}\end{array}$$ 30

Using again (6) to deduce that

$$\begin{array}{c}\hfill {\gamma }_{k+1}{\rho }_{k+1}\left(F^{k+1},(x_k),-,F^{k+1},(x^{\ast }),+,{\mu }_{k+1},\frac{L_g^2}{2}\right)\ge {\gamma }_{k+1}{\rho }_{k+1}(F(x_k)-F(x^{\ast }))\ge 0\end{array}$$

we can now discard said term from (30), giving

$$\begin{array}{c}\hfill \begin{array}{cc}& {\mathbb{E}}_k\left({\gamma }_{k+1},t_{k+1}^2,\left(F^{k+1},(x_{k+1}),-,F^{k+1},(x^{\ast }),+,{\mu }_{k+1},\frac{L_g^2}{2}\right),+,\frac{1}{2},{\Vert u_{k+1}-x^{\ast }\Vert }^2\right)\le \hfill \\ \hfill & {\gamma }_{k+1}{t}_k^2\left(F^k,(x_k),-,F^k,(x^{\ast }),+,{\mu }_k,\frac{L_g^2}{2}\right)+\frac{1}{2}{\Vert u_k-x^{\ast }\Vert }^2\hfill \\ \hfill & +{\gamma }_{k+1}{\mu }_{k+1}\frac{L_g^2}{2}(t_{k+1}+{\rho }_{k+1})+t_{k+1}^2{\gamma }_{k+1}^2\left({\sigma }^2,+,\frac{{\Vert K\Vert }^2{L}_g^2}{2}\right).\hfill \end{array}\end{array}$$ 31

Last but not least we use that $F^k(x_k)-F^k(x^{\ast })+{\mu }_k\frac{L_g^2}{2}\ge F(x_k)-F(x^{\ast })\ge 0$ and ${\gamma }_{k+1}\le {\gamma }_k$ to follow that

$$\begin{array}{c}\hfill {\gamma }_{k+1}{t}_k^2\left(F^k,(x_k),-,F^k,(x^{\ast }),+,{\mu }_k,\frac{L_g^2}{2}\right)\le {\gamma }_k{t}_k^2\left(F^k,(x_k),-,F^k,(x^{\ast }),+,{\mu }_k,\frac{L_g^2}{2}\right).\end{array}$$ 32

Combining (31) and (32) yields

$$\begin{array}{c}\hfill \begin{array}{cc}& {\mathbb{E}}_k\left({\gamma }_{k+1},t_{k+1}^2,\left(F^{k+1},(x_{k+1}),-,F^{k+1},(x^{\ast }),+,{\mu }_{k+1},\frac{L_g^2}{2}\right),+,\frac{1}{2},{\Vert u_{k+1}-x^{\ast }\Vert }^2\right)\le \hfill \\ \hfill & {\gamma }_k{t}_k^2\left(F^k,(x_k),-,F^k,(x^{\ast }),+,{\mu }_k,\frac{L_g^2}{2}\right)+\frac{1}{2}{\Vert u_k-x^{\ast }\Vert }^2\hfill \\ \hfill & +{\gamma }_{k+1}{\mu }_{k+1}\frac{L_g^2}{2}(t_{k+1}+{\rho }_{k+1})+t_{k+1}^2{\gamma }_{k+1}^2\left({\sigma }^2,+,\frac{{\Vert K\Vert }^2{L}_g^2}{2}\right).\hfill \end{array}\end{array}$$ 33

Let $N\ge 2$. We take the expected value on both sides (33) and sum from $k=1$ to $N-1$. Getting rid of the non-negative terms $\Vert u_N-x^{\ast }{\Vert }^2$ gives

$$\begin{array}{c}\hfill \begin{array}{cc}& \mathbb{E}\left({\gamma }_N,t_N^2,\left(F^N,(x_N),-,F^N,(x^{\ast }),+,{\mu }_N,\frac{L_g^2}{2}\right)\right)\le \hfill \\ \hfill & \mathbb{E}\left({\gamma }_1,\left(F^1,(x_1),-,F^1,(x^{\ast }),+,{\mu }_1,\frac{L_g^2}{2}\right)\right)+\frac{1}{2}{\Vert u_1-x^{\ast }\Vert }^2+\sum _{k=2}^N{\gamma }_k{\mu }_k\frac{L_g}{2}(t_k+{\rho }_k)\hfill \\ \hfill & +\sum _{k=2}^N{t}_k^2{\gamma }_k^2\left({\sigma }^2,+,\frac{{\Vert K\Vert }^2{L}_g^2}{2}\right).\hfill \end{array}\end{array}$$

Since $t_1=1$, the above inequality holds also for $N=1$. Now, using Lemma 4.2 we get that for every $N\ge 1$

$$\begin{array}{cc}& \mathbb{E}\left({\gamma }_N,t_N^2,\left(F^N,(x_N),-,F^N,(x^{\ast }),+,{\mu }_N,\frac{L_g^2}{2}\right)\right)\le \frac{1}{2}{\Vert x_0-x^{\ast }\Vert }^2+\sum _{k=1}^N{\gamma }_k{\mu }_k\frac{L_g^2}{2}(t_k+{\rho }_k)\hfill \\ \hfill & +\sum _{k=1}^N{t}_k^2{\gamma }_k^2\left({\sigma }^2,+,\frac{{\Vert K\Vert }^2{L}_g^2}{2}\right).\hfill \end{array}$$

From (7) we follow that

$$\begin{array}{c}\hfill \begin{array}{cc}& {\gamma }_N{t}_N^2\left(F,(x_N),-,F,(x^{\ast })\right)\le {\gamma }_N{t}_N^2\left(F^N,(x_N),-,F^N,(x^{\ast }),+,{\mu }_N,\frac{L_g^2}{2}\right),\hfill \end{array}\end{array}$$

therefore, for every $N\ge 1$

$$\begin{array}{cc}& \mathbb{E}\left({\gamma }_N,t_N^2,\left(F^N,(x_N),-,F^N,(x^{\ast })\right)\right)\le \frac{1}{2}{\Vert x_0-x^{\ast }\Vert }^2+\sum _{k=1}^N{\gamma }_k{\mu }_k\frac{L_g^2}{2}(t_k+{\rho }_k)\hfill \\ \hfill & +\sum _{k=1}^N{t}_k^2{\gamma }_k^2\left({\sigma }^2,+,\frac{{\Vert K\Vert }^2{L}_g^2}{2}\right).\hfill \end{array}$$

By using the fact that ${\mu }_k={\gamma }_k{\Vert K\Vert }^2$ for every $k\ge 1$ gives

$$\begin{array}{cc}& \mathbb{E}\left({\gamma }_N,t_N^2,(F(x_N)-F(x^{\ast }))\right)\le \frac{1}{2}{\Vert x_0-x^{\ast }\Vert }^2+\frac{{\Vert K\Vert }^2{L}_g^2}{2}\sum _{k=1}^N{\gamma }_k^2(t_k+{\rho }_k)\hfill \\ \hfill & +\left({\sigma }^2,+,\frac{{\Vert K\Vert }^2{L}_g^2}{2}\right)\sum _{k=1}^N{t}_k^2{\gamma }_k^2\forall N\ge 1.\hfill \end{array}$$

Thus,

$$\begin{array}{cc}& \mathbb{E}\left(F,(x_N),-,F,(x^{\ast })\right)\le \frac{1}{{\gamma }_N{t}_N^2}\frac{1}{2}{\Vert x_0-x^{\ast }\Vert }^2+\frac{1}{{\gamma }_N{t}_N^2}\frac{{\Vert K\Vert }^2{L}_g^2}{2}\sum _{k=1}^N{\gamma }_k^2(t_k+{\rho }_k)\hfill \\ \hfill & +\frac{1}{{\gamma }_N{t}_N^2}\left({\sigma }^2,+,\frac{{\Vert K\Vert }^2{L}_g^2}{2}\right)\sum _{k=1}^N{t}_k^2{\gamma }_k^2\forall N\ge 1.\hfill \end{array}$$

$\square$

Corollary 4.1

Let

$$\begin{array}{c}\hfill t_1=1,t_{k+1}=\frac{1+\sqrt{1+4t_k^2}}{2}\forall k\ge 1,\end{array}$$

and, for $b>0$,

$$\begin{array}{c}\hfill {\mu }_k=\frac{b}{k^{\frac{3}{2}}}{\Vert K\Vert }^2,\text{and}{\gamma }_k=\frac{b}{k^{\frac{3}{2}}}\forall k\ge 1.\end{array}$$

Then,

$$\begin{array}{cc}& \mathbb{E}\left(F,(x_N),-,F,(x^{\ast })\right)\le 2\frac{\Vert x_0-x^{\ast }{\Vert }^2}{b\sqrt{N}}+b{\Vert K\Vert }^2{L}_g^2\frac{{\pi }^2}{3}\frac{1}{\sqrt{N}}\hfill \\ \hfill & +2b\left(2,{\sigma }^2,+,{\Vert K\Vert }^2,L_g^2\right)\frac{1+log(N)}{\sqrt{N}}\forall N\ge 1.\hfill \end{array}$$

Furthermore, we have that $F(x_N)$ converges almost surely to $F(x^{\ast })$ as $N\to +\infty$.

Proof

First we notice that the choice of $t_{k+1}=\frac{1+\sqrt{1+4t_k^2}}{2}$ fulfills that

$$\begin{array}{c}\hfill {\rho }_{k+1}=t_k^2-t_{k+1}^2+t_{k+1}=0\forall k\ge 1.\end{array}$$

Now we derive the stated convergence result by first showing via induction that

$$\begin{array}{c}\hfill \frac{1}{k}\le \frac{1}{t_k}\le \frac{2}{k}\forall k\ge 1.\end{array}$$

Assuming that this holds for $k\ge 1$, we have that

$$\begin{array}{c}\hfill t_{k+1}=\frac{1+\sqrt{1+4t_k^2}}{2}\le \frac{1+\sqrt{1+4k^2}}{2}\le \frac{1+\sqrt{1+4k+4k^2}}{2}=k+1\end{array}$$

and

$$\begin{array}{c}\hfill t_{k+1}=\frac{1+\sqrt{1+4t_k^2}}{2}\ge \frac{1+\sqrt{1+4{(\frac{k}{2})}^2}}{2}\ge \frac{1+\sqrt{k^2}}{2}\ge \frac{k+1}{2}.\end{array}$$

Furthermore, for every $N\ge 1$ we have that

$$\begin{array}{c}\hfill \begin{array}{cc}& \frac{1}{{\gamma }_N{t}_N^2}\frac{{\Vert K\Vert }^2{L}_g^2}{2}\sum _{k=1}^N{\gamma }_k^2(t_k+{\rho }_k)\le \frac{4}{b\sqrt{N}}\frac{{\Vert K\Vert }^2{L}_g^2}{2}\sum _{k=1}^N\frac{b^2}{k^3}k=\frac{{2b\Vert K\Vert }^2{L}_g^2}{\sqrt{N}}\sum _{k=1}^N{k}^{-2}\hfill \\ \hfill & \le \frac{{2b\Vert K\Vert }^2{L}_g^2}{\sqrt{N}}\sum _{k=1}^{\infty }{k}^{-2}=b{\Vert K\Vert }^2{L}_g^2\frac{{\pi }^2}{3}\frac{1}{\sqrt{N}}.\hfill \end{array}\end{array}$$ 34

The statement of the convergence rate in expectation follows now by plugging in our parameter choices into the statement of Theorem 4.1, using the estimate (34) and checking that

$$\begin{array}{c}\hfill \sum _{k=1}^N{t}_k^2{\gamma }_k^2\le b^2\sum _{k=1}^N\frac{1}{k}\le b^2(1+log(N))\forall N\ge 1.\end{array}$$

The almost sure convergence of ${(F(x_N))}_{N\ge 1}$ can be deduced by looking at (33) and dividing by ${\gamma }_{k+1}{t}_{k+1}^2$ and using that ${\gamma }_{k+1}{t}_{k+1}^2\ge {\gamma }_k{t}_k^2$ as well as ${\rho }_k=0$, which gives for every $k\ge 0$

$$\begin{array}{c}\hfill \begin{array}{cc}& {\mathbb{E}}_k\left(F^{k+1},(x_{k+1}),-,F^{k+1},(x^{\ast }),+,{\mu }_{k+1},\frac{L_g^2}{2},+,\frac{1}{2{\gamma }_{k+1}{t}_{k+1}^2},{\Vert u_{k+1}-x^{\ast }\Vert }^2\right)\hfill \\ \hfill & \le F^k(x_k)-F^k(x^{\ast })+{\mu }_k\frac{L_g^2}{2}+\frac{1}{2{\gamma }_k{t}_k^2}{\Vert u_k-x^{\ast }\Vert }^2+\frac{{\mu }_{k+1}}{t_{k+1}}\frac{L_g^2}{2}+{\gamma }_{k+1}\left({\sigma }^2,+,\frac{{\Vert K\Vert }^2{L}_g^2}{2}\right).\hfill \end{array}\end{array}$$

Plugging in our choice of parameters gives for every $k\ge 0$

$$\begin{array}{c}\hfill \begin{array}{cc}& {\mathbb{E}}_k\left(F^{k+1},(x_{k+1}),-,F^{k+1},(x^{\ast }),+,{\mu }_{k+1},\frac{L_g^2}{2},+,\frac{1}{2{\gamma }_{k+1}{t}_{k+1}^2},{\Vert u_{k+1}-x^{\ast }\Vert }^2\right)\hfill \\ \hfill & \le F^k(x_k)-F^k(x^{\ast })+{\mu }_k\frac{L_g^2}{2}+\frac{1}{2{\gamma }_k{t}_k^2}{\Vert u_k-x^{\ast }\Vert }^2+\frac{C}{k^{\frac{3}{2}}},\hfill \end{array}\end{array}$$

where $C>0$.

Thus, by the famous Robbins-Siegmund Theorem (see [25, Theorem 1]) we get that ${(F^{k+1}(x_{k+1})-F^{k+1}(x^{\ast })+{\mu }_{k+1}\frac{L_g^2}{2})}_{k\ge 0}$ converges almost surely. In particular, from the convergence to 0 in expectation we know that the almost sure limit must also be the constant zero. $\square$

Finite Sum The formulation of the previous section can be used to deal e.g. with problems of the form

$$\begin{array}{c}\hfill \underset{x\in \mathcal{H}}{min}f(x)+\sum _{i=1}^m{g}_i(K_{i}x)\end{array}$$ 35

for $f:\mathcal{H}\to \overline{\mathbb{R}}$ a proper, convex and lower semicontinuous function, $g_i:{\mathcal{G}}_i\to \mathbb{R}$ convex and $L_{g_i}$-Lipschitz continuous functions and $K_i:\mathcal{H}\to {\mathcal{G}}_i$ linear continuous operators for $i=1,\cdots ,m$.

Clearly one could consider

with ${\Vert \mathit{K}\Vert }^2={\sum }_{i=1}^m{\Vert K_i\Vert }^2$ and

in order to reformulate the problem as

$$\begin{array}{c}\hfill \underset{x\in \mathcal{H}}{min}f(x)+\mathit{g}(\mathit{K}x)\end{array}$$

and use Algorithm 3.1 together with the parameter choices described in Corollary 3.1 on this. This results in the following algorithm.

Algorithm 4.2

Let $y_0=x_0\in \mathcal{H},{\mu }_1=b\Vert \mathit{K}\Vert$, for $b>0$, and $t_1=1$. Consider the following iterative scheme

$$\begin{array}{c}\hfill (\forall k\ge 1)\left.⌊\begin{array}{c}{\gamma }_k=\frac{{\sum }_{i=1}^m{\Vert K_i\Vert }^2}{{\mu }_k}\hfill \\ x_k={\text{prox}}_{{\gamma }_{k}f}^{}\left(y_{k-1},-,{\gamma }_k,{\sum }_{i=1}^m,K_i^{\ast },{\text{prox}}_{\frac{1}{{\mu }_k}{g}_i^{\ast }}^{},\left(\frac{K_i{y}_{k-1}}{{\mu }_k}\right)\right)\hfill \\ t_{k+1}=\sqrt{t_k^2+2t_k}\hfill \\ y_k=x_k+\frac{t_k-1}{t_{k+1}}(x_k-x_{k-1})\hfill \\ {\mu }_{k+1}={\mu }_k\frac{t_k^2}{t_{k+1}^2-t_{k+1}}.\hfill \end{array}\right)\end{array}$$

However, Problem (35) also lends itself to be tackled via the stochastic version of our method, Algorithm 4.1, by randomly choosing a subset of the summands. Together with the parameter choices described in Corollary 4.1 which results in the following scheme.

Algorithm 4.3

Let $y_0=x_0\in \mathcal{H},b>0$, and $t_1=1$. Consider the following iterative scheme

$$\begin{array}{c}\hfill (\forall k\ge 1)\left.⌊\begin{array}{c}{\mu }_k=b{\sum }_{i=1}^m{\Vert K_i\Vert }^2{k}^{-\frac{3}{2}}\hfill \\ {\gamma }_k=b{k}^{-\frac{3}{2}}\hfill \\ x_k={\text{prox}}_{{\gamma }_{k}f}^{}\left(y_{k-1},-,{\gamma }_k,\frac{{ϵ}_{i,k}}{p_i},{\sum }_{i=1}^m,K_i^{\ast },{\text{prox}}_{\frac{1}{{\mu }_k}{g}_i^{\ast }}^{},\left(\frac{K_i{y}_{k-1}}{{\mu }_k}\right)\right)\hfill \\ t_{k+1}=\frac{1+\sqrt{1+4t_k^2}}{2}\hfill \\ y_k=x_k+\frac{t_k-1}{t_{k+1}}(x_k-x_{k-1}),\hfill \end{array}\right)\end{array}$$

with ${ϵ}_k:=({ϵ}_{1,k},{ϵ}_{2,k},\cdots ,{ϵ}_{m,k})$ a sequence of i.i.d., ${\{0,1\}}^m$ random variables and $p_i=\mathbb{P}[{ϵ}_{i,1}=1]$.

Since the above two methods were not explicitly developed for this separable case and can therefore not make use of more refined estimation of the constant $\Vert \mathit{K}\Vert$, as it is done in e.g. [14]. However, in the stochastic case, this fact is remedied due to the scaling of the stepsize with respect to the i-th component by $p_i^{-1}$.

Remark 4.2

In theory Algorithm 4.1 could be used to treat more general stochastic problems than finite sums like (35), but in the former case it is not clear anymore how a gradient estimator can be found, so we do not discuss it here.

Numerical Examples

We will focus our numerical experiments on image processing problems. The examples are implemented in python using the operator discretization library (ODL) [1]. We define the discrete gradient operators $D_1$ and $D_2$ representing the discretized derivative in the first and second coordinate respectively, which we will need for the numerical examples. Both map from ${\mathbb{R}}^{m\times n}$ to ${\mathbb{R}}^{m\times n}$ and are defined by

$$\begin{array}{c}\hfill {(D_{1}u)}_{i,j}:=\left\{\begin{array}{cc}{u}_{i+1,j}-u_{i,j}\hfill & 1\le i<m,\hfill \\ 0\hfill & \text{else,}\hfill \end{array}\right)\end{array}$$

and

$$\begin{array}{c}\hfill {(D_{2}u)}_{i,j}:=\left\{\begin{array}{cc}{u}_{i,j+1}-u_{i,j}\hfill & 1\le j<m,\hfill \\ 0\hfill & \text{else.}\hfill \end{array}\right)\end{array}$$

The operator norm of $D_1$ and $D_2$, respectively, is 2 (where we equipped ${\mathbb{R}}^{m\times n}$ with the Frobenius norm). This yields an operator norm of $\sqrt{8}$ for the total gradient $D:=D_1\times D_2$ as a map from ${\mathbb{R}}^{m\times n}$ to ${\mathbb{R}}^{m\times n}\times {\mathbb{R}}^{m\times n}$, see also [12].

We will compare our methods, i.e. the Variable Accelerated SmooThing (VAST) and its stochastic counterpart (sVAST) to the Primal Dual Hybrid Gradient (PDHG) of [15] as well as its stochastic version (sPDHG) from [14]. Furthermore, we will illustrate another competitor, the method by Pesquet and Repetti, see [24], which is another stochastic version of PDHG (see also [29]).

In all examples we choose the parameters in accordance with [14]:

• for PDHG and Pesquet&Repetti: $\tau ={\sigma }_i=\frac{\gamma }{\Vert K\Vert }$

• for sPDHG: ${\sigma }_i=\frac{\gamma }{\Vert K\Vert }$ and $\tau =\frac{\gamma }{n{max}_i\Vert K_i\Vert }$,

where $\gamma =0.99$.

Total Variation Denoising

The task at hand is to reconstruct an image from its noisy observation. We do this by solving

$$\begin{array}{c}\hfill \underset{x\in {\mathbb{R}}^{m\times n}}{min}{\alpha \Vert x-b\Vert }_2+\Vert D_1{x\Vert }_1+{\Vert D_{2}x\Vert }_1,\end{array}$$

with $\alpha >0$ as regularization parameter, in the following setting: $f={\alpha \Vert ·-b\Vert }_2,g_1=g_2={\Vert ·\Vert }_1,K_1=D_1,K_2=D_2$.

Figure 1 illustrates the images (of dimension $m=442$ and $n=331$) used in for this example. These include the groundtruth, i.e. the uncorrupted image, as well as the data for the optimization problem b, which visualizes the level of noise. In Fig. 2 we can see that for the deterministic setting our method is as good as PDHG. For the objective function values, Fig. 2b, this is not too surprising as both algorithms share the same convergence rate. For the distance to a solution however we completely lack a convergence result. Nevertheless in Fig. 2a we can see that our method performs also well with respect to this measure.

Fig. 1.

TV denoising. Images used. The approximate solution is computed by running PDHG for 7000 iterations

Fig. 2.

TV denoising. Plots illustrating the performance of different methods

In the stochastic setting we can see in Fig. 2 that, while sPDHG provides some benefit over its deterministic counterpart, the stochastic version of our method, although significantly increasing the variance, provides great benefit, at least for the objective function values.

Furthermore, Fig. 3, shows the reconstructions of sPDHG and our method which are, despite the different objective function values, quite comparable.

Fig. 3.

TV Denoising. A comparison of the reconstruction for the stochastic variable smoothing method and the stochastic PDHG

Total Variation Deblurring

For this example we want to reconstruct an image from a blurred and noisy image. We assume to know the blurring operator $C:{\mathbb{R}}^{m\times n}\to {\mathbb{R}}^{m\times n}$. This is done by solving

$$\begin{array}{c}\hfill \underset{x\in {\mathbb{R}}^{m\times n}}{min}{\alpha \Vert Cx-b\Vert }_2+\Vert D_1{x\Vert }_1+{\Vert D_{2}x\Vert }_1,\end{array}$$ 36

for $\alpha >0$ as regularization parameter, in the following setting: $f=0,g_1={\alpha \Vert ·-b\Vert }_2,g_2=g_3={\Vert ·\Vert }_1,K_1=C,K_2=D_1,K_2=D_2$.

Figure 4 shows the images used to set up the optimization problem (36), in particular Fig. 4b which corresponds to b in said problem.

Fig. 4.

TV Deblurring. The approximate solution is computed by running PDHG for 3000 iterations

In Fig. 5 we see that while PDGH performs better in the deterministic setting, in particular in the later iteration, the stochastic variable smoothing method provides a significant improvement where sPDHG method seems not to converge. It is interesting to note that in this setting even the deterministic version of our algorithm exhibits a slightly chaotic behaviour. Although neither of the two methods is monotone in the primal objective function PDHG seems here much more stable.

Fig. 5.

TV deblurring. Plots illustrating the performance of different methods

Matrix Factorization

In this section we want to solve a nonconvex and nonsmooth optimization problem of completely positive matrix factorization, see [16, 19, 27]. For an observed matrix $A\in {\mathbb{R}}^{d\times d}$ we want to find a completely positive low rank factorization, meaning we are looking for $x\in {\mathbb{R}}_{\ge 0}^{r\times d}$ with $r\ll d$ such that $x^{T}x=A$. This can be formulated as the following optimization problem

$$\begin{array}{c}\hfill \underset{x\in {\mathbb{R}}_{\ge 0}^{r\times d}}{min}{\Vert x^{T}x-A\Vert }_1,\end{array}$$ 37

where $x^T$ denotes the transpose of the matrix x. The more natural approach might be to use a smooth formulation where ${\Vert ·\Vert }_2^2$ is used instead of the 1-Norm we are suggesting. However, the former choice of distance measure, albeit smooth, comes with its own set of problems (mainly a non-Lipschitz gradient).

The so called Prox-Linear method presented in [18] solves the above problem (37), by linearizing the smooth (${\mathbb{R}}^{d\times d}$-valued) function $x\mapsto x^{T}x$ inside the nonsmooth distance function. In particular for the problem

$$\begin{array}{c}\hfill \underset{x}{min}g(c(x)),\end{array}$$

for a smooth vector valued function c and a convex and Lipschitz function g, [18] proposes to iteratively solve the subproblem

$$\begin{array}{c}\hfill x_{k+1}=\underset{x}{\arg \min }\left\{g,(c(x_k)+\mathrm{\nabla }c(x_k)(x-x_k)),+,\frac{1}{2t},{\Vert x-x_k\Vert }_2^2\right\}\forall k\ge 0,\end{array}$$ 38

for a stepsize $t\le {(L_g{L}_{D\mathrm{\nabla }c})}^{-1}$. For our particular problem described in (37) the subproblem looks as follows

$$\begin{array}{c}\hfill x_{k+1}=\underset{x\in {\mathbb{R}}_{\ge 0}^{r\times d}}{\arg \min }\left\{\Vert ,x_k^T,{x-A\Vert }_1,+,\frac{1}{2},{\Vert x-x_k\Vert }_2^2\right\},\end{array}$$ 39

and therefore fits our general setup described in (1) with the identification $f=\Vert ·-x_k{\Vert }_2^2+{\delta }_{{\mathbb{R}}_{\ge 0}^{r\times d}}(x)$, $g={\Vert ·\Vert }_1$ and $K=x_k^T$. Moreover, due to its separable structure, the subproblem (39) fits the special case described in (35) and can therefore be tackled by the stochastic version of our algorithm presented in Algorithm 4.3. In particular reformulating (38) for the stochastic finite sum setting we interpret the subproblem as

$$\begin{array}{c}\hfill x_{k+1}=\underset{x\in {\mathbb{R}}_{\ge 0}^{r\times d}}{\arg \min }\left\{\sum _{i=1}^d,{∥x_k^T,[i,:],x,-,A,[i,:]∥}_1,+,\frac{1}{2},{\Vert x-x_k\Vert }_2^2\right\},\end{array}$$

where A[i,  : ] denotes the i-th row of the matrix A (Fig. 6).

Fig. 6.

Comparison of the evolutions of the objective function values for different starting points. We run 40 epochs with 5 iterations each. For each epoch we choose the last iterate of the previous epoch as the linearization. For the stochastic methods we fix the number of rows (batch size) which are randomly chosen in each update a priori and count d divided by this number as one iteration. For the randomly chosen initial point we use a batch size of 3 (to allow for more exploration) and for the one close to the solution we use 5 in order to give a more accuracy. The parameter b in the variable smoothing method was chosen with minimal tuning to be 0.1 for both the deterministic and the stochastic version

In comparison to Sects. 5.1 and 5.2 a new aspect becomes important when evaluating methods for solving (38). Now, it is not only relevant how well subproblem (39) is solved, but also the trajectory taken in doing so as different paths might lead to different local minima. This can be seen in Fig. 6 where PDHG gets stuck early on in bad local minima. The variable smoothing method (especially the stochastic version) is able to move further from the starting point and find better local minima. Note that in general the methods have a difficulty in finding the global minimum $x_{\mathit{true}}\in {\mathbb{R}}^{3\times 60}$ (with optimal objective function value zero, as constructed $A:=x_{\mathit{true}}^T{x}_{\mathit{true}}\in {\mathbb{R}}^{60\times 60}$ in all examples).

Funding

Open access funding provided by Austrian Science Fund (FWF).