Subgradient ellipsoid method for nonsmooth convex problems

Abstract

In this paper, we present a new ellipsoid-type algorithm for solving nonsmooth problems with convex structure. Examples of such problems include nonsmooth convex minimization problems, convex-concave saddle-point problems and variational inequalities with monotone operator. Our algorithm can be seen as a combination of the standard Subgradient and Ellipsoid methods. However, in contrast to the latter one, the proposed method has a reasonable convergence rate even when the dimensionality of the problem is sufficiently large. For generating accuracy certificates in our algorithm, we propose an efficient technique, which ameliorates the previously known recipes (Nemirovski in Math Oper Res 35(1):52–78, 2010).

Introduction

The Ellipsoid Method is a classical algorithm in Convex Optimization. It was proposed in 1976 by Yudin and Nemirovski [23] as the modified method of centered cross-sections and then independently rediscovered a year later by Shor [21] in the form of the subgradient method with space dilation. However, the popularity came to the Ellipsoid Method only when Khachiyan used it in 1979 for proving his famous result on polynomial solvability of Linear Programming [10]. Shortly after, several polynomial algorithms, based on the Ellipsoid Method, were developed for some combinatorial optimization problems [9]. For more details and historical remarks on the Ellipsoid Method, see [2, 3, 14].

Despite its long history, the Ellipsoid Method still has some issues which have not been fully resolved or have been resolved only recently. One of them is the computation of accuracy certificates which is important for generating approximate solutions to dual problems or for solving general problems with convex structure (saddle-point problems, variational inequalities, etc.). For a long time, the procedure for calculating an accuracy certificate in the Ellipsoid Method required solving an auxiliary piecewise linear optimization problem (see, e.g., sect. 5 and 6 in [14]). Although this auxiliary computation did not use any additional calls to the oracle, it was still computationally expensive and, in some cases, could take even more time than the Ellipsoid Method itself. Only recently an efficient alternative has been proposed [16].

Another issue with the Ellipsoid Method is related to its poor dependency on the dimensionality of the problem. Consider, e.g., the minimization problem

$$\begin{array}{c}\hfill \underset{x\in Q}{min}f(x),\end{array}$$ 1

where $f:{\mathbb{R}}^n\to \mathbb{R}$ is a convex function and $Q:=\{x\in {\mathbb{R}}^n:\Vert x\Vert \le R\}$ is the Euclidean ball of radius $R>0$. The Ellipsoid Method for solving (1) can be written as follows (see, e.g., sect. 3.2.8 in [19]):

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill x_{k+1}& :=x_k-\frac{1}{n+1}\frac{W_k{g}_k}{{⟨g_k,W_k{g}_k⟩}^{1/2}},\hfill \\ \hfill W_{k+1}& :=\frac{n^2}{n^2-1}(W_k-\frac{2}{n+1}\frac{W_k{g}_k{g}_k^T{W}_k}{⟨g_k,W_k{g}_k⟩}),k\ge 0,\hfill \end{array}\end{array}$$ 2

where $x_0:=0$, $W_0:=R^{2}I$ (I is the identity matrix) and $g_k:=f^{\prime }(x_k)$ is an arbitrary nonzero subgradient if $x_k\in Q$, and $g_k$ is an arbitrary separator1 of $x_k$ from Q if $x_k\notin Q$.

To solve problem (1) with accuracy $ϵ>0$ (in terms of the function value), the Ellipsoid Method needs

$$\begin{array}{c}\hfill O(n^{2}ln\frac{\mathit{MR}}{ϵ})\end{array}$$ 3

iterations, where $M>0$ is the Lipschitz constant of f on Q (see theorem 3.2.11 in [19]). Looking at this estimate, we can see an immediate drawback: it directly depends on the dimension and becomes useless when $n\to \infty$. In particular, we cannot guarantee any reasonable rate of convergence for the Ellipsoid Method when the dimensionality of the problem is sufficiently big.

Note that the aforementioned drawback is an artifact of the method itself, not its analysis. Indeed, when $n\to \infty$, iteration (2) reads

$$\begin{array}{c}\hfill x_{k+1}:=x_k,W_{k+1}:=W_k,k\ge 0.\end{array}$$

Thus, the method stays at the same point and does not make any progress.

On the other hand, the simplest Subgradient Method for solving (1) possesses the “dimension-independent” $O(M^2{R}^2/{ϵ}^2)$ iteration complexity bound (see, e.g., sect. 3.2.3 in [19]). Comparing this estimate with (3), we see that the Ellipsoid Method is significantly faster than the Subgradient Method only when n is not too big compared to $MR/ϵ$ and significantly slower otherwise. Clearly, this situation is strange because the former algorithm does much more work at every iteration by “improving” the “metric” $W_k$ which is used for measuring the norm of the subgradients.

In this paper, we propose a new ellipsoid-type algorithm for solving nonsmooth problems with convex structure, which does not have the discussed above drawback. Our algorithm can be seen as a combination of the Subgradient and Ellipsoid methods and its convergence rate is basically as good as the best of the corresponding rates of these two methods (up to some logarithmic factors). In particular, when $n\to \infty$, the convergence rate of our algorithm coincides with that of the Subgradient Method.

Contents

This paper is organized as follows. In Sect. 2.1, we review the general formulation of a problem with convex structure and the associated with it notions of accuracy certificate and residual. Our presentation mostly follows [16] with examples taken from [18]. Then, in Sect. 2.2, we introduce the notions of accuracy semicertificate and gap and discuss their relation with those of accuracy certificate and residual.

In Sect. 3, we present the general algorithmic scheme of our methods. To measure the convergence rate of this scheme, we introduce the notion of sliding gap and establish some preliminary bounds on it.

In Sect. 4, we discuss different choices of parameters in our general scheme. First, we show that, by setting some of the parameters to zero, we obtain the standard Subgradient and Ellipsoid methods. Then we consider a couple of other less trivial choices which lead to two new algorithms. The principal of these new algorithms is the latter one, which we call the Subgradient Ellipsoid Method. We demonstrate that the convergence rate of this algorithm is basically as good as the best of those of the Subgradient and Ellipsoid methods.

In Sect. 5, we show that, for both our new methods, it is possible to efficiently generate accuracy semicertificates whose gap is upper bounded by the sliding gap. We also compare our approach with the recently proposed technique from [16] for building accuracy certificates for the standard Ellipsoid Method.

In Sect. 6, we discuss how to efficiently implement our general scheme and the procedure for generating accuracy semicertificates. In particular, we show that the time and memory requirements of our scheme are the same as in the standard Ellipsoid Method.

Finally, in Sect. 7, we discuss some open questions.

Notation and generalities

In this paper, $\mathbb{E}$ denotes an arbitrary n-dimensional real vector space. Its dual space, composed of all linear functionals on $\mathbb{E}$, is denoted by ${\mathbb{E}}^{\ast }$. The value of $s\in {\mathbb{E}}^{\ast }$, evaluated at $x\in \mathbb{E}$, is denoted by $⟨s,x⟩$. See [19, sect. 4.2.1] for the supporting discussion of abstract real vector spaces in Optimization.

Let us introduce in the spaces $\mathbb{E}$ and ${\mathbb{E}}^{\ast }$ a pair of conjugate Euclidean norms. To this end, let us fix a self-adjoint positive definite linear operator $B:\mathbb{E}\to {\mathbb{E}}^{\ast }$ and define

$$\begin{array}{c}\hfill \Vert x\Vert :={⟨Bx,x⟩}^{1/2},x\in \mathbb{E},{\Vert s\Vert }_{\ast }:={⟨s,B^{-1}s⟩}^{1/2},s\in {\mathbb{E}}^{\ast }.\end{array}$$

Note that, for any $s\in {\mathbb{E}}^{\ast }$ and $x\in \mathbb{E}$, we have the Cauchy-Schwarz inequality

$$\begin{array}{c}\hfill |⟨s,x⟩|\le {\Vert s\Vert }_{\ast }\Vert x\Vert ,\end{array}$$

which becomes an equality if and only if s and Bx are collinear. In addition to $\Vert x\Vert$ and ${\Vert ·\Vert }_{\ast }$, we often work with other Euclidean norms defined in the same way but using another reference operator instead of B. In this case, we write ${\Vert ·\Vert }_G$ and ${\Vert ·\Vert }_G^{\ast }$, where $G:\mathbb{E}\to {\mathbb{E}}^{\ast }$ is the corresponding self-adjoint positive definite linear operator.

Sometimes, in the formulas, involving products of linear operators, it is convenient to treat $x\in \mathbb{E}$ as a linear operator from $\mathbb{R}$ to $\mathbb{E}$, defined by $x\alpha :=\alpha x$, and $x^{\ast }$ as a linear operator from ${\mathbb{E}}^{\ast }$ to $\mathbb{R}$, defined by $x^{\ast }s:=⟨s,x⟩$. Likewise, any $s\in {\mathbb{E}}^{\ast }$ can be treated as a linear operator from $\mathbb{R}$ to ${\mathbb{E}}^{\ast }$, defined by $s\alpha :=\alpha s$, and $s^{\ast }$ as a linear operator from $\mathbb{E}$ to $\mathbb{R}$, defined by $s^{\ast }x:=⟨s,x⟩$. Then, $x{x}^{\ast }$ and $s{s}^{\ast }$ are rank-one self-adjoint linear operators from ${\mathbb{E}}^{\ast }$ to $\mathbb{E}$ and from $\mathbb{E}$ to ${\mathbb{E}}^{\ast }$ respectively, acting as follows: $(x{x}^{\ast })s=⟨s,x⟩x$ and $(s{s}^{\ast })x=⟨s,x⟩s$ for any $x\in \mathbb{E}$ and $s\in {\mathbb{E}}^{\ast }$.

For a self-adjoint linear operator $G:\mathbb{E}\to {\mathbb{E}}^{\ast }$, by $\mathrm{tr}G$ and $detG$, we denote the trace and determinant of G with respect to our fixed operator B:

$$\begin{array}{c}\hfill \mathrm{tr}G:=\mathrm{tr}(B^{-1}G),detG:=det(B^{-1}G).\end{array}$$

Note that, in these definitions, $B^{-1}G$ is a linear operator from $\mathbb{E}$ to $\mathbb{E}$, so $\mathrm{tr}(B^{-1}G)$ and $det(B^{-1}G)$ are the standard well-defined notions of trace and determinant of a linear operator acting on the same space. For example, they can be defined as the trace and determinant of the matrix representation of $B^{-1}G$ with respect to an arbitrary chosen basis in $\mathbb{E}$ (the result is independent of the particular choice of basis). Alternatively, $\mathrm{tr}G$ and $detG$ can be equivalently defined as the sum and product, respectively, of the eigenvalues of G with respect to B.

For a point $x\in \mathbb{E}$ and a real $r>0$, by

$$\begin{array}{c}\hfill B\left(x,r\right):=\{y\in \mathbb{E}:\Vert x\Vert \le r\},\end{array}$$

we denote the closed Euclidean ball with center x and radius r.

Given two solids2$Q,Q_0\subseteq \mathbb{E}$, we can define the relative volume of Q with respect to $Q_0$ by $\mathrm{vol}(Q/Q_0):=\mathrm{vol}{Q}^e/\mathrm{vol}{Q}_0^e$, where e is an arbitrary basis in $\mathbb{E}$, $Q^e,Q_0^e\subseteq {\mathbb{R}}^n$ are the coordinate representations of the sets $Q,Q_0$ in the basis e and $\mathrm{vol}$ is the Lebesgue measure in ${\mathbb{R}}^n$. Note that the relative volume is independent of the particular choice of the basis e. Indeed, for any other basis f, we have $Q^e=T_f^e{Q}^f$, $Q_0^e=T_f^e{Q}_0^f$, where $T_f^e$ is the $n\times n$ change-of-basis matrix, so $\mathrm{vol}{Q}^e=(det{T}_f^e)(\mathrm{vol}{Q}^f)$, $\mathrm{vol}{Q}_0^e=(det{T}_f^e)(\mathrm{vol}{Q}_0^f)$ and hence $\mathrm{vol}{Q}^e/\mathrm{vol}{Q}_0^e=\mathrm{vol}{Q}^f/\mathrm{vol}{Q}_0^f$.

For us, it will be convenient to define the volume of a solid $Q\subseteq \mathbb{E}$ as the relative volume of Q with respect to the unit ball:

$$\begin{array}{c}\hfill \mathrm{vol}Q:=\mathrm{vol}(Q/B\left(0,1\right)).\end{array}$$

For an ellipsoid $W:=\{x\in \mathbb{E}:⟨Gx,x⟩\le 1\}$, where $G:\mathbb{E}\to {\mathbb{E}}^{\ast }$ is a self-adjoint positive definite linear operator, we have $\mathrm{vol}W={(detG)}^{-1/2}$.

Convex problems and accuracy certificates

Description and examples

In this paper, we consider numerical algorithms for solving problems with convex structure. The main examples of such problems are convex minimization problems, convex-concave saddle-point problems, convex Nash equilibrium problems, and variational inequalities with monotone operators.

The general formulation of a problem with convex structure involves two objects:

• Solid $Q\subseteq \mathbb{E}$ (called the feasible set), represented by the Separation Oracle: given any point $x\in \mathbb{E}$, this oracle can check whether $x\in \mathrm{int}Q$, and if not, it reports a vector $g_Q(x)\in {\mathbb{E}}^{\ast }\backslash \{0\}$ which separates x from Q:

  $$\begin{array}{c}\hfill ⟨g_Q(x),x-y⟩\ge 0,\forall y\in Q.\end{array}$$ 4

• Vector field $g:\mathrm{int}Q\to {\mathbb{E}}^{\ast }$, represented by the First-Order Oracle: given any point $x\in \mathrm{int}Q$, this oracle returns the vector g(x).

In what follows, we only consider the problems satisfying the following condition:

$$\begin{array}{c}\hfill \exists x^{\ast }\in Q:⟨g(x),x-x^{\ast }⟩\ge 0,\forall x\in \mathrm{int}Q.\end{array}$$ 5

Remark 1

A careful reader may note that the notation $x^{\ast }$ overlaps with our general notation for the linear operator generated by a point x (see Sect. 1). However, there should be no risk of confusion since the precise meaning of $x^{\ast }$ can usually be easily inferred from the context.

A numerical algorithm for solving a problem with convex structure starts at some point $x_0\in \mathbb{E}$. At each step $k\ge 0$, it queries the oracles at the current test point $x_k$ to obtain the new information about the problem, and then somehow uses this new information to form the next test point $x_{k+1}$. Depending on whether $x_k\in \mathrm{int}Q$, the kth step of the algorithm is called productive or nonproductive.

The total information, obtained by the algorithm from the oracles after $k\ge 1$ steps, comprises its execution protocol which consists of:

• The test points $x_0,\dots ,x_{k-1}\in \mathbb{E}$.

• The set of productive steps $I_k:=\{0\le i\le k-1:x_i\in \mathrm{int}Q\}$.

• The vectors $g_0,\dots ,g_{k-1}\in {\mathbb{E}}^{\ast }$ reported by the oracles: $g_i:=g(x_i)$, if $i\in I_k$, and $g_i:=g_Q(x_i)$, if $i\notin I_k$, $0\le i\le k-1$.

An accuracy certificate, associated with the above execution protocol, is a nonnegative vector $\lambda :=({\lambda }_0,\dots ,{\lambda }_{k-1})$ such that $S_k(\lambda ):={\sum }_{i\in I_k}{\lambda }_i>0$ (and, in particular, $I_k\ne \mathrm{\varnothing }$). Given any solid $\Omega$, containing Q, we can define the following residual of $\lambda$ on $\Omega$:

$$\begin{array}{c}\hfill {ϵ}_k(\lambda ):=\underset{x\in \Omega }{max}\frac{1}{S_k(\lambda )}\sum _{i=0}^{k-1}{\lambda }_i⟨g_i,x_i-x⟩,\end{array}$$ 6

which is easily computable whenever $\Omega$ is a simple set (e.g., a Euclidean ball). Note that

$$\begin{array}{c}\hfill {ϵ}_k(\lambda )\ge \underset{x\in Q}{max}\frac{1}{S_k(\lambda )}\sum _{i=0}^{k-1}{\lambda }_i⟨g_i,x_i-x⟩\ge \underset{x\in Q}{max}\frac{1}{S_k(\lambda )}\sum _{i\in I_k}{\lambda }_i⟨g_i,x_i-x⟩\end{array}$$ 7

and, in particular, ${ϵ}_k(\lambda )\ge 0$ in view of (5).

In what follows, we will be interested in the algorithms, which can produce accuracy certificates ${\lambda }^{(k)}$ with ${ϵ}_k({\lambda }^{(k)})\to 0$ at a certain rate. This is a meaningful goal because, for all known instances of problems with convex structure, the residual ${ϵ}_k(\lambda )$ upper bounds a certain natural inaccuracy measure for the corresponding problem. Let us briefly review some standard examples (for more examples, see [16, 18] and the references therein).

Example 1

(Convex minimization problem) Consider the problem

$$\begin{array}{c}\hfill f^{\ast }:=\underset{x\in Q}{min}f(x),\end{array}$$ 8

where $Q\subseteq \mathbb{E}$ is a solid and $f:\mathbb{E}\to \mathbb{R}\cup \{+\infty \}$ is closed convex and finite on $\mathrm{int}Q$.

The First-Order Oracle for (8) is $g(x):=f^{\prime }(x)$, $x\in \mathrm{int}Q$, where $f^{\prime }(x)$ is an arbitrary subgradient of f at x. Clearly, (5) holds for $x^{\ast }$ being any solution of (8).

One can verify that, in this example, the residual ${ϵ}_k(\lambda )$ upper bounds the functional residual: for ${\widehat{x}}_k:=\frac{1}{S_k(\lambda )}{\sum }_{i\in I_k}{\lambda }_i{x}_i$ or $x_k^{\ast }:=\mathrm{argmin}\{f(x):x\in X_k\}$, where $X_k:=\{x_i:i\in I_k\}$, we have $f({\widehat{x}}_k)-f^{\ast }\le {ϵ}_k(\lambda )$ and $f(x_k^{\ast })-f^{\ast }\le {ϵ}_k(\lambda )$.

Moreover, ${ϵ}_k(\lambda )$, in fact, upper bounds the primal-dual gap for a certain dual problem for (8). Indeed, let $f_{\ast }:{\mathbb{E}}^{\ast }\to \mathbb{R}\cup \{+\infty \}$ be the conjugate function of f. Then, we can represent (8) in the following dual form:

$$\begin{array}{c}\hfill f^{\ast }=\underset{x\in Q}{min}\underset{s\in \mathrm{dom}{f}_{\ast }}{max}[⟨s,x⟩-f_{\ast }(s)]=\underset{s\in \mathrm{dom}{f}_{\ast }}{max}[-f_{\ast }(s)-{\xi }_Q(-s)],\end{array}$$ 9

where $\mathrm{dom}{f}_{\ast }:=\{s\in {\mathbb{E}}^{\ast }:f_{\ast }(s)<+\infty \}$ and ${\xi }_Q(-s):={max}_{x\in Q}⟨-s,x⟩$. Denote $s_k:=\frac{1}{S_k(\lambda )}{\sum }_{i\in I_k}{\lambda }_i{g}_i$. Then, using (7) and the convexity of f and $f_{\ast }$, we obtain

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill {ϵ}_k(\lambda )& \ge \frac{1}{S_k(\lambda )}\sum _{i\in I_k}{\lambda }_i⟨g_i,x_i⟩+{\xi }_Q(-s_k)\hfill \\ \hfill & =\frac{1}{S_k(\lambda )}\sum _{i\in I_k}{\lambda }_i[f(x_i)+f_{\ast }(g_i)]+{\xi }_Q(-s_k)\hfill \\ \hfill & \ge f({\widehat{x}}_k)+f_{\ast }(s_k)+{\xi }_Q(-s_k).\hfill \end{array}\end{array}$$

Thus, ${\widehat{x}}_k$ and $s_k$ are ${ϵ}_k(\lambda )$-approximate solutions (in terms of function value) to problems (8) and (9), respectively. Note that the same is true if we replace ${\widehat{x}}_k$ with $x_k^{\ast }$.

Example 2

(Convex-concave saddle-point problem) Consider the following problem: Find $(u^{\ast },v^{\ast })\in U\times V$ such that

$$\begin{array}{c}\hfill f(u^{\ast },v)\le f(u^{\ast },v^{\ast })\le f(u,v^{\ast }),\forall (u,v)\in U\times V,\end{array}$$ 10

where U, V are solids in some finite-dimensional vector spaces ${\mathbb{E}}_u$, ${\mathbb{E}}_v$, respectively, and $f:U\times V\to \mathbb{R}$ is a continuous function which is convex-concave, i.e., $f(·,v)$ is convex and $f(u,·)$ is concave for any $u\in U$ and any $v\in V$.

In this example, we set $\mathbb{E}:={\mathbb{E}}_u\times {\mathbb{E}}_v$, $Q:=U\times V$ and use the First-Order Oracle

$$\begin{array}{c}\hfill g(x):=(f_u^{\prime }(x),-f_v^{\prime }(x)),x:=(u,v)\in \mathrm{int}Q,\end{array}$$

where $f_u^{\prime }(x)$ is an arbitrary subgradient of $f(·,v)$ at u and $f_v^{\prime }(y)$ is an arbitrary supergradient of $f(u,·)$ at v. Then, for any $x:=(u,v)\in \mathrm{int}Q$ and any $x^{\prime }:=(u^{\prime },v^{\prime })\in Q$,

$$\begin{array}{c}\hfill ⟨g(x),x-x^{\prime }⟩=⟨f_u^{\prime }(x),u-u^{\prime }⟩-⟨f_v^{\prime }(x),v-v^{\prime }⟩\ge f(u,v^{\prime })-f(u^{\prime },v).\end{array}$$ 11

In particular, (5) holds for $x^{\ast }:=(u^{\ast },v^{\ast })$ in view of (10).

Let $\varphi :U\to \mathbb{R}$ and $\psi :V\to \mathbb{R}$ be the functions

$$\begin{array}{c}\hfill \varphi (u):=\underset{v\in V}{max}f(u,v),\psi (v):=\underset{u\in U}{min}f(u,v).\end{array}$$

In view of (10), we have $\psi (v)\le f(u^{\ast },v^{\ast })\le \varphi (u)$ for all $(u,v)\in U\times V$. Therefore, the difference $\varphi (u)-\psi (v)$ (called the primal-dual gap) can be used for measuring the quality of an approximate solution $x:=(u,v)\in Q$ to problem (10).

Denoting ${\widehat{x}}_k:=\frac{1}{S_k(\lambda )}{\sum }_{i\in I_k}{\lambda }_i{x}_i=:({\widehat{u}}_k,{\widehat{v}}_k)$ and using (7), we obtain

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill {ϵ}_k(\lambda )& \ge \underset{x\in Q}{max}\frac{1}{S_k(\lambda )}\sum _{i\in I_k}{\lambda }_i⟨g_i,x_i-x⟩\hfill \\ \hfill & \ge \underset{u\in U,v\in V}{max}\frac{1}{S_k(\lambda )}\sum _{i\in I_k}{\lambda }_i[f(u_i,v)-f(u,v_i)]\hfill \\ \hfill & \ge \underset{u\in U,v\in V}{max}[f({\widehat{u}}_k,v)-f(u,{\widehat{v}}_k)]=\varphi ({\widehat{u}}_k)-\psi ({\widehat{v}}_k),\hfill \end{array}\end{array}$$

where the second inequality is due to (11) and the last one follows from the convexity-concavity of f. Thus, the residual ${ϵ}_k(\lambda )$ upper bounds the primal-dual gap for the approximate solution ${\widehat{x}}_k$.

Example 3

(Variational inequality with monotone operator) Let $Q\subseteq \mathbb{E}$ be a solid and let $V:Q\to {\mathbb{E}}^{\ast }$ be a continuous operator which is monotone, i.e., $⟨V(x)-V(y),x-y⟩\ge 0$ for all $x,y\in Q$. The goal is to solve the following (weak) variational inequality:

$$\begin{array}{c}\hfill \text{Find}{x}^{\ast }\in Q:⟨V(x),x-x^{\ast }⟩\ge 0,\forall x\in Q.\end{array}$$ 12

Since V is continuous, this problem is equivalent to its strong variant: find $x^{\ast }\in Q$ such that $⟨V(x^{\ast }),x-x^{\ast }⟩\ge 0$ for all $x\in Q$.

A standard tool for measuring the quality of an approximate solution to (12) is the dual gap function, introduced in [1]:

$$\begin{array}{c}\hfill f(x):=\underset{y\in Q}{max}⟨V(y),x-y⟩,x\in Q.\end{array}$$

It is easy to see that f is a convex nonnegative function which equals 0 exactly at the solutions of (12).

In this example, the First-Order Oracle is defined by $g(x):=V(x)$ for any $x\in \mathrm{int}Q$. Denote ${\widehat{x}}_k:=\frac{1}{S_k(\lambda )}{\sum }_{i\in I_k}{\lambda }_i{x}_i$. Then, using (7) and the monotonicity of V, we obtain

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill {ϵ}_k(\lambda )& \ge \underset{x\in Q}{max}\frac{1}{S_k(\lambda )}\sum _{i\in I_k}{\lambda }_i⟨V(x_i),x_i-x⟩\hfill \\ \hfill & \ge \underset{x\in Q}{max}\frac{1}{S_k(\lambda )}\sum _{i\in I_k}{\lambda }_i⟨V(x),x_i-x⟩=f({\widehat{x}}_k).\hfill \end{array}\end{array}$$

Thus, ${ϵ}_k(\lambda )$ upper bounds the dual gap function for the approximate solution ${\widehat{x}}_k$.

Establishing convergence of residual

For the algorithms, considered in this paper, instead of accuracy certificates and residuals, it turns out to be more convenient to speak about closely related notions of accuracy semicertificates and gaps, which we now introduce.

As before, let $x_0,\cdots ,x_{k-1}$ be the test points, generated by the algorithm after $k\ge 1$ steps, and let $g_0,\cdots ,g_{k-1}$ be the corresponding oracle outputs. An accuracy semicertificate, associated with this information, is a nonnegative vector $\lambda :=({\lambda }_0,\cdots ,{\lambda }_{k-1})$ such that ${\Gamma }_k(\lambda ):={\sum }_{i=0}^{k-1}{\lambda }_i{\Vert g_i\Vert }_{\ast }>0$. Given any solid $\Omega$, containing Q, the gap of $\lambda$ on $\Omega$ is defined in the following way:

$$\begin{array}{c}\hfill {\delta }_k(\lambda ):=\underset{x\in \Omega }{max}\frac{1}{{\Gamma }_k(\lambda )}\sum _{i=0}^{k-1}{\lambda }_i⟨g_i,x_i-x⟩.\end{array}$$ 13

Comparing these definitions with those of accuracy certificate and residual, we see that the only difference between them is that now we use a different “normalizing” coefficient: ${\Gamma }_k(\lambda )$ instead of $S_k(\lambda )$. Also, in the definitions of semicertificate and gap, we do not make any distinction between productive and nonproductive steps. Note that ${\delta }_k(\lambda )\ge 0$.

Let us demonstrate that by making the gap sufficiently small, we can make the corresponding residual sufficiently small as well. For this, we need the following standard assumption about our problem with convex structure (see, e.g., [16]).

Assumption 1

The vector field g, reported by the First-Order Oracle, is semibounded:

$$\begin{array}{c}\hfill ⟨g(x),y-x⟩\le V,\forall x\in \mathrm{int}Q,\forall y\in Q.\end{array}$$

A classical example of a semibounded field is a bounded one: if there is $M\ge 0$, such that ${\Vert g(x)\Vert }_{\ast }\le M$ for all $x\in \mathrm{int}Q$, then g is semibounded with $V:=MD$, where D is the diameter of Q. However, there exist other examples. For instance, if g is the subgradient field of a convex function $f:\mathbb{E}\to \mathbb{R}\cup \{+\infty \}$, which is finite and continuous on Q, then g is semibounded with $V:={max}_{Q}f-{min}_{Q}f$ (variation of f on Q); however, g is not bounded if f is not Lipschitz continuous (e.g., $f(x):=-\sqrt{x}$ on $Q:=[0,1]$). Another interesting example is the subgradient field g of a $\nu$-self-concordant barrier $f:\mathbb{E}\to \mathbb{R}\cup \{+\infty \}$ for the set Q; in this case, g is semibounded with $V:=\nu$ (see, e.g., [19, Theorem 5.3.7]), while $f(x)\to +\infty$ at the boundary of Q.

Lemma 1

Let $\lambda$ be a semicertificate such that ${\delta }_k(\lambda )<r$, where r is the largest of the radii of Euclidean balls contained in Q. Then, $\lambda$ is a certificate and

$$\begin{array}{c}\hfill {ϵ}_k(\lambda )\le \frac{{\delta }_k(\lambda )}{r-{\delta }_k(\lambda )}V.\end{array}$$

Proof

Denote ${\delta }_k:={\delta }_k(\lambda )$, ${\Gamma }_k:={\Gamma }_k(\lambda )$, $S_k:=S_k(\lambda )$. Let $\overline{x}\in Q$ be such that $B\left(\overline{x},r\right)\subseteq Q$. For each $0\le i\le k-1$, let $z_i$ be a maximizer of $z\mapsto ⟨g_i,z-\overline{x}⟩$ on $B\left(\overline{x},r\right)$. Then, for any $0\le i\le k-1$, we have $⟨g_i,\overline{x}-x_i⟩=⟨g_i,z_i-x_i⟩-r{\Vert g_i\Vert }_{\ast }$ with $z_i\in Q$. Therefore,

$$\begin{array}{c}\hfill \sum _{i=0}^{k-1}{\lambda }_i⟨g_i,\overline{x}-x_i⟩=\sum _{i=0}^{k-1}{\lambda }_i⟨g_i,z_i-x_i⟩-r{\Gamma }_k\le S_{k}V-r{\Gamma }_k,\end{array}$$ 14

where the inequality follows from the separation property (4) and Assumption 1.

Let $x\in \Omega$ be arbitrary. Denoting $y:=({\delta }_k\overline{x}+(r-{\delta }_k)x)/r\in \Omega$, we obtain

$$\begin{array}{cc}\hfill (r-{\delta }_k)\sum _{i=0}^{k-1}{\lambda }_i⟨g_i,x_i-x⟩& =r\sum _{i=0}^{k-1}{\lambda }_i⟨g_i,x_i-y⟩+{\delta }_k\sum _{i=0}^{k-1}{\lambda }_i⟨g_i,\overline{x}-x_i⟩\hfill \\ \hfill & \le r{\delta }_k{\Gamma }_k+{\delta }_k\sum _{i=0}^{k-1}{\lambda }_i⟨g_i,\overline{x}-x_i⟩\le {\delta }_k{S}_{k}V,\hfill \end{array}$$ 15

where the inequalities follow from the definition (13) of ${\delta }_k$ and (14), respectively.

It remains to show that $\lambda$ is a certificate, i.e., $S_k>0$. But this is simple. Indeed, if $S_k=0$, then, taking $x:=\overline{x}$ in (15) and using (14), we get $0\ge {\sum }_{i=0}^{k-1}{\lambda }_i⟨g_i,x_i-\overline{x}⟩\ge r{\Gamma }_k$, which contradicts our assumption that $\lambda$ is a semicertificate, i.e., ${\Gamma }_k>0$. $\square$

According to Lemma 1, from the convergence rate of the gap ${\delta }_k({\lambda }^{(k)})$ to zero, we can easily obtain the corresponding convergence rate of the residual ${ϵ}_k({\lambda }^{(k)})$. In particular, to ensure that ${ϵ}_k({\lambda }^{(k)})\le ϵ$ for some $ϵ>0$, it suffices to make ${\delta }_k({\lambda }^{(k)})\le \delta (ϵ):=ϵr/(ϵ+V)$. For this reason, in the rest of this paper, we can focus our attention on studying the convergence rate only for the gap.

General algorithmic scheme

Consider the general scheme presented in Algorithm 1. This scheme works with an arbitrary oracle $\mathcal{G}:\mathbb{E}\to {\mathbb{E}}^{\ast }$ satisfying the following condition:

$$\begin{array}{c}\hfill \exists x^{\ast }\in B\left(x_0,R\right):⟨\mathcal{G}(x),x-x^{\ast }⟩\ge 0,\forall x\in \mathbb{E}.\end{array}$$ 16

The point $x^{\ast }$ from (16) is typically called a solution of our problem. For the general problem with convex structure, represented by the First-Order Oracle g and the Separation Oracle $g_Q$ for the solid Q, the oracle $\mathcal{G}$ is usually defined as follows: $\mathcal{G}(x):=g(x)$, if $x\in \mathrm{int}Q$, and $\mathcal{G}(x):=g_Q(x)$, otherwise. To ensure that (16) holds, the constant R needs to be chosen sufficiently big so that $Q\subseteq B\left(x_0,R\right)$.

Note that, in Algorithm 1, ${\omega }_k$ are strictly convex quadratic functions and ${\ell }_k$ are affine functions. Therefore, the sets ${\Omega }_k$ are certain ellipsoids and $L_k^{-}$ are certain halfspaces (possibly degenerate).

Let us show that Algorithm 1 is a cutting-plane scheme in which the sets ${\Omega }_k\cap L_k^{-}$ are the localizers of the solution $x^{\ast }$.

Lemma 2

In Algorithm 1, for all $k\ge 0$, we have $x^{\ast }\in {\Omega }_k\cap L_k^{-}$ and ${\widehat{Q}}_{k+1}\subseteq {\Omega }_{k+1}\cap L_{k+1}^{-}$, where ${\widehat{Q}}_{k+1}:=\{x\in {\Omega }_k\cap L_k^{-}:⟨g_k,x-x_k⟩\le 0\}$.

Proof

Let us prove the claim by induction. Clearly, ${\Omega }_0=B\left(x_0,R\right)$, $L_0^{-}=\mathbb{E}$, hence ${\Omega }_0\cap L_0^{-}=B\left(x_0,R\right)\ni x^{\ast }$ by (16). Suppose we have already proved that $x^{\ast }\in {\Omega }_k\cap L_k^{-}$ for some $k\ge 0$. Combining this with (16), we obtain $x^{\ast }\in {\widehat{Q}}_{k+1}$, so it remains to show that ${\widehat{Q}}_{k+1}\subseteq {\Omega }_{k+1}\cap L_{k+1}^{-}$. Let $x\in {\widehat{Q}}_{k+1}$ ($\subseteq {\Omega }_k\cap L_k^{-}$) be arbitrary. Note that $0\le ⟨g_k,x_k-x⟩\le U_k$. Hence, by (17), ${\ell }_{k+1}(x)\le {\ell }_k(x)\le 0$ and ${\omega }_{k+1}(x)\le {\omega }_k(x)\le \frac{1}{2}{R}^2$, which means that $x\in {\Omega }_{k+1}\cap L_{k+1}^{-}$. $\square$

Next, let us establish an important representation of the ellipsoids ${\Omega }_k$ via the functions ${\ell }_k$ and the test points $x_k$. For this, let us define $G_k:={\mathrm{\nabla }}^2{\omega }_k(0)$ for each $k\ge 0$. Observe that these operators satisfy the following simple relations (cf. (17)):

$$\begin{array}{c}\hfill G_0=B,G_{k+1}=G_k+b_k{g}_k{g}_k^{\ast },k\ge 0.\end{array}$$ 18

Also, let us define the sequence $R_k>0$ by the recurrence

$$\begin{array}{c}\hfill R_0=R,R_{k+1}^2=R_k^2+{(a_k+\frac{1}{2}{b}_k{U}_k)}^2\frac{\Vert g_k{\Vert }_{G_k}^2}{1+b_k{\Vert g_k\Vert }_{G_k}^2},k\ge 0.\end{array}$$ 19

Lemma 3

In Algorithm 1, for all $k\ge 0$, we have

$$\begin{array}{c}\hfill {\Omega }_k=\{x\in \mathbb{E}:-{\ell }_k(x)+\frac{1}{2}{\Vert x-x_k\Vert }_{G_k}^2\le \frac{1}{2}{R}_k^2\}.\end{array}$$

In particular, for all $k\ge 0$ and all $x\in {\Omega }_k\cap L_k^{-}$, we have ${\Vert x-x_k\Vert }_{G_k}\le R_k$.

Proof

Let ${\psi }_k:\mathbb{E}\to \mathbb{R}$ be the function ${\psi }_k(x):={\ell }_k(x)+{\omega }_k(x)$. Note that ${\psi }_k$ is a quadratic function with Hessian $G_k$ and minimizer $x_k$. Hence, for any $x\in \mathbb{E}$, we have

$$\begin{array}{c}\hfill {\psi }_k(x)={\psi }_k^{\ast }+\frac{1}{2}{\Vert x-x_k\Vert }_{G_k}^2,\end{array}$$ 20

where ${\psi }_k^{\ast }:={min}_{x\in \mathbb{E}}{\psi }_k(x)$.

Let us compute ${\psi }_k^{\ast }$. Combining (17), (18) and (20), for any $x\in \mathbb{E}$, we obtain

$$\begin{array}{cc}\hfill {\psi }_{k+1}(x)& ={\psi }_k(x)+(a_k+\frac{1}{2}{b}_k{U}_k)⟨g_k,x-x_k⟩+\frac{1}{2}{b}_k{⟨g_k,x-x_k⟩}^2\hfill \\ \hfill & ={\psi }_k^{\ast }+\frac{1}{2}{\Vert x-x_k\Vert }_{G_k}^2+(a_k+\frac{1}{2}{b}_k{U}_k)⟨g_k,x-x_k⟩+\frac{1}{2}{b}_k{⟨g_k,x-x_k⟩}^2\hfill \\ \hfill & ={\psi }_k^{\ast }+\frac{1}{2}{\Vert x-x_k\Vert }_{G_{k+1}}^2+(a_k+\frac{1}{2}{b}_k{U}_k)⟨g_k,x-x_k⟩,\hfill \end{array}$$ 21

Therefore,

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill {\psi }_{k+1}^{\ast }& ={\psi }_k^{\ast }-\frac{1}{2}{(a_k+\frac{1}{2}{b}_k{U}_k)}^2{\Vert g_k\Vert }_{G_{k+1}}^2\hfill \\ \hfill & ={\psi }_k^{\ast }-\frac{1}{2}{(a_k+\frac{1}{2}{b}_k{U}_k)}^2\frac{{\Vert g_k\Vert }_{G_k}^2}{1+b_k{\Vert g_k\Vert }_{G_k}^2},\hfill \end{array}\end{array}$$ 22

where the last identity follows from the fact that $G_{k+1}^{-1}{g}_k=G_k^{-1}{g}_k/(1+b_k{\Vert g_k\Vert }_{G_k}^2)$ (since $G_{k+1}{G}_k^{-1}{g}_k=(1+b_k{\Vert g_k\Vert }_{G_k}^2)g_k$ in view of (18)). Since (22) is true for any $k\ge 0$ and since ${\psi }_0^{\ast }=0$, we thus obtain, in view of (19),

$$\begin{array}{c}\hfill {\psi }_k^{\ast }=\frac{1}{2}(R^2-R_k^2).\end{array}$$ 23

Let $x\in {\Omega }_k$ be arbitrary. Using the definition of ${\psi }_k(x)$ and (23), we obtain

$$\begin{array}{c}\hfill -{\ell }_k(x)+\frac{1}{2}{\Vert x-x_k\Vert }_{G_k}^2={\omega }_k(x)-{\psi }_k^{\ast }={\omega }_k(x)+\frac{1}{2}(R_k^2-R^2).\end{array}$$

Thus, $x\in {\Omega }_k⟺{\omega }_k(x)\le \frac{1}{2}{R}^2⟺-{\ell }_k(x)+\frac{1}{2}{\Vert x-x_k\Vert }_{G_k}^2\le \frac{1}{2}{R}_k^2$. In particular, for any $x\in {\Omega }_k\cap L_k^{-}$, we have ${\ell }_k(x)\le 0$ and ${\Vert x-x_k\Vert }_{G_k}\le R_k$. $\square$

Lemma 3 has several consequences. First, we see that the localizers ${\Omega }_k\cap L_k^{-}$ are contained in the ellipsoids $\{x:{\Vert x-x_k\Vert }_{G_k}\le R_k\}$ whose centers are the test points $x_k$.

Second, we get a uniform upper bound on the function $-{\ell }_k$ on the ellipsoid ${\Omega }_k$: $-{\ell }_k(x)\le \frac{1}{2}{R}_k^2$ for all $x\in {\Omega }_k$. This observation leads us to the following definition of the sliding gap:

$$\begin{array}{c}\hfill {\Delta }_k:=\underset{x\in {\Omega }_k}{max}\frac{1}{{\Gamma }_k}[-{\ell }_k(x)]=\underset{x\in {\Omega }_k}{max}\frac{1}{{\Gamma }_k}\sum _{i=0}^{k-1}{a}_i⟨g_i,x_i-x⟩,k\ge 1,\end{array}$$ 24

provided that ${\Gamma }_k:={\sum }_{i=0}^{k-1}{a}_i{\Vert g_i\Vert }_{\ast }>0$. According to our observation, we have

$$\begin{array}{c}\hfill {\Delta }_k\le \frac{R_k^2}{2{\Gamma }_k}.\end{array}$$ 25

At the same time, ${\Delta }_k\ge 0$ in view of Lemma 2 and 16

Comparing the definition (24) of the sliding gap ${\Delta }_k$ with the definition (13) of the gap ${\delta }_k(a^{(k)})$ for the semicertificate $a^{(k)}:=(a_0,\cdots ,a_{k-1})$, we see that they are almost identical. The only difference between them is that the solid ${\Omega }_k$, over which the maximum is taken in the definition of the sliding gap, depends on the iteration counter k. This seems to be unfortunate because we cannot guarantee that each ${\Omega }_k$ contains the feasible set Q (as required in the definition of gap) even if so does the initial solid ${\Omega }_0=B\left(x_0,R\right)$. However, this problem can be dealt with. Namely, in Sect. 5, we will show that the semicertificate $a^{(k)}$ can be efficiently converted into another semicertificate ${\lambda }^{(k)}$ for which ${\delta }_k({\lambda }^{(k)})\le {\Delta }_k$ when taken over the initial solid $\Omega :={\Omega }_0$. Thus, the sliding gap ${\Delta }_k$ is a meaningful measure of convergence rate of Algorithm 1, and it makes sense to call the coefficients $a^{(k)}$ a preliminary semicertificate.

Let us now demonstrate that, for a suitable choice of the coefficients $a_k$ and $b_k$ in Algorithm 1, we can ensure that the sliding gap ${\Delta }_k$ converges to zero.

Remark 2

From now on, in order to avoid taking into account some trivial degenerate cases, it will be convenient to make the following minor technical assumption:

$$\begin{array}{c}\hfill \text{In Algorithm}1,g_k\ne 0\text{for all}k\ge 0.\end{array}$$

Indeed, when the oracle reports $g_k=0$ for some $k\ge 0$, it usually means that the test point $x_k$, at which the oracle was queried, is, in fact, an exact solution to our problem. For example, if the standard oracle for a problem with convex structure has reported $g_k=0$, we can terminate the method and return the certificate $\lambda :=(0,\cdots ,0,1)$ for which the residual ${ϵ}_k(\lambda )=0$.

Let us choose the coefficients $a_k$ and $b_k$ in the following way:

$$\begin{array}{c}\hfill a_k:=\frac{{\alpha }_{k}R+\frac{1}{2}\theta \gamma R_k}{{\Vert g_k\Vert }_{G_k}^{\ast }},b_k:=\frac{\gamma }{{\Vert g_k\Vert }_{G_k}^2},k\ge 0,\end{array}$$ 26

where ${\alpha }_k,\theta ,\gamma \ge 0$ are certain coefficients to be chosen later.

According to (25), to estimate the convergence rate of the sliding gap, we need to estimate the rate of growth of the coefficients $R_k$ and ${\Gamma }_k$ from above and below, respectively. Let us do this.

Lemma 4

In Algorithm 1 with parameters (26), for all $k\ge 0$, we have

$$\begin{array}{c}\hfill R_k^2\le {[q_c(\gamma )]}^k{C}_k{R}^2,\end{array}$$ 27

where $q_c(\gamma ):=1+\frac{c{\gamma }^2}{2(1+\gamma )}$, $c:=\frac{1}{2}(\tau +1){(\theta +1)}^2$, $C_k:=1+\frac{\tau +1}{\tau }{\sum }_{i=0}^{k-1}{\alpha }_i^2$ and $\tau >0$ can be chosen arbitrarily. Moreover, if ${\alpha }_k=0$ for all $k\ge 0$, then, $R_k^2={[q_c(\gamma )]}^k{R}^2$ for all $k\ge 0$ with $c:=\frac{1}{2}{(\theta +1)}^2$.

Proof

By the definition of $U_k$ and Lemma 3, we have

$$\begin{array}{c}\hfill U_k=\underset{x\in {\Omega }_k\cap L_k^{-}}{max}⟨g_k,x_k-x⟩\le \underset{{\Vert x-x_k\Vert }_{G_k}\le R_k}{max}⟨g_k,x_k-x⟩=R_k{\Vert g_k\Vert }_{G_k}^{\ast }.\end{array}$$ 28

At the same time, $U_k\ge 0$ in view of Lemma 2 and (16). Hence,

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill {(a_k+\frac{1}{2}{b}_k{U}_k)}^2\frac{{\Vert g_k\Vert }_{G_k}^2}{1+b_k{\Vert g_k\Vert }_{G_k}^2}& \le {(a_k+\frac{1}{2}{b}_k{R}_k{\Vert g_k\Vert }_{G_k}^{\ast })}^2\frac{{\Vert g_k\Vert }_{G_k}^2}{1+b_k{\Vert g_k\Vert }_{G_k}^2}\hfill \\ \hfill & =\frac{1}{1+\gamma }({\alpha }_{k}R+\frac{1}{2}(\theta +1)\gamma R_k{)}^2,\hfill \end{array}\end{array}$$

where the identity follows from (26). Combining this with (19), we obtain

$$\begin{array}{c}\hfill R_{k+1}^2\le R_k^2+\frac{1}{1+\gamma }({\alpha }_{k}R+\frac{1}{2}(\theta +1)\gamma R_k{)}^2.\end{array}$$ 29

Note that, for any ${\xi }_1,{\xi }_2\ge 0$ and any $\tau >0$, we have

$$\begin{array}{c}\hfill {({\xi }_1+{\xi }_2)}^2={\xi }_1^2+2{\xi }_1{\xi }_2+{\xi }_2^2\le \frac{\tau +1}{\tau }{\xi }_1^2+(\tau +1){\xi }_2^2=(\tau +1)(\frac{1}{\tau }{\xi }_1^2+{\xi }_2^2)\end{array}$$

(look at the minimum of the right-hand side in $\tau$). Therefore, for arbitrary $\tau >0$,

$$\begin{array}{c}\hfill R_{k+1}^2\le R_k^2+\frac{\tau +1}{1+\gamma }(\frac{1}{\tau }{\alpha }_k^2{R}^2+\frac{1}{4}{(\theta +1)}^2{\gamma }^2{R}_k^2)=q{R}_k^2+{\beta }_k{R}^2,\end{array}$$

where we denote $q:=q_c(\gamma )\ge 1$ and ${\beta }_k:=\frac{\tau +1}{\tau (1+\gamma )}{\alpha }_k^2$. Dividing both sides by $q^{k+1}$, we get

$$\begin{array}{c}\hfill \frac{R_{k+1}^2}{q^{k+1}}\le \frac{R_k^2}{q^k}+\frac{{\beta }_k{R}^2}{q^{k+1}}.\end{array}$$

Since this is true for any $k\ge 0$, we thus obtain, in view of (19), that

$$\begin{array}{c}\hfill \frac{R_k^2}{q^k}\le \frac{R_0^2}{q^0}+R^2\sum _{i=0}^{k-1}\frac{{\beta }_i}{q^{i+1}}=(1+\sum _{i=0}^{k-1}\frac{{\beta }_i}{q^{i+1}})R^2,\end{array}$$

Multiplying both sides by $q^k$ and using that $\frac{{\beta }_i}{q^{i+1}}\le \frac{\tau +1}{\tau }{\alpha }_i^2$, we come to (27).

When ${\alpha }_k=0$ for all $k\ge 0$, we have ${\ell }_k=0$ and $L_k^{-}=\mathbb{E}$ for all $k\ge 0$. Therefore, by Lemma 3, ${\Omega }_k=\{x:{\Vert x-x_k\Vert }_{G_k}\le R_k\}$ and hence (28) is, in fact, an equality. Consequently, (29) becomes $R_{k+1}^2=R_k^2+\frac{c{\gamma }^2}{2(1+\gamma )}{R}_k^2=q_c(\gamma )R_k^2$, where $c:=\frac{1}{2}{(\theta +1)}^2$. $\square$

Remark 3

From the proof, one can see that the quantity $C_k$ in Lemma 4 can be improved up to $C_k^{\prime }:=1+\frac{\tau +1}{\tau (1+\gamma )}{\sum }_{i=0}^{k-1}\frac{{\alpha }_i^2}{{[q_c(\gamma )]}^{i+1}}$.

Lemma 5

In Algorithm 1 with parameters (26), for all $k\ge 1$, we have

$$\begin{array}{c}\hfill {\Gamma }_k\ge R(\sum _{i=0}^{k-1}{\alpha }_i+\frac{1}{2}\theta \sqrt{\gamma n[{(1+\gamma )}^{k/n}-1]}).\end{array}$$ 30

Proof

By the definition of ${\Gamma }_k$ and (26), we have

$$\begin{array}{c}\hfill {\Gamma }_k=\sum _{i=0}^{k-1}{a}_i{\Vert g_i\Vert }_{\ast }=R\sum _{i=0}^{k-1}{\alpha }_i{\rho }_i+\frac{1}{2}\theta \gamma \sum _{i=0}^{k-1}{R}_i{\rho }_i,\end{array}$$

where ${\rho }_i:={\Vert g_i\Vert }_{\ast }/{\Vert g_i\Vert }_{G_i}^{\ast }$. Let us estimate each sum from below separately.

For the first sum, we can use the trivial bound ${\rho }_i\ge 1$, which is valid for any $i\ge 0$ (since $G_i⪰B$ in view of (18)). This gives us ${\sum }_{i=0}^{k-1}{\alpha }_i{\rho }_i\ge {\sum }_{i=0}^{k-1}{\alpha }_i$.

Let us estimate the second sum. According to (19), for any $i\ge 0$, we have $R_i\ge R$. Hence, ${\sum }_{i=0}^{k-1}{R}_i{\rho }_i\ge R{\sum }_{i=0}^{k-1}{\rho }_i\ge R{\left({\sum }_{i=0}^{k-1},{\rho }_i^2\right)}^{1/2}$ and it remains to lower bound ${\sum }_{i=0}^{k-1}{\rho }_i^2$. By 18 and 26, $G_0=B$ and $G_{i+1}=G_i+\gamma g_i{g}_i^{\ast }/{\Vert g_i\Vert }_{G_i}^2$ for all $i\ge 0$. Therefore,

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill \sum _{i=0}^{k-1}{\rho }_i^2& =\frac{1}{\gamma }\sum _{i=0}^{k-1}(\mathrm{tr}{G}_{i+1}-\mathrm{tr}{G}_i)=\frac{1}{\gamma }(\mathrm{tr}{G}_k-\mathrm{tr}B)=\frac{1}{\gamma }(\mathrm{tr}{G}_k-n)\hfill \\ \hfill & \ge \frac{n}{\gamma }[{(det{G}_k)}^{1/n}-1]=\frac{n}{\gamma }[{(1+\gamma )}^{k/n}-1],\hfill \end{array}\end{array}$$

where we have applied the arithmetic-geometric mean inequality. Combining the obtained estimates, we get (30). $\square$

Main instances of general scheme

Let us now consider several possibilities for choosing the coefficients ${\alpha }_k$, $\theta$ and $\gamma$ in (26).

Subgradient method

The simplest possibility is to choose

$$\begin{array}{c}\hfill {\alpha }_k>0,\theta :=0,\gamma :=0.\end{array}$$

In this case, $b_k=0$ for all $k\ge 0$, so $G_k=B$ and ${\omega }_k(x)={\omega }_0(x)=\frac{1}{2}{\Vert x\Vert }^2$ for all $x\in \mathbb{E}$ and all $k\ge 0$ (see (17) and (18)). Consequently, the new test points $x_{k+1}$ in Algorithm 1 are generated according to the following rule:

$$\begin{array}{c}\hfill x_{k+1}=\underset{x\in \mathbb{E}}{\mathrm{argmin}}[\sum _{i=0}^k{a}_i⟨g_i,x-x_i⟩+\frac{1}{2}{\Vert x\Vert }^2],\end{array}$$

where $a_i={\alpha }_{i}R/{\Vert g_i\Vert }_{\ast }$. Thus, Algorithm 1 is the Subgradient Method: $x_{k+1}=x_k-a_k{g}_k$.

In this example, each ellipsoid ${\Omega }_k$ is simply a ball: ${\Omega }_k=B\left(x_0,R\right)$ for all $k\ge 0$. Hence, the sliding gap ${\Delta }_k$, defined in (24), does not “slide” and coincides with the gap of the semicertificate $a:=(a_0,\cdots ,a_{k-1})$ on the solid $B\left(x_0,R\right)$:

$$\begin{array}{c}\hfill {\Delta }_k=\underset{x\in B\left(x_0,R\right)}{max}\frac{1}{{\Gamma }_k}\sum _{i=0}^{k-1}{a}_i⟨g_i,x_i-x⟩.\end{array}$$

In view of Lemmas 4 and 5, for all $k\ge 1$, we have

$$\begin{array}{c}\hfill R_k^2\le (1+\sum _{i=0}^{k-1}{\alpha }_i^2)R^2,{\Gamma }_k\ge R\sum _{i=0}^{k-1}{\alpha }_i\end{array}$$

(tend $\tau \to +\infty$ in Lemma 4). Substituting these estimates into (25), we obtain the following well-known estimate for the gap in the Subgradient Method:

$$\begin{array}{c}\hfill {\Delta }_k\le \frac{1+{\sum }_{i=0}^{k-1}{\alpha }_i^2}{2{\sum }_{i=0}^{k-1}{\alpha }_i}R.\end{array}$$

The standard strategies for choosing the coefficients ${\alpha }_i$ are as follows (see, e.g., sect. 3.2.3 in [19]):

1. We fix in advance the number of iterations $k\ge 1$ of the method and use constant coefficients ${\alpha }_i:=\frac{1}{\sqrt{k}}$, $0\le i\le k-1$. This corresponds to the so-called Short-Step Subgradient Method, for which we have

   $$\begin{array}{c}\hfill {\Delta }_k\le \frac{R}{\sqrt{k}}.\end{array}$$
2. Alternatively, we can use time-varying coefficients ${\alpha }_i:=\frac{1}{\sqrt{i+1}}$, $i\ge 0$. This approach does not require us to fix in advance the number of iterations k. However, the corresponding convergence rate estimate becomes slightly worse:

   $$\begin{array}{c}\hfill {\Delta }_k\le \frac{lnk+2}{2\sqrt{k}}R.\end{array}$$

   (Indeed, ${\sum }_{i=0}^{k-1}{\alpha }_i^2={\sum }_{i=1}^k\frac{1}{i}\le lnk+1$, while ${\sum }_{i=0}^{k-1}{\alpha }_i\ge \sqrt{k}$.)

Remark 4

If we allow projections onto the feasible set, then, for the resulting Subgradient Method with time-varying coefficients ${\alpha }_i$, one can establish the $O(1/\sqrt{k})$ convergence rate for the “truncated” gap

$$\begin{array}{c}\hfill {\Delta }_{k_0,k}:=\underset{x\in B\left(x_0,R\right)}{max}\frac{1}{{\Gamma }_{k_0,k}}\sum _{i=k_0}^k{a}_i⟨g_i,x_i-x⟩,\end{array}$$

where ${\Gamma }_{k_0,k}:={\sum }_{i=k_0}^k{a}_i{\Vert g_i\Vert }_{\ast }$, $k_0:=\lceil k/2\rceil$. For more details, see sect. 5.2.1 in [2] or sect. 3.1.1 in [12].

Standard ellipsoid method

Another extreme choice is the following one:

$$\begin{array}{c}\hfill {\alpha }_k:=0,\theta :=0,\gamma >0.\end{array}$$ 31

For this choice, we have $a_k=0$ for all $k\ge 0$. Hence, ${\ell }_k=0$ and $L_k^{-}=\mathbb{E}$ for all $k\ge 0$. Therefore, the localizers in this method are the following ellipsoids (see Lemma 3):

$$\begin{array}{c}\hfill {\Omega }_k\cap L_k^{-}={\Omega }_k=\{x\in \mathbb{E}:{\Vert x-x_k\Vert }_{G_k}\le R_k\},k\ge 0.\end{array}$$ 32

Observe that, in this example, ${\Gamma }_k\equiv {\sum }_{i=0}^{k-1}{a}_i{\Vert g_i\Vert }_{\ast }=0$ for all $k\ge 1$, so there is no preliminary semicertificate and the sliding gap is undefined. However, we can still ensure the convergence to zero of a certain meaningful measure of optimality, namely, the average radius of the localizers ${\Omega }_k$:

$$\begin{array}{c}\hfill \mathrm{avrad}{\Omega }_k:={(\mathrm{vol}{\Omega }_k)}^{1/n},k\ge 0.\end{array}$$ 33

Indeed, let us define the following functions for any real $c,p>0$:

$$\begin{array}{c}\hfill q_c(\gamma ):=1+\frac{c{\gamma }^2}{2(1+\gamma )},{\zeta }_{p,c}(\gamma ):=\frac{{[q_c(\gamma )]}^p}{1+\gamma },\gamma >0.\end{array}$$ 34

According to Lemma 4, for any $k\ge 0$, we have

$$\begin{array}{c}\hfill R_k^2={[q_{1/2}(\gamma )]}^k{R}^2.\end{array}$$ 35

At the same time, in view of (18) and (26), $det{G}_k={\prod }_{i=0}^{k-1}(1+b_i\Vert g_i{\Vert }_{G_i}^2)={(1+\gamma )}^k$ for all $k\ge 0$. Combining this with (32)–(34), we obtain, for any $k\ge 0$, that

$$\begin{array}{c}\hfill \mathrm{avrad}{\Omega }_k=\frac{R_k}{{(det{G}_k)}^{1/(2n)}}=\frac{{[q_{1/2}(\gamma )]}^{k/2}R}{{(1+\gamma )}^{k/(2n)}}={[{\zeta }_{n,1/2}(\gamma )]}^{k/(2n)}R.\end{array}$$ 36

Let us now choose $\gamma$ which minimizes $\mathrm{avrad}{\Omega }_k$. For such computations, the following auxiliary result is useful (see Sect. A for the proof).

Lemma 6

For any $c\ge 1/2$ and any $p\ge 2$, the function ${\zeta }_{p,c}$, defined in (34), attains its minimum at a unique point

$$\begin{array}{c}\hfill {\gamma }_c(p):=\frac{2}{\sqrt{c^2{p}^2-(2c-1)}+cp-1}\in \left[\frac{1}{\mathit{cp}},\frac{2}{\mathit{cp}}\right]\end{array}$$ 37

with the corresponding value ${\zeta }_{p,c}({\gamma }_c(p))\le e^{-1/(2cp)}$.

Applying Lemma 6 to 36, we see that the optimal value of $\gamma$ is

$$\begin{array}{c}\hfill \gamma :={\gamma }_{1/2}(n)=\frac{2}{n/2+n/2-1}=\frac{2}{n-1},\end{array}$$ 38

for which ${\zeta }_{n,1/2}(\gamma )\le e^{-1/n}$. With this choice of $\gamma$, we obtain, for all $k\ge 0$, that

$$\begin{array}{c}\hfill \mathrm{avrad}{\Omega }_k\le e^{-k/(2n^2)}R.\end{array}$$ 39

One can check that Algorithm 1 with parameters (26), (31) and (38) is, in fact, the standard Ellipsoid Method (see Remark 6).

Ellipsoid method with preliminary semicertificate

As we have seen, we cannot measure the convergence rate of the standard Ellipsoid Method using the sliding gap because there is no preliminary semicertificate in this method. Let us present a modification of the standard Ellipsoid Method which does not have this drawback but still enjoys the same convergence rate as the original method (up to some absolute constants).

For this, let us choose the coefficients in the following way:

$$\begin{array}{c}\hfill {\alpha }_k:=0,\theta :=\sqrt{2}-1(\approx 0.41),\gamma >0.\end{array}$$ 40

Then, in view of Lemma 4, for all $k\ge 0$, we have

$$\begin{array}{c}\hfill R_k^2={[q_1(\gamma )]}^k{R}^2,\end{array}$$ 41

Also, by Lemma 5, ${\Gamma }_k\ge \frac{1}{2}\theta R\sqrt{\gamma n[{(1+\gamma )}^{k/n}-1]}$ for all $k\ge 1$. Thus, for each $k\ge 1$, we obtain the following estimate for the sliding gap (see (25)):

$$\begin{array}{c}\hfill {\Delta }_k\le \frac{{[q_1(\gamma )]}^{k}R}{\theta \sqrt{\gamma n[{(1+\gamma )}^{k/n}-1]}}=\frac{1}{\theta {\kappa }_k(\gamma ,n)}{[{\zeta }_{2n,1}(\gamma )]}^{k/(2n)}R,\end{array}$$ 42

where ${\kappa }_k(\gamma ,n):=\sqrt{\gamma n(1-\frac{1}{{(1+\gamma )}^{k/n}})}$ and ${\zeta }_{2n,1}(\gamma )$ is defined in (34).

Note that the main factor in estimate (42) is ${[{\zeta }_{2n,1}(\gamma )]}^{k/(2n)}$. Let us choose $\gamma$ by minimizing this expression. Applying Lemma 6, we obtain

$$\begin{array}{c}\hfill \gamma :={\gamma }_1(2n)\in \left[\frac{1}{2n},\frac{1}{n}\right].\end{array}$$ 43

Theorem 1

In Algorithm 1 with parameters (26), (40), (43), for all $k\ge 1$,

$$\begin{array}{c}\hfill {\Delta }_k\le 6e^{-k/(8n^2)}R.\end{array}$$

Proof

Suppose $k\ge n^2$. According to Lemma 6, we have ${\zeta }_{2n,1}(\gamma )\le e^{-1/(4n)}$. Hence, by (42), ${\Delta }_k\le \frac{1}{\theta {\kappa }_k(\gamma ,n)}{e}^{-k/(8n^2)}R$. It remains to estimate from below $\theta {\kappa }_k(\gamma ,n)$.

Since $k\ge n^2$, we have ${(1+\gamma )}^{k/n}\ge {(1+\gamma )}^n\ge 1+\gamma n$. Hence, ${\kappa }_k(\gamma ,n)\ge \frac{\gamma n}{\sqrt{1+\gamma n}}$. Note that the function $\tau \mapsto \frac{\tau }{\sqrt{1+\tau }}$ is increasing on ${\mathbb{R}}_{+}$. Therefore, using (43), we obtain ${\kappa }_k(\gamma ,n)\ge \frac{1/2}{\sqrt{1+1/2}}=\frac{1}{\sqrt{6}}$. Thus, $\theta {\kappa }_k(\gamma ,n)\ge \frac{\sqrt{2}-1}{\sqrt{6}}\ge \frac{1}{6}$ for our choice of $\theta$.

Now suppose $k\le n^2$. Then, $6e^{-k/(8n^2)}\ge 6e^{-1/8}\ge 5$. Therefore, it suffices to prove that ${\Delta }_k\le 5R$ or, in view of (24), that $⟨g_i,x_i-x⟩\le 5R{\Vert g_i\Vert }_{\ast }$, where $x\in {\Omega }_k\cap L_k^{-}$ and $0\le i\le k-1$ are arbitrary. Note that $⟨g_i,x_i-x⟩\le {\Vert g_i\Vert }_{G_i}^{\ast }{\Vert x_i-x\Vert }_{G_i}\le {\Vert g_i\Vert }_{\ast }{\Vert x_i-x\Vert }_{G_i}$ since $G_i⪰B$ (see (18)). Hence, it remains to prove that ${\Vert x_i-x\Vert }_{G_i}\le 5R$.

Recall from (18) and (19) that $G_i⪯G_k$ and $R_i\le R_k$. Therefore,

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill {\Vert x_i-x\Vert }_{G_i}& \le {\Vert x_i-x^{\ast }\Vert }_{G_i}+{\Vert x^{\ast }-x\Vert }_{G_i}\le {\Vert x_i-x^{\ast }\Vert }_{G_i}+{\Vert x^{\ast }-x\Vert }_{G_k}\hfill \\ \hfill & \le {\Vert x_i-x^{\ast }\Vert }_{G_i}+{\Vert x_k-x^{\ast }\Vert }_{G_k}+{\Vert x_k-x\Vert }_{G_k}\le R_i+2R_k\le 3R_k,\hfill \end{array}\end{array}$$

where the penultimate inequality follows from Lemma 2 and 3. According to (41), $R_k={[q_1(\gamma )]}^{k/2}R\le {[q_1(\gamma )]}^{n^2/2}R$ (recall that $q_1(\gamma )\ge 1$). Thus, it remains to show that $3{[q_1(\gamma )]}^{n^2/2}\le 5$. But this is immediate. Indeed, by (34) and (43), we have ${[q_1(\gamma )]}^{n^2/2}\le e^{n^2{\gamma }^2/(4(1+\gamma ))}\le e^{1/4}$, so $3{[q_1(\gamma )]}^{n^2/2}\le 3e^{1/4}\le 5$. $\square$

Subgradient ellipsoid method

The previous algorithm still shares the drawback of the original Ellipsoid Method, namely, it does not work when $n\to \infty$. To eliminate this drawback, let us choose ${\alpha }_k$ similarly to how this is done in the Subgradient Method.

Consider the following choice of parameters:

$$\begin{array}{c}\hfill {\alpha }_i:={\beta }_i\sqrt{\frac{\theta }{\theta +1}},\theta :=\sqrt[3]{2}-1(\approx 0.26),\gamma :={\gamma }_1(2n)\in \left[\frac{1}{2n},\frac{1}{n}\right],\end{array}$$ 44

where ${\beta }_i>0$ are certain coefficients (to be specified later) and ${\gamma }_1(2n)$ is defined in (37).

Theorem 2

In Algorithm 1 with parameters (26) and (44), where ${\beta }_0\ge 1$, we have, for all $k\ge 1$,

$$\begin{array}{c}\hfill {\Delta }_k\le \left\{\begin{array}{cc}\frac{2}{{\sum }_{i=0}^{k-1}{\beta }_i}(1+{\sum }_{i=0}^{k-1}{\beta }_i^2)R,\hfill & \text{if}k\le n^2,\hfill \\ 6e^{-k/(8n^2)}(1+{\sum }_{i=0}^{k-1}{\beta }_i^2)R,\hfill & \text{if}k\ge n^2.\hfill \end{array}\right)\end{array}$$ 45

Proof

Applying Lemma 4 with $\tau :=\theta$ and using (44), we obtain

$$\begin{array}{c}\hfill R_k^2\le {[q_1(\gamma )]}^k{C}_k{R}^2,C_k=1+\sum _{i=0}^{k-1}{\beta }_i^2.\end{array}$$ 46

At the same time, by Lemma 5, we have

$$\begin{array}{c}\hfill {\Gamma }_k\ge R(\sqrt{\frac{\theta }{\theta +1}}\sum _{i=0}^{k-1}{\beta }_i+\frac{1}{2}\theta \sqrt{\gamma n[{(1+\gamma )}^{k/n}-1]}).\end{array}$$ 47

Note that $\frac{1}{2}\theta \sqrt{\gamma n}\le \frac{1}{2}\theta \le \sqrt{\theta /(\theta +1)}$ by (44). Since ${\beta }_0\ge 1$, we thus obtain

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill {\Gamma }_k& \ge \frac{1}{2}R\theta \sqrt{\gamma n}(1+\sqrt{{(1+\gamma )}^{k/n}-1})\ge \frac{1}{2}R\theta \sqrt{\gamma n}{(1+\gamma )}^{k/(2n)}\hfill \\ \hfill & \ge \frac{1}{2\sqrt{2}}R\theta {(1+\gamma )}^{k/(2n)}\ge \frac{1}{12}R{(1+\gamma )}^{k/(2n)},\hfill \end{array}\end{array}$$ 48

where the last two inequalities follow from (44). Therefore, by (25), (46) and (48),

$$\begin{array}{c}\hfill {\Delta }_k\le \frac{R_k^2}{2{\Gamma }_k}\le 6\frac{{[q_1(\gamma )]}^k}{{(1+\gamma )}^{k/(2n)}}{C}_{k}R=6{[{\zeta }_{2n,1}(\gamma )]}^{k/(2n)}{C}_{k}R,\end{array}$$

where ${\zeta }_{2n,1}(\gamma )$ is defined in (34). Observe that, for our choice of $\gamma$, by Lemma 6, we have ${\zeta }_{2n,1}(\gamma )\le e^{-1/(4n)}$. This proves the second estimate3 in (45).

On the other hand, dropping the second term in (47), we can write

$$\begin{array}{c}\hfill {\Gamma }_k\ge R\sqrt{\frac{\theta }{\theta +1}}\sum _{i=0}^{k-1}{\beta }_i.\end{array}$$ 49

Suppose $k\le n^2$. Then, from (34) and (44), it follows that

$$\begin{array}{c}\hfill {[q_1(\gamma )]}^k\le {[q_1(\gamma )]}^{n^2}\le e^{{\gamma }^2{n}^2/(2(1+\gamma ))}\le \sqrt{e}.\end{array}$$

Hence, by (46), $R_k\le \sqrt{e}{C}_k{R}^2$. Combining this with (25) and (49), we obtain

$$\begin{array}{c}\hfill {\Delta }_k\le \frac{1}{2}\sqrt{\frac{e(\theta +1)}{\theta }}\frac{1}{{\sum }_{i=0}^{k-1}{\beta }_i}{C}_{k}R.\end{array}$$

By numerical evaluation, one can verify that, for our choice of $\theta$, we have $\frac{1}{2}\sqrt{\frac{e(\theta +1)}{\theta }}\le 2$. This proves the first estimate in (45). $\square$

Exactly as in the Subgradient Method, we can use the following two strategies for choosing the coefficients ${\beta }_i$:

1. We fix in advance the number of iterations $k\ge 1$ of the method and use constant coefficients ${\beta }_i:=\frac{1}{\sqrt{k}}$, $0\le i\le k-1$. In this case,

   $$\begin{array}{c}\hfill {\Delta }_k\le \left\{\begin{array}{cc}4R/\sqrt{k},\hfill & \text{if}k\le n^2,\hfill \\ 12R{e}^{-k/(8n^2)},\hfill & \text{if}k\ge n^2.\hfill \end{array}\right)\end{array}$$ 50
2. We use time-varying coefficients ${\beta }_i:=\frac{1}{\sqrt{i+1}}$, $i\ge 0$. In this case,

   $$\begin{array}{c}\hfill {\Delta }_k\le \left\{\begin{array}{cc}2(lnk+2)R/\sqrt{k},\hfill & \text{if}k\le n^2,\hfill \\ 6(lnk+2)R{e}^{-k/(8n^2)},\hfill & \text{if}k\ge n^2.\hfill \end{array}\right)\end{array}$$

Let us discuss convergence rate estimate (50). Up to absolute constants, this estimate is exactly the same as in the Subgradient Method when $k\le n^2$ and as in the Ellipsoid Method when $k\ge n^2$. In particular, when $n\to \infty$, we recover the convergence rate of the Subgradient Method.

To provide a better interpretation of the obtained results, let us compare the convergence rates of the Subgradient and Ellipsoid methods:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill \text{Subgradient Method:}& 1/\sqrt{k}\hfill \\ \hfill \text{Ellipsoid Method:}& e^{-k/(2n^2)}.\hfill \end{array}\end{array}$$

To compare these rates, let us look at their squared ratio:

$$\begin{array}{c}\hfill {\rho }_k:=(\frac{1/\sqrt{k}}{e^{-k/(2n^2)}}{)}^2=\frac{e^{k/n^2}}{k}.\end{array}$$

Let us find out for which values of k the rate of the Subgradient Method is better than that of the Ellipsoid Method and vice versa. We assume that $n\ge 2$.

Note that the function $\tau \mapsto e^{\tau }/\tau$ is strictly decreasing on $\left(0,1\right]$ and strictly increasing on $\left[1,+\infty \right)$ (indeed, its derivative equals $e^{\tau }(\tau -1)/{\tau }^2$). Hence, ${\rho }_k$ is strictly decreasing in k for $1\le k\le n^2$ and strictly increasing in k for $k\ge n^2$. Since $n\ge 2$, we have ${\rho }_2=e^{2/n^2}/2\le e^{1/2}/2\le 1$. At the same time, ${\rho }_k\to +\infty$ when $k\to \infty$. Therefore, there exists a unique integer $K_0\ge 2$ such that ${\rho }_k\le 1$ for all $k\le K_0$ and ${\rho }_k\ge 1$ for all $k\ge K_0$.

Let us estimate $K_0$. Clearly, for any $n^2\le k\le n^{2}ln(2n)$, we have

$$\begin{array}{c}\hfill {\rho }_k\le \frac{e^{n^{2}ln(2n)/n^2}}{n^{2}ln(2n)}=\frac{2}{nln(2n)}\le 1,\end{array}$$

while, for any $k\ge 3n^{2}ln(2n)$, we have

$$\begin{array}{c}\hfill {\rho }_k\ge \frac{e^{3n^{2}ln(2n)}/n^2}{3n^{2}ln(2n)}=\frac{{(2n)}^3}{3n^{2}ln(2n)}=\frac{8n}{3ln(2n)}\ge 1.\end{array}$$

Hence,

$$\begin{array}{c}\hfill n^{2}ln(2n)\le K_0\le 3n^{2}ln(2n).\end{array}$$

Thus, up to an absolute constant, $n^{2}ln(2n)$ is the switching moment, starting from which the rate of the Ellipsoid Method becomes better than that of the Subgradient Method.

Returning to our obtained estimate (50), we see that, ignoring absolute constants and ignoring the “small” region of the values of k between $n^2$ and $n^{2}lnn$, our convergence rate is basically the best of the corresponding convergence rates of the Subgradient and Ellipsoid methods.

Constructing accuracy semicertificate

Let us show how to convert a preliminary accuracy semicertificate, produced by Algorithm 1, into a semicertificate whose gap on the initial solid is upper bounded by the sliding gap. The key ingredient here is the following auxiliary algorithm which was first proposed in [16] for building accuracy certificates in the standard Ellipsoid Method.

Augmentation algorithm

Let $k\ge 0$ be an integer and let $Q_0,\cdots ,Q_k$ be solids in $\mathbb{E}$ such that

$$\begin{array}{c}\hfill {\widehat{Q}}_i:=\{x\in Q_i:⟨g_i,x-x_i⟩\le 0\}\subseteq Q_{i+1},0\le i\le k-1,\end{array}$$ 51

where $x_i\in \mathbb{E}$, $g_i\in {\mathbb{E}}^{\ast }$. Further, suppose that, for any $s\in {\mathbb{E}}^{\ast }$ and any $0\le i\le k-1$, we can compute a dual multiplier $\mu \ge 0$ such that

$$\begin{array}{c}\hfill \underset{x\in {\widehat{Q}}_i}{max}⟨s,x⟩=\underset{x\in Q_i}{max}[⟨s,x⟩+\mu ⟨g_i,x_i-x⟩]\end{array}$$ 52

(provided that certain regularity conditions hold). Let us abbreviate any solution $\mu$ of this problem by $\mu (s,Q_i,x_i,g_i)$.

Consider now the following routine.

Lemma 7

Let ${\mu }_0,\cdots ,{\mu }_{k-1}\ge 0$ be generated by Algorithm 2. Then,

$$\begin{array}{c}\hfill \underset{x\in Q_0}{max}[⟨s_k,x⟩+\sum _{i=0}^{k-1}{\mu }_i⟨g_i,x_i-x⟩]\le \underset{x\in Q_k}{max}⟨s_k,x⟩.\end{array}$$

Proof

Indeed, at every iteration $i=k-1,\cdots ,0$, we have

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill \underset{x\in Q_{i+1}}{max}⟨s_{i+1},x⟩& \ge \underset{x\in {\widehat{Q}}_i}{max}⟨s_{i+1},x⟩=\underset{x\in Q_i}{max}[⟨s_{i+1},x⟩+{\mu }_i⟨g_i,x_i-x⟩]\hfill \\ \hfill & =\underset{x\in Q_i}{max}⟨s_i,x⟩+{\mu }_i⟨g_i,x_i⟩.\hfill \end{array}\end{array}$$

Summing up these inequalities for $i=0,\cdots ,k-1$, we obtain

$$\begin{array}{c}\hfill \underset{x\in Q_k}{max}⟨s_k,x⟩\ge \underset{x\in Q_0}{max}⟨s_0,x⟩+\sum _{i=0}^{k-1}{\mu }_i⟨g_i,x_i⟩=\underset{x\in Q_0}{max}[⟨s_k,x⟩+\sum _{i=0}^{k-1}⟨g_i,x_i-x⟩],\end{array}$$

where the identity follows from the fact that $s_0=s_k-{\sum }_{i=0}^{k-1}{\mu }_i{g}_i$. $\square$

Methods with preliminary certificate

Let us apply the Augmentation Algorithm for building an accuracy semicertificate for Algorithm 1. We only consider those instances for which ${\Gamma }_k:={\sum }_{i=0}^{k-1}{a}_i{\Vert g_i\Vert }_{\ast }>0$ so that the sliding gap ${\Delta }_k$ is well-defined:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill {\Delta }_k:=\underset{x\in {\Omega }_k}{max}\frac{1}{{\Gamma }_k}[-{\ell }_k(x)]& =\underset{x\in {\Omega }_k\cap L_k^{-}}{max}\frac{1}{{\Gamma }_k}[-{\ell }_k(x)]\hfill \\ \hfill & =\underset{x\in {\Omega }_k\cap L_k^{-}}{max}\frac{1}{{\Gamma }_k}\sum _{i=0}^{k-1}{a}_i⟨g_i,x_i-x⟩.\hfill \end{array}\end{array}$$

Recall that the vector $a:=(a_0,\cdots ,a_{k-1})$ is called a preliminary semicertificate.

For technical reasons, it will be convenient to add the following termination criterion into Algorithm 1:

$$\begin{array}{c}\hfill \text{Terminate Algorithm}1\text{at Step}2\text{if}{U}_k\le \delta {\Vert g_k\Vert }_{\ast },\end{array}$$ 53

where $\delta >0$ is a fixed constant. Depending on whether this termination criterion has been satisfied at iteration k, we call it a terminal or nonterminal iteration, respectively.

Remark 5

In practice, one can set $\delta$ to an arbitrarily small value (within machine precision) if the desired target accuracy is unknown. As can be seen from the subsequent discussion, the main purpose of the termination criterion (53) is to ensure that $U_k$ never becomes equal to zero during the iterations of Algorithm 1. This guarantees the existence of dual multiplier in (52) for any $s\in {\mathbb{E}}^{\ast }$ at every nonterminal iteration. The case $U_k=0$ corresponds to the degenerate situation when Algorithm 1 has “accidentally” found an exact solution.

Let $k\ge 1$ be an iteration of Algorithm 1. According to Lemma 2, the sets $Q_i:={\Omega }_i\cap L_i^{-}$ satisfy (51). Since the method has not been terminated during the course of the previous iterations, we have4$U_i>0$ for all $0\le i\le k-1$. Therefore, for any $0\le i\le k-1$, there exists $x\in Q_i$ such that $⟨g_i,x-x_i⟩<0$. This guarantees the existence of dual multiplier in (52).

Let us apply Algorithm 2 to $s_k:=-{\sum }_{i=0}^{k-1}{a}_i{g}_i$ in order to obtain dual multipliers $\mu :=({\mu }_0,\cdots ,{\mu }_{k-1})$. From Lemma 7, it follows that

$$\begin{array}{c}\hfill \underset{x\in B\left(x_0,R\right)}{max}\sum _{i=0}^{k-1}(a_i+{\mu }_i)⟨g_i,x_i-x⟩\le \underset{x\in Q_k}{max}\sum _{i=0}^{k-1}{a}_i⟨g_i,x_i-x⟩={\Gamma }_k{\Delta }_k,\end{array}$$

(note that $Q_0={\Omega }_0\cap L_0^{-}=B\left(x_0,R\right)$). Thus, defining $\lambda :=a+\mu$, we obtain ${\Gamma }_k(\lambda )\equiv {\sum }_{i=0}^{k-1}{\lambda }_i{\Vert g_i\Vert }_{\ast }\ge {\sum }_{i=0}^{k-1}{a}_i{\Vert g_i\Vert }_{\ast }\equiv {\Gamma }_k>0$ and

$$\begin{array}{c}\hfill {\delta }_k(\lambda )\equiv \underset{x\in B\left(x_0,R\right)}{max}\frac{1}{{\Gamma }_k(\lambda )}\sum _{i=0}^{k-1}{\lambda }_i⟨g_i,x_i-x⟩\le \frac{{\Gamma }_k}{{\Gamma }_k(\lambda )}{\Delta }_k\le {\Delta }_k,\end{array}$$

Thus, $\lambda$ is a semicertificate whose gap on $B\left(x_0,R\right)$ is bounded by the sliding gap ${\Delta }_k$.

If $k\ge 0$ is a terminal iteration, then, by the termination criterion and the definition of $U_k$ (see Algorithm 1), we have ${max}_{x\in {\Omega }_k\cap L_k^{-}}\frac{1}{{\Vert g_k\Vert }_{\ast }}⟨g_k,x_k-x⟩\le \delta$. In this case, we apply Algorithm 2 to $s_k:=-g_k$ to obtain dual multipliers ${\mu }_0,\cdots ,{\mu }_{k-1}$. By the same reasoning as above but with the vector $(0,\cdots ,0,1)$ instead of $(a_0,\cdots ,a_{k-1})$, we can obtain that ${\delta }_{k+1}(\lambda )\le \delta$, where $\lambda :=({\mu }_0,\cdots ,{\mu }_{k-1},1)$.

Standard ellipsoid method

In the standard Ellipsoid Method, there is no preliminary semicertificate. Therefore, we cannot apply the above procedure. However, in this method, it is still possible to generate an accuracy semicertificate, although the corresponding procedure is slightly more involved. Let us now briefly describe this procedure and discuss how it differs from the previous approach. For details, we refer the reader to [16].

Let $k\ge 1$ be an iteration of the method. There are two main steps. The first step is to find a direction $s_k$, in which the “width” of the ellipsoid ${\Omega }_k$ (see (32)) is minimal:

$$\begin{array}{c}\hfill s_k:=\underset{{\Vert s\Vert }_{\ast }=1}{\mathrm{argmin}}\underset{x,y\in {\Omega }_k}{max}⟨s,x-y⟩=\underset{{\Vert s\Vert }_{\ast }=1}{\mathrm{argmin}}[\underset{x\in {\Omega }_k}{max}⟨s,x⟩-\underset{x\in {\Omega }_k}{min}⟨s,x⟩].\end{array}$$

It is not difficult to see that $s_k$ is given by the unit eigenvector5 of the operator $G_k$, corresponding to the largest eigenvalue. For the corresponding minimal “width” of the ellipsoid, we have the following bound via the average radius:

$$\begin{array}{c}\hfill \underset{x,y\in {\Omega }_k}{max}⟨s_k,x-y⟩\le {\rho }_k,\end{array}$$ 54

where ${\rho }_k:=2\mathrm{avrad}{\Omega }_k$. Recall that $\mathrm{avrad}{\Omega }_k\le e^{-k/(2n^2)}R$ in view of (39).

At the second step, we apply Algorithm 2 two times with the sets $Q_i:={\Omega }_i$: first, to the vector $s_k$ to obtain dual multipliers $\mu :=({\mu }_0,\cdots ,{\mu }_{k-1})$ and then to the vector $-s_k$ to obtain dual multipliers ${\mu }^{\prime }:=({\mu }_0^{\prime },\cdots ,{\mu }_{k-1}^{\prime })$. By Lemma 7 and (54), we have

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill \underset{x\in B\left(x_0,R\right)}{max}[⟨s_k,x-x_k⟩+\sum _{i=0}^{k-1}{\mu }_i⟨g_i,x_i-x⟩]& \le \underset{x\in {\Omega }_k}{max}⟨s_k,x-x_k⟩\le {\rho }_k,\hfill \\ \hfill \underset{x\in B\left(x_0,R\right)}{max}[⟨s_k,x_k-x⟩+\sum _{i=0}^{k-1}{\mu }_i^{\prime }⟨g_i,x_i-x⟩]& \le \underset{x\in {\Omega }_k}{max}⟨s_k,x_k-x⟩\le {\rho }_k\hfill \end{array}\end{array}$$

(note that $Q_0={\Omega }_0=B\left(x_0,R\right)$). Consequently, for $\lambda :=\mu +{\mu }^{\prime }$, we obtain

$$\begin{array}{c}\hfill \underset{x\in B\left(x_0,R\right)}{max}\sum _{i=0}^{k-1}{\lambda }_i⟨g_i,x_i-x⟩\le 2{\rho }_k.\end{array}$$

Finally, one can show that

$$\begin{array}{c}\hfill {\Gamma }_k(\lambda )\equiv \sum _{i=0}^{k-1}{\lambda }_i{\Vert g_i\Vert }_{\ast }\ge \frac{r-{\rho }_k}{D},\end{array}$$

where D is the diameter of Q and r is the maximal of the radii of Euclidean balls contained in Q. Thus, whenever ${\rho }_k<r$, $\lambda$ is a semicertificate with the following gap on $B\left(x_0,R\right)$:

$$\begin{array}{c}\hfill {\delta }_k(\lambda )\equiv \underset{x\in B\left(x_0,R\right)}{max}\frac{1}{{\Gamma }_k(\lambda )}\sum _{i=0}^{k-1}{\lambda }_i⟨g_i,x_i-x⟩\le \frac{2{\rho }_{k}D}{r-{\rho }_k}.\end{array}$$

Compared to the standard Ellipsoid Method, we see that, in the Subgradient Ellipsoid methods, the presence of the preliminary semicertificate removes the necessity in finding the minimal-“width” direction and requires only one run of the Augmentation Algorithm.

Implementation details

Explicit representations

In the implementation of Algorithm 1, instead of the operators $G_k$, it is better to work with their inverses $H_k:=G_k^{-1}$. Applying the Sherman-Morrison formula to (18), we obtain the following update rule for $H_k$:

$$\begin{array}{c}\hfill H_{k+1}=H_k-\frac{b_k{H}_k{g}_k{g}_k^{\ast }{H}_k}{1+b_k⟨g_k,H_k{g}_k⟩},k\ge 0.\end{array}$$ 55

Let us now obtain an explicit formula for the next test point $x_{k+1}$. This has already been partly done in the proof of Lemma 3. Indeed, recall that $x_{k+1}$ is the minimizer of the function ${\psi }_{k+1}(x)$. From (21), we see that $x_{k+1}=x_k-(a_k+\frac{1}{2}{b}_k{U}_k)H_{k+1}{g}_k$. Combining it with (55), we obtain

$$\begin{array}{c}\hfill x_{k+1}=x_k-\frac{a_k+\frac{1}{2}{b}_k{U}_k}{1+b_k⟨g_k,H_k{g}_k⟩}{H}_k{g}_k,k\ge 0.\end{array}$$ 56

Finally, one can obtain the following explicit representations for $L_k^{-}$ and ${\Omega }_k$:

$$\begin{array}{c}\hfill L_k^{-}=\{x\in \mathbb{E}:⟨c_k,x⟩\le {\sigma }_k\},{\Omega }_k=\{x\in \mathbb{E}:{\Vert x-z_k\Vert }_{H_k^{-1}}^2\le D_k\},\end{array}$$ 57

where, for any $k\ge 0$,

$$\begin{array}{c}\hfill \begin{array}{c}\hfill c_0:=0,{\sigma }_0:=0,c_{k+1}:=c_k+a_k{g}_k,{\sigma }_{k+1}:={\sigma }_k+a_k⟨g_k,x_k⟩,\\ \hfill z_k:=x_k-H_k{c}_k,D_k:=R_k^2+2({\sigma }_k-⟨c_k,x_k⟩)+⟨c_k,H_k{c}_k⟩.\end{array}\end{array}$$ 58

Indeed, recalling the definition of functions ${\ell }_k$, we see that ${\ell }_k(x)=⟨c_k,x⟩-{\sigma }_k$ for all $x\in \mathbb{E}$. Therefore, $L_k^{-}\equiv \{x:{\ell }_k(x)\le 0\}=\{x:⟨c_k,x⟩\le {\sigma }_k\}$. Further, by Lemma 3, ${\Omega }_k=\{x:⟨c_k,x⟩+\frac{1}{2}{\Vert x-x_k\Vert }_{G_k}^2\le \frac{1}{2}{R}_k^2+{\sigma }_k\}$. Note that $⟨c_k,x⟩+\frac{1}{2}{\Vert x-x_k\Vert }_{G_k}^2=\frac{1}{2}{\Vert x-z_k\Vert }_{G_k}^2+⟨c_k,x_k⟩-\frac{1}{2}{\Vert c_k\Vert }_{G_k}^2$ for any $x\in \mathbb{E}$. Hence, ${\Omega }_k=\{x:\frac{1}{2}{\Vert x-z_k\Vert }_{G_k}^2\le \frac{1}{2}{D}_k\}$.

Remark 6

Now we can justify the claim made in Sect. 4.2 that Algorithm 1 with parameters (26), (31) and (38) is the standard Ellipsoid Method. Indeed, from (26) and (32), we see that $b_k=\frac{\gamma }{⟨g_k,H_k{g}_k⟩}$ and $U_k=R_k{⟨g_k,H_k{g}_k⟩}^{1/2}$. Also, in view of (38), $\frac{\gamma }{1+\gamma }=\frac{2}{n+1}$. Hence, by (56) and (55),

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill x_{k+1}& =x_k-\frac{R_k}{n+1}\frac{H_k{g}_k}{{⟨g_k,H_k{g}_k⟩}^{1/2}},\hfill \\ \hfill H_{k+1}& =H_k-\frac{2}{n+1}\frac{H_k{g}_k{g}_k^{\ast }{H}_k}{⟨g_k,H_k{g}_k⟩},k\ge 0.\hfill \end{array}\end{array}$$ 59

Further, according to (35) and (38), for any $k\ge 0$, we have $R_k^2=q^k{R}^2$, where $q=1+\frac{1}{(n-1)(n+1)}=\frac{n^2}{n^2-1}$. Thus, method (59) indeed coincides6 with the standard Ellipsoid Method (2) under the change of variables $W_k:=R_k^2{H}_k$.

Computing support function

To calculate $U_k$ in Algorithm 1, we need to compute the following quantity (see (57)):

$$\begin{array}{c}\hfill U_k=\underset{x}{max}\{⟨g_k,x_k-x⟩:{\Vert x-z_k\Vert }_{H_k^{-1}}^2\le D_k,⟨c_k,x⟩\le {\sigma }_k\}.\end{array}$$

Let us discuss how to do this.

First, let us introduce the following support function to simplify our notation:

$$\begin{array}{c}\hfill \xi (H,s,a,\beta ):=\underset{x}{max}\{⟨s,x⟩:{\Vert x\Vert }_{H^{-1}}^2\le 1,⟨a,x⟩\le \beta \},\end{array}$$

where $H:{\mathbb{E}}^{\ast }\to \mathbb{E}$ is a self-adjoint positive definite linear operator, $s,a\in {\mathbb{E}}^{\ast }$ and $\beta \in \mathbb{R}$. In this notation, assuming that $D_k>0$, we have

$$\begin{array}{c}\hfill U_k=⟨g_k,x_k-z_k⟩+\xi (D_k{H}_k,-g_k,c_k,{\sigma }_k-⟨c_k,z_k⟩).\end{array}$$

Let us show how to compute $\xi (H,s,a,\beta )$. Dualizing the linear constraint, we obtain

$$\begin{array}{c}\hfill \xi (H,s,a,\beta )=\underset{\tau \ge 0}{min}[{\Vert s-\tau a\Vert }_{H^{-1}}^{\ast }+\tau \beta ],\end{array}$$ 60

provided that there exists some $x\in \mathbb{E}$ such that ${\Vert x\Vert }_{H^{-1}}<1$, $⟨a,x⟩\le \beta$ (Slater condition). One can show that (60) has the following solution (see Lemma 10):

$$\begin{array}{c}\hfill \tau (H,s,a,\beta ):=\left\{\begin{array}{cc}0,\hfill & \text{if}⟨a,Hs⟩\le \beta {\Vert s\Vert }_{H^{-1}}^{\ast },\hfill \\ u(H,s,a,\beta ),\hfill & \text{otherwise},\hfill \end{array}\right)\end{array}$$ 61

where $u(H,s,a,\beta )$ is the unconstrained minimizer of the objective function in (60).

Let us present an explicit formula for $u(H,s,a,\beta )$. For future use, it will be convenient to write down this formula in a slightly more general form for the following multidimensional7 variant of problem (60):

$$\begin{array}{c}\hfill \underset{u\in {\mathbb{R}}^m}{min}[{\Vert s-Au\Vert }_{H^{-1}}^{\ast }+⟨u,b⟩],\end{array}$$ 62

where $s\in {\mathbb{E}}^{\ast }$, $H:{\mathbb{E}}^{\ast }\to \mathbb{E}$ is a self-adjoint positive definite linear operator, $A:{\mathbb{R}}^m\to {\mathbb{E}}^{\ast }$ is a linear operator with trivial kernel and $b\in {\mathbb{R}}^m$, $⟨b,{(A^{\ast }HA)}^{-1}b⟩<1$. It is not difficult to show that problem (62) has the following unique solution (see Lemma 9):

$$\begin{array}{cc}& u(H,s,A,b):={(A^{\ast }HA)}^{-1}(A^{\ast }s-rb),\hfill \\ \hfill & r:=\sqrt{\frac{⟨s,Hs⟩-⟨s,A{(A^{\ast }HA)}^{-1}{A}^{\ast }s⟩}{1-⟨b,{(A^{\ast }HA)}^{-1}b⟩}}.\hfill \end{array}$$ 63

Note that, in order for the above approach to work, we need to guarantee that the sets ${\Omega }_k$ and $L_k^{-}$ satisfy a certain regularity condition, namely, $\mathrm{int}{\Omega }_k\cap L_k^{-}\ne \mathrm{\varnothing }$. This condition can be easily fulfilled by adding into Algorithm 1 the termination criterion (53).

Lemma 8

Consider Algorithm 1 with termination criterion (53). Then, at each iteration $k\ge 0$, at the beginning of Step , we have $\mathrm{int}{\Omega }_k\cap L_k^{-}\ne \mathrm{\varnothing }$. Moreover, if k is a nonterminal iteration, we also have $⟨g_k,x-x_k⟩\le 0$ for some $x\in \mathrm{int}{\Omega }_k\cap L_k^{-}$.

Proof

Note that $\mathrm{int}{\Omega }_0\cap L_0^{-}=\mathrm{int}B\left(x_0,R\right)\ne \mathrm{\varnothing }$. Now suppose $\mathrm{int}{\Omega }_k\cap L_k^{-}\ne \mathrm{\varnothing }$ for some nonterminal iteration $k\ge 0$. Denote $P_k^{-}:=\{x\in \mathbb{E}:⟨g_k,x-x_k⟩\le 0\}$. Since iteration k is nonterminal, $U_k>0$ and hence ${\Omega }_k\cap L_k^{-}\cap \mathrm{int}{P}_k^{-}\ne \mathrm{\varnothing }$. Combining it with the fact that $\mathrm{int}{\Omega }_k\cap L_k^{-}\ne \mathrm{\varnothing }$, we obtain $\mathrm{int}{\Omega }_k\cap L_k^{-}\cap \mathrm{int}{P}_k^{-}\ne \mathrm{\varnothing }$ and, in particular, $\mathrm{int}{\Omega }_k\cap L_k^{-}\cap P_k^{-}\ne \mathrm{\varnothing }$. At the same time, slightly modifying the proof of Lemma 2 (using that $\mathrm{int}{\Omega }_i=\{x\in \mathbb{E}:{\omega }_i(x)<\frac{1}{2}{R}^2\}$ for any $i\ge 0$ since ${\omega }_i$ is a strictly convex quadratic function), it is not difficult to show that $\mathrm{int}{\Omega }_k\cap L_k^{-}\cap P_k^{-}\subseteq \mathrm{int}{\Omega }_{k+1}\cap L_{k+1}^{-}$. Thus, $\mathrm{int}{\Omega }_{k+1}\cap L_{k+1}^{-}\ne \mathrm{\varnothing }$, and we can continue by induction. $\square$

Computing dual multipliers

Recall from Sect. 5 that the procedure for generating an accuracy semicertificate for Algorithm 1 requires one to repeatedly carry out the following operation: given $s\in {\mathbb{E}}^{\ast }$ and some iteration number $i\ge 0$, compute a dual multiplier $\mu \ge 0$ such that

$$\begin{array}{c}\hfill \underset{x\in {\Omega }_i\cap L_i^{-}}{max}\{⟨s,x⟩:⟨g_i,x-x_i⟩\le 0\}=\underset{x\in {\Omega }_i\cap L_i^{-}}{max}[⟨s,x⟩+\mu ⟨g_i,x_i-x⟩].\end{array}$$

This can be done as follows.

First, using (57), let us rewrite the above primal problem more explicitly:

$$\begin{array}{c}\hfill \underset{x}{max}\{⟨s,x⟩:{\Vert x-z_i\Vert }_{H_i^{-1}}^2\le D_i,⟨c_i,x⟩\le {\sigma }_i,⟨g_i,x-x_i⟩\le 0\}.\end{array}$$

Our goal is to dualize the second linear constraint and find the corresponding multiplier. However, for the sake of symmetry, it is better to dualize both linear constraints, find the corresponding multipliers and then keep only the second one.

Let us simplify our notation by introducing the following problem:

$$\begin{array}{c}\hfill \underset{x}{max}\{⟨s,x⟩:{\Vert x\Vert }_{H^{-1}}\le 1,⟨a_1,x⟩\le b_1,⟨a_2,x⟩\le b_2\},\end{array}$$ 64

where $H:{\mathbb{E}}^{\ast }\to \mathbb{E}$ is a self-adjoint positive definite linear operator, $s,a_1,a_2\in {\mathbb{E}}^{\ast }$ and $b_1,b_2\in \mathbb{R}$. Clearly, our original problem can be transformed into this form by setting $H:=D_i{H}_i$, $a_1:=c_i$, $a_2:=g_i$, $b_1:={\sigma }_i-⟨c_i,z_i⟩$, $b_2:=⟨g_i,x_i-z_i⟩$. Note that this transformation does not change the dual multipliers.

Dualizing the linear constraints in (64), we obtain the following dual problem:

$$\begin{array}{c}\hfill \underset{\mu \in {\mathbb{R}}_{+}^2}{min}[{\Vert s-{\mu }_1{a}_1-{\mu }_2{a}_2\Vert }_{H^{-1}}^{\ast }+{\mu }_1{b}_1+{\mu }_2{b}_2],\end{array}$$ 65

which is solvable provided the following Slater condition holds:

$$\begin{array}{c}\hfill \exists x\in \mathbb{E}:{\Vert x\Vert }_{H^{-1}}<1,⟨a_1,x⟩\le b_1,⟨a_2,x⟩\le b_2.\end{array}$$ 66

Note that (66) can be ensured by adding termination criterion (53) into Algorithm 1 (see Lemma 8).

A solution of (65) can be found using Algorithm 3. In this routine, $\tau (·)$, $\xi (·)$ and $u(·)$ are the auxiliary operations, defined in Sect. 6.2, and $A:=(a_1,a_2)$ is the linear operator $Au:=u_1{a}_1+u_2{a}_2$ acting from ${\mathbb{R}}^2$ to ${\mathbb{E}}^{\ast }$. The correctness of Algorithm 3 is proved in Theorem 3.

Time and memory requirements

Let us discuss the time and memory requirements of Algorithm 1, taking into account the previously mentioned implementation details.

The main objects in Algorithm 1, which need to be stored and updated between iterations, are the test points $x_k$, matrices $H_k$, scalars $R_k$, vectors $c_k$ and scalars ${\sigma }_k$, see (19), (55), 56 and (58) for the corresponding updating formulas. To store all these objects, we need $O(n^2)$ memory.

Consider now what happens at each iteration k. First, we compute $U_k$. For this, we calculate $z_k$ and $D_k$ according to (58) and then perform the calculations described in Sect. 6.2. The most difficult operation there is computing the matrix-vector product, which takes $O(n^2)$ time. After that, we calculate the coefficients $a_k$ and $b_k$ according to (26), where ${\alpha }_k$, $\theta$ and $\gamma$ are certain scalars, easily computable for all main instances of Algorithm 1 (see Sects. 4.1–4.4). The most expensive step there is computing the norm ${\Vert g_k\Vert }_{G_k}^{\ast }$, which can be done in $O(n^2)$ operations by evaluating the product $H_k{g}_k$. Finally, we update our main objects, which takes $O(n^2)$ time.

Thus, each iteration of Algorithm 1 has $O(n^2)$ time and memory complexities, exactly as in the standard Ellipsoid Method.

Now let us analyze the complexity of the auxiliary procedure from Sect. 5 for converting a preliminary semicertificate into a semicertificate. The main operation in this procedure is running Algorithm 2, which iterates “backwards”, computing some dual multiplier ${\mu }_i$ at each iteration $i=k-1,\cdots ,0$. Using the approach from Sect. 6.3, we can compute ${\mu }_i$ in $O(n^2)$ time, provided that the objects $x_i$, $g_i$, $H_i$, $z_i$, $D_i$, $c_i$, ${\sigma }_i$ are stored in memory. Note, however, that, in contrast to the “forward” pass, when iterating “backwards”, there is no way to efficiently recompute all these objects without storing in memory a certain “history” of the main process from iteration 0 up to k. The simplest choice is to keep in this “history” all the objects mentioned above, which requires $O(k{n}^2)$ memory. A slightly more efficient idea is to keep the matrix-vector products $H_i{g}_i$ instead of $H_i$ and then use (55) to recompute $H_i$ from $H_{i+1}$ in $O(n^2)$ operations. This allows us to reduce the size of the “history” down to $O(kn)$ while still keeping the $O(k{n}^2)$ total time complexity of the auxiliary procedure. Note that these estimates are exactly the same as those for the best currently known technique for generating accuracy certificates in the standard Ellipsoid Method [16]. In particular, if we generate a semicertificate only once at the very end, then the time complexity of our procedure is comparable to that of running the standard Ellipsoid Method without computing any certificates. Alternatively, as suggested in [16], one can generate semicertificates, say, every $2,4,8,16,\cdots$ iterations. Then, the total “overhead” of the auxiliary procedure for generating semicertificates will be comparable to the time complexity of the method itself.

Conclusion

In this paper, we have addressed one of the issues of the standard Ellipsoid Method, namely, its poor convergence for problems of large dimension n. For this, we have proposed a new algorithm which can be seen as the combination of the Subgradient and Ellipsoid methods.

Our developments can be considered as a first step towards constructing universal methods for nonsmooth problems with convex structure. Such methods could significantly improve the practical efficiency of solving various applied problems.

Note that there are still some open questions. First, the convergence estimate of our method with time-varying coefficients contains an extra factor proportional to the logarithm of the iteration counter. We have seen that this logarithmic factor has its roots yet in the Subgradient Method. However, as discussed in Remark 4, for the Subgradient Method, this issue can be easily resolved by allowing projections onto the feasible set and working with “truncated” gaps. An even better alternative, which does not require any of this machinery, is to use Dual Averaging [18] instead of the Subgradient Method. It is an interesting question whether one can combine the Dual Averaging with the Ellipsoid Method similarly to how we have combined the Subgradient and Ellipsoid methods.

Second, the convergence rate estimate, which we have obtained for our method, is not continuous in the dimension n. Indeed, for small values of the iteration counter k, this estimate behaves as that of the Subgradient Method and then, at some moment (around $n^2$), it switches to the estimate of the Ellipsoid Method. As discussed at the end of Sect. 4.4, there exists some “small” gap between these two estimates around the switching moment. Nevertheless, the method itself is continuous in n and does not contain any explicit switching rules. Therefore, there should be some continuous convergence rate estimate for our method, and it is an open question to find it.

Another interesting question is to understand what happens with the proposed method on other (less general) classes of convex problems than those, considered in this paper. For example, it is well-known that, on smooth and/or strongly convex problems, (sub)gradient methods have much better convergence rates than on the general nonsmooth problems. We expect that similar conclusions should also be valid for the proposed Subgradient Ellipsoid Method. However, to achieve the acceleration, it may be necessary to introduce some modifications in the algorithm such as using different step sizes. We leave this direction for future research.

Finally, apart from the Ellipsoid Method, there exist other “dimension-dependent” methods (e.g., the Center-of-Gravity Method8 [13, 20], the Inscribed Ellipsoid Method [22], the Circumscribed Simplex Method [6], etc.). Similarly, the Subgradient Method is not the only “dimension-independent” method and there exist numerous alternatives which are better suited for certain problem classes (e.g., the Fast Gradient Method [17] for Smooth Convex Optimization or methods for Stochastic Programming [7, 8, 11, 15]). Of course, it is interesting to consider different combinations of the aforementioned “dimension-dependent” and “dimension-independent” methods. In this regard, it is also worth mentioning the works [4, 5], where the authors propose new variants of gradient-type methods for smooth strongly convex minimization problems inspired by the geometric construction of the Ellipsoid Method.

Proof of Lemma 6

Proof

Everywhere in the proof, we assume that the parameter c is fixed and drop all the indices related to it.

Let us show that ${\zeta }_p$ is a convex function. Indeed, the function $\omega :\mathbb{R}\times {\mathbb{R}}_{++}\to \mathbb{R}$, defined by $\omega (x,t):=\frac{x^2}{t}$, is convex. Hence, the function q, defined in (34), is also convex. Further, since $\omega$ is increasing in its first argument on ${\mathbb{R}}_{+}$, the function ${\omega }_p:{\mathbb{R}}_{+}\times {\mathbb{R}}_{++}\to \mathbb{R}$, defined by ${\omega }_p(x,t):=\frac{x^p}{t}$, is also convex as the composition of $\omega$ with the mapping $(x,t)\mapsto (x^{p/2},t)$, whose first component is convex (since $p\ge 2$) and the second one is affine. Note that ${\omega }_p$ is increasing in its first argument. Hence, ${\zeta }_p$ is indeed a convex function as the composition of ${\omega }_p$ with the mapping $\gamma \mapsto (q(\gamma ),1+\gamma )$, whose first part is convex and the second one is affine.

Differentiating, for any $\gamma >0$, we obtain

$$\begin{array}{c}\hfill {\zeta }_p^{\prime }(\gamma )=\frac{p{[q(\gamma )]}^{p-1}{q}^{\prime }(\gamma )(1+\gamma )-{[q(\gamma )]}^p}{{(1+\gamma )}^2}=\frac{{[q(\gamma )]}^{p-1}(p{q}^{\prime }(\gamma )(1+\gamma )-q(\gamma ))}{{(1+\gamma )}^2}.\end{array}$$

Therefore, the minimizers of ${\zeta }_p$ are exactly solutions to the following equation:

$$\begin{array}{c}\hfill p{q}^{\prime }(\gamma )(1+\gamma )=q(\gamma ).\end{array}$$ 67

Note that $q^{\prime }(\gamma )=\frac{c[2\gamma (1+\gamma )-{\gamma }^2]}{2{(1+\gamma )}^2}=\frac{c\gamma (2+\gamma )}{2{(1+\gamma )}^2}$ (see (34)). Hence, (67) can be written as $cp\gamma (2+\gamma )=2(1+\gamma )+c{\gamma }^2$ or, equivalently, $c(p-1){\gamma }^2+2(cp-1)\gamma =2$. Clearly, $\gamma =0$ is not a solution of this equation. Making the change of variables $\gamma =\frac{2}{u}$, $u\ne 0$, we come the quadratic equation $u^2-2(cp-1)u=2c(p-1)$ or, equivalently, to ${[u-(cp-1)]}^2=2c(p-1)+{(cp-1)}^2=c^2{p}^2-(2c-1)$. This equation has two solutions: $u_1:=cp-1+\sqrt{c^2{p}^2-(2c-1)}$ and $u_2:=cp-1-\sqrt{c^2{p}^2-(2c-1)}$. Note that $u_2\ge cp-1-\sqrt{c^2{p}^2+1}\ge cp-1-(cp+1)=-2$. Hence, ${\gamma }_2:=\frac{2}{u_2}\le -1$ cannot be a minimizer of ${\zeta }_p$. Consequently, only $u_1$ is an acceptable solution (note that $u_1>0$ in view of our assumptions on c and p). Thus, (37) is proved.

Let us show that $\gamma (p)$ belongs to the interval specified in (37). For this, we need to prove that $1\le cp\gamma (p)\le 2$. Note that the function $h_a(t):=\frac{t}{\sqrt{t^2-a}+t-1}$, where $a\ge 0$, is decreasing in t. Indeed, $\frac{1}{h_a(t)}=\sqrt{1-\frac{a}{t^2}}-\frac{1}{t}+1$ is an increasing function in t. Hence, $cp\gamma (p)=2h_{2c-1}(cp)\ge 2{lim}_{t\to \infty }{h}_{2c-1}(t)=1$. On the other hand, using that $p\ge 2$ and denoting $\alpha :=2c\ge 1$, we get $cp\gamma (p)=2h_{\alpha -1}(cp)\le 2g(\alpha )$, where $g(\alpha ):=h_{\alpha -1}(\alpha )=\frac{\alpha }{\sqrt{{\alpha }^2-\alpha +1}+\alpha -1}$. Note that g is decreasing in $\alpha$. Indeed, denoting $\tau :=\frac{1}{\alpha }\in \left(0,1\right]$, we obtain $\frac{1}{g(\alpha )}=\sqrt{1-\tau +{\tau }^2}-\tau +1$, which is a decreasing function in $\tau$. Thus, $cp\gamma (p)\le 2g(1)=2$.

It remains to prove that ${\zeta }_p(\gamma (p))\le e^{-1/(2cp)}$. Let $\varphi :\left[2,+\infty \right)\to \mathbb{R}$ be the function

$$\begin{array}{c}\hfill \varphi (p):=-ln{\zeta }_p(\gamma (p))=ln(1+\gamma (p))-plnq(\gamma (p)).\end{array}$$ 68

We need to show that $\varphi (p)\ge 1/(2cp)$ for all $p\ge 2$ or, equivalently, that the function $\chi :\left(0,\frac{1}{2}\right]\to \mathbb{R}$, defined by $\chi (\tau ):=\varphi (\frac{1}{\tau })$, satisfies $\chi (\tau )\ge \frac{\tau }{2c}$ for all $\tau \in \left(0,\frac{1}{2}\right]$. For this, it suffices to show that $\chi$ is convex, ${lim}_{\tau \to 0}\chi (\tau )=0$ and ${lim}_{\tau \to 0}{\chi }^{\prime }(\tau )=\frac{1}{2c}$. Differentiating, we see that ${\chi }^{\prime }(\tau )=-\frac{1}{{\tau }^2}{\varphi }^{\prime }(\frac{1}{\tau })$ and ${\chi }^{\prime \prime }(\tau )=\frac{2}{{\tau }^3}{\varphi }^{\prime }(\frac{1}{\tau })+\frac{1}{{\tau }^4}{\varphi }^{\prime \prime }(\frac{1}{\tau })$ for all $\tau \in \left(0,\frac{1}{2}\right]$. Thus, we need to justify that

$$\begin{array}{c}\hfill 2{\varphi }^{\prime }(p)+p{\varphi }^{\prime \prime }(p)\ge 0\end{array}$$ 69

for all $p\ge 2$ and that

$$\begin{array}{c}\hfill \underset{p\to \infty }{lim}\varphi (p)=0,\underset{p\to \infty }{lim}[-p^2{\varphi }^{\prime }(p)]=\frac{1}{2c}.\end{array}$$ 70

Let $p\ge 2$ be arbitrary. Differentiating and using (67), we obtain

$$\begin{array}{cc}\hfill {\varphi }^{\prime }(p)& =\frac{{\gamma }^{\prime }(p)}{1+\gamma (p)}-lnq(\gamma (p))-\frac{p{q}^{\prime }(\gamma (p)){\gamma }^{\prime }(p)}{q(\gamma (p))}=-lnq(\gamma (p)),\hfill \\ \hfill {\varphi }^{\prime \prime }(p)& =-\frac{q^{\prime }(\gamma (p)){\gamma }^{\prime }(p)}{q(\gamma (p))}=-\frac{{\gamma }^{\prime }(p)}{p(1+\gamma (p))}.\hfill \end{array}$$ 71

Therefore,

$$\begin{array}{c}\hfill 2{\varphi }^{\prime }(p)+p{\varphi }^{\prime \prime }(p)=-2lnq(\gamma (p))-\frac{{\gamma }^{\prime }(p)}{1+\gamma (p)}\ge -\frac{c{\gamma }^2(p)+{\gamma }^{\prime }(p)}{1+\gamma (p)},\end{array}$$

where the inequality follows from (34) and the fact that $ln(1+\tau )\le \tau$ for any $\tau >-1$. Thus, to show (69), we need to prove that $-{\gamma }^{\prime }(p)\ge c{\gamma }^2(p)$ or, equivalently, $\frac{d}{\mathit{dp}}\frac{1}{\gamma (p)}\ge c$. But this is immediate. Indeed, using (37), we obtain $\frac{d}{\mathit{dp}}\frac{1}{\gamma (p)}=\frac{c}{2}\left(\frac{\mathit{cp}}{\sqrt{c^2{p}^2-(2c-1)}},+,1\right)\ge c$ since the function $\tau \mapsto \frac{\tau }{\sqrt{{\tau }^2-1}}$ is decreasing. Thus, (69) is proved.

It remains to show (70). From (37), we see that $\gamma (p)\to 0$ and $p\gamma (p)\to \frac{1}{c}$ as $p\to \infty$. Hence, using (34), we obtain

$$\begin{array}{c}\hfill \underset{p\to \infty }{lim}{p}^{2}lnq(\gamma (p))=\underset{p\to \infty }{lim}\frac{c{p}^2{\gamma }^2(p)}{2(1+\gamma (p))}=\frac{c}{2}\underset{p\to \infty }{lim}{p}^2{\gamma }^2(p)=\frac{1}{2c}.\end{array}$$

Consequently, in view of (68) and (71), we have

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill \underset{p\to \infty }{lim}\varphi (p)& =\underset{p\to \infty }{lim}[ln(1+\gamma (p))-plnq(\gamma (p))]=0,\hfill \\ \hfill \underset{p\to \infty }{lim}[-p^2{\varphi }^{\prime }(p)]& =\underset{p\to \infty }{lim}{p}^{2}lnq(\gamma (p))=\frac{1}{2c},\hfill \end{array}\end{array}$$

which is exactly (70). $\square$

Support function and dual multipliers: proofs

For brevity, everywhere in this section, we write $\Vert x\Vert$ and ${\Vert ·\Vert }_{\ast }$ instead of ${\Vert ·\Vert }_{H^{-1}}$ and ${\Vert ·\Vert }_{H^{-1}}^{\ast }$, respectively. We also denote $B_0:=\{x\in \mathbb{E}:\Vert x\Vert \le 1\}$.

Auxiliary operations

Lemma 9

Let $s\in {\mathbb{E}}^{\ast }$, let $A:{\mathbb{R}}^m\to {\mathbb{E}}^{\ast }$ be a linear operator with trivial kernel and let $b\in {\mathbb{R}}^m$, $⟨b,{(A^{\ast }HA)}^{-1}b⟩<1$. Then, problem (62) has a unique solution given by (63).

Proof

Note that the sublevel sets of the objective function in (62) are bounded:

$$\begin{array}{c}\hfill {\Vert s-Au\Vert }_{\ast }+⟨u,b⟩\ge {\Vert \mathit{Au}\Vert }_{\ast }-{\Vert s\Vert }_{\ast }+⟨u,b⟩\ge (1-{⟨b,{(A^{\ast }HA)}^{-1}b⟩}^{1/2}){\Vert \mathit{Au}\Vert }_{\ast }-{\Vert s\Vert }_{\ast }\end{array}$$

for all $u\in {\mathbb{R}}^m$. Hence, problem (62) has a solution.

Let $u\in {\mathbb{R}}^m$ be a solution of problem (62). If $s=Au$, then $u={(A^{\ast }HA)}^{-1}{A}^{\ast }s$, which coincides with the solution given by (63) (note that, in this case, $r=0$).

Now suppose $s\ne Au$. Then, from the first-order optimality condition, we obtain that $b=A^{\ast }(s-Au)/\rho$, where $\rho :={\Vert s-Au\Vert }_{\ast }>0$. Hence, $u={(A^{\ast }HA)}^{-1}(A^{\ast }s-\rho b)$ and

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill {\rho }^2& ={\Vert s-Au\Vert }_{\ast }^2={\Vert s\Vert }_{\ast }^2-2⟨A^{\ast }s,u⟩+⟨A^{\ast }HAu,u⟩\hfill \\ \hfill & ={\Vert s\Vert }_{\ast }^2-2⟨A^{\ast }s,{(A^{\ast }HA)}^{-1}(A^{\ast }s-\rho b)⟩+⟨A^{\ast }s-\rho b,{(A^{\ast }HA)}^{-1}(A^{\ast }s-\rho b)⟩\hfill \\ \hfill & ={\Vert s\Vert }_{\ast }^2-⟨s,A{(A^{\ast }HA)}^{-1}{A}^{\ast }s⟩+{\rho }^2⟨b,{(A^{\ast }HA)}^{-1}b⟩.\hfill \end{array}\end{array}$$

Thus, $\rho =r$ and $u=u(H,s,A,b)$ given by (63). $\square$

Lemma 10

Let $s,a\in {\mathbb{E}}^{\ast }$, $\beta \in \mathbb{R}$ be such that $⟨a,x⟩\le \beta$ for some $x\in \mathrm{int}{B}_0$. Then, problem (60) has a solution given by (61). Moreover, this solution is unique if $\beta <{\Vert a\Vert }_{\ast }$.

Proof

Let $\varphi :\mathbb{R}\to \mathbb{R}$ be the function $\varphi (\tau ):={\Vert s-\tau a\Vert }_{\ast }+\tau \beta$. By our assumptions, $\beta >-{\Vert a\Vert }_{\ast }$ if $a\ne 0$ and $\beta \ge 0$ if $a=0$. If additionally $\beta <{\Vert a\Vert }_{\ast }$, then $|\beta |<{\Vert a\Vert }_{\ast }$.

If $s=0$, then $\varphi (\tau )=\tau ({\Vert a\Vert }_{\ast }+\beta )\ge \varphi (0)$ for all $\tau \ge 0$, so 0 is a solution of (60). Clearly, this solution is unique when $\beta <{\Vert a\Vert }_{\ast }$ because then $|\beta |<{\Vert a\Vert }_{\ast }$.

From now on, suppose $s\ne 0$. Then, $\varphi$ is differentiable at 0 with ${\varphi }^{\prime }(0)=\beta -⟨a,s⟩/{\Vert s\Vert }_{\ast }$. If $⟨a,s⟩\le \beta {\Vert s\Vert }_{\ast }$, then ${\varphi }^{\prime }(0)\ge 0$, so 0 is a solution of (60). Note that this solution is unique if $⟨a,s⟩<\beta {\Vert s\Vert }_{\ast }$ because then ${\varphi }^{\prime }(0)>0$, i.e., $\varphi$ is strictly increasing on ${\mathbb{R}}_{+}$.

Suppose $⟨a,s⟩>\beta {\Vert s\Vert }_{\ast }$. Then, $\beta <{\Vert a\Vert }_{\ast }$ and thus $|\beta |<{\Vert a\Vert }_{\ast }$. Note that, for any $\tau \ge 0$, we have $\varphi (\tau )\ge \tau ({\Vert a\Vert }_{\ast }+\beta )-{\Vert s\Vert }_{\ast }$. Hence, the sublevel sets of $\varphi$, intersected with ${\mathbb{R}}_{+}$, are bounded, so problem (60) has a solution. Since ${\varphi }^{\prime }(0)<0$, any solution of (60) is strictly positive and so must be a solution of problem (62) for $A:=a$ and $b:=\beta$. But, by Lemma 9, the latter solution is unique and equals $u(H,s,a,\beta )$.

We have proved that (61) is indeed a solution of (60). Moreover, when $⟨a,s⟩\ne \beta {\Vert s\Vert }_{\ast }$, we have shown that this solution is unique. It remains to prove the uniqueness of solution when $⟨a,s⟩=\beta {\Vert s\Vert }_{\ast }$, assuming additionally that $\beta <{\Vert a\Vert }_{\ast }$. But this is simple. Indeed, by our assumptions, $|\beta |<{\Vert a\Vert }_{\ast }$, so $|⟨a,s⟩|=|\beta |{\Vert s\Vert }_{\ast }<{\Vert a\Vert }_{\ast }{\Vert s\Vert }_{\ast }$. Hence, a and s are linearly independent. But then $\varphi$ is strictly convex, and thus its minimizer is unique. $\square$

Computation of dual multipliers

In this section, we prove the correctness of Algorithm 3.

For $s\in {\mathbb{E}}^{\ast }$, let X(s) be the subdifferential of ${\Vert ·\Vert }_{\ast }$ at the point s:

$$\begin{array}{c}\hfill X(s):=\left\{\begin{array}{cc}\{Hs/{\Vert s\Vert }_{\ast }\},\hfill & \text{if}s\ne 0,\hfill \\ B_0,\hfill & \text{if}s=0.\hfill \end{array}\right)\end{array}$$ 72

Clearly, $X(s)\subseteq B_0$ for any $s\in {\mathbb{E}}^{\ast }$. When $s\ne 0$, we denote the unique element of X(s) by x(s).

Let us formulate a convenient optimality condition.

Lemma 11

Let A be the linear operator from ${\mathbb{R}}^m$ to ${\mathbb{E}}^{\ast }$, defined by $Au:={\sum }_{i=1}^m{u}_i{a}_i$, where $a_1,\cdots ,a_m\in {\mathbb{E}}^{\ast }$, and let $b\in {\mathbb{R}}^m$, $s\in {\mathbb{E}}^{\ast }$. Then, ${\mu }^{\ast }\in {\mathbb{R}}_{+}^m$ is a minimizer of $\psi (\mu ):={\Vert s-A\mu \Vert }_{\ast }+⟨\mu ,b⟩$ over ${\mathbb{R}}_{+}^m$ if and only if $X(s-A{\mu }^{\ast })\cap L_1({\mu }_1^{\ast })\cdots L_m({\mu }_m^{\ast })\ne \mathrm{\varnothing }$, where, for each $1\le i\le m$ and $\tau >0$, we denote $L_i(\tau ):=\{x\in \mathbb{E}:⟨a_i,x⟩\le b_i\}$, if $\tau =0$, and $L_i(\tau ):=\{x\in \mathbb{E}:⟨a_i,x⟩=b_i\}$, if $\tau >0$.

Proof

Indeed, the standard optimality condition for a convex function over the nonnegative orthant is as follows: ${\mu }^{\ast }\in {\mathbb{R}}_{+}^m$ is a minimizer of $\psi$ on ${\mathbb{R}}_{+}^m$ if and only if there exists $g^{\ast }\in \partial \psi ({\mu }^{\ast })$ such that $g_i^{\ast }\ge 0$ and $g_i^{\ast }{\mu }_i^{\ast }=0$ for all $1\le i\le m$. It remains to note that $\partial \psi ({\mu }^{\ast })=b-A^{\ast }X(s-A{\mu }^{\ast })$. $\square$

Theorem 3

Algorithm 3 is well-defined and returns a solution of (65).

Proof

i. For each $i=1,2$ and $\tau \ge 0$, denote $L_i^{-}:=\{x\in \mathbb{E}:⟨a_i,x⟩\le b_i\}$, $L_i:=\{x\in \mathbb{E}:⟨a_i,x⟩=b_i\}$, $L_i(\tau ):=L_i^{-}$, if $\tau =0$, and $L_i(\tau ):=L_i$, if $\tau >0$.

ii. From (66) and Lemma 10, it follows that Step  is well-defined and, for each $i=1,2$, ${\tau }_i$ is a solution of (60) with parameters $(s,a_i,b_i)$. Hence, by Lemma 11,

$$\begin{array}{c}\hfill X(s-{\tau }_i{a}_i)\cap L_i({\tau }_i)\ne \mathrm{\varnothing },i=1,2.\end{array}$$ 73

iii. Consider Step . Note that the condition ${\xi }_1\le b_2$ is equivalent to $B_0\cap L_1^{-}\subseteq L_2^{-}$ since ${\xi }_1={max}_{x\in B_0\cap L_1^{-}}⟨a_2,x⟩$. If $B_0\cap L_1^{-}\subseteq L_2^{-}$, then, by (73), $X(s-{\tau }_1{a}_1)\cap L_1({\tau }_1)\cap L_2^{-}=X(s-{\tau }_1{a}_1)\cap L_1({\tau }_1)\ne \mathrm{\varnothing }$, so, by Lemma 11, $({\tau }_1,0)$ is indeed a solution of (65).

Similarly, if ${\xi }_2\le b_1$, then $B_0\cap L_2^{-}\subseteq L_1^{-}$ and $(0,{\tau }_2)$ is a solution of (65).

iv. From now on, we can assume that $B_0\cap L_1^{-}\cap \mathrm{int}{L}_2^{+}\ne \mathrm{\varnothing }$, $B_0\cap L_2^{-}\cap \mathrm{int}{L}_1^{+}\ne \mathrm{\varnothing }$, where $\mathrm{int}{L}_i^{+}:=\{x\in \mathbb{E}:⟨a_i,x⟩>b_i\}$, $i=1,2$. Combining this with (66), we obtain9

$$\begin{array}{c}\hfill \mathrm{int}{B}_0\cap L_1\cap L_2^{-}\ne \mathrm{\varnothing },\mathrm{int}{B}_0\cap L_2\cap L_1^{-}\ne \mathrm{\varnothing }.\end{array}$$ 74

Suppose $⟨a_2,H(s-{\tau }_1{a}_1)⟩\le b_2{\Vert s-{\tau }_1{a}_1\Vert }_{\ast }$ at Step . 1) If $s\ne {\tau }_1{a}_1$, then $X(s-{\tau }_1{a}_1)$ is a singleton, $x(s-{\tau }_1{a}_1)=H(s-{\tau }_1{a}_1)/{\Vert s-{\tau }_1{a}_1\Vert }_{\ast }$, so we obtain $x(s-{\tau }_1{a}_1)\in L_2^{-}$. Combining this with (73), we get $x(s-{\tau }_1{a}_1)\in L_1({\tau }_1)\cap L_2^{-}$. 2) If $s={\tau }_1{a}_1$, then $X(s-{\tau }_1{a}_1)\cap L_1({\tau }_1)\cap L_2^{-}=B_0\cap L_1({\tau }_1)\cap L_2^{-}\ne \mathrm{\varnothing }$ in view of the first claim in (74) (recall that $L_1\subseteq L_1({\tau }_1)$). Thus, in any case, $X(s-\tau a_1)\cap L_1({\tau }_1)\cap L_2^{-}\ne \mathrm{\varnothing }$, and so, by Lemma 11, $({\tau }_1,0)$ is a solution of (65).

Similarly, one can consider the case when $⟨a_1,H(s-{\tau }_2{a}_2)⟩\le b_1{\Vert s-{\tau }_2{a}_2\Vert }_{\ast }$ at Step .

Suppose we have reached Step . From now on, we can assume that

$$\begin{array}{c}\hfill X(s-{\tau }_1{a}_1)\cap L_1({\tau }_1)\cap \mathrm{int}{L}_2^{+}\ne \mathrm{\varnothing },X(s-{\tau }_2{a}_2)\cap L_2({\tau }_2)\cap \mathrm{int}{L}_1^{+}\ne \mathrm{\varnothing }.\\ \hfill \end{array}$$ 75

Indeed, since both conditions at Step  have not been satisfied, $s\ne {\tau }_i{a}_i$, $i=1,2$, and $x(s-{\tau }_1{a}_1)\notin L_2^{-}$, $x(s-{\tau }_2{a}_2)\notin L_1^{-}$. Also, by (73), $x(s-{\tau }_i{a}_i)\in L_i({\tau }_i)$, $i=1,2$.

Let $\mu \in {\mathbb{R}}_{+}^2$ be any solution of (65). By Lemma 11, $X(s-A\mu )\cap L_1({\mu }_1)\cap L_2({\mu }_2)\ne \mathrm{\varnothing }$. Note that we cannot have ${\mu }_2=0$. Indeed, otherwise, we get $X(s-{\mu }_1{a}_1)\cap L_1({\mu }_1)\cap L_2^{-}\ne \mathrm{\varnothing }$, so ${\mu }_1$ must be a solution of (60) with parameters $(s,a_1,b_1)$. But, by Lemma 10, such a solution is unique (in view of the second claim in (75), $⟨a_1,x⟩>b_1$ for some $x\in B_0$, so $b_1<{\Vert a_1\Vert }_{\ast }$). Hence, ${\mu }_1={\tau }_1$, and we obtain a contradiction with (75). Similarly, we can show that ${\mu }_1\ne 0$. Consequently, ${\mu }_1,{\mu }_2>0$, which means that $\mu$ is a solution of (62).

Thus, at this point, any solution of (65) must be a solution of (62). In view of Lemma 9, to finish the proof, it remains to show that the vectors $a_1$, $a_2$ are linearly independent and $⟨b,{(A^{\ast }HA)}^{-1}b⟩<1$. But this is simple. Indeed, from (75), it follows that

$$\begin{array}{c}\hfill \text{either}{B}_0\cap L_1\cap \mathrm{int}{L}_2^{+}\ne \mathrm{\varnothing }\text{or}{B}_0\cap L_2\cap \mathrm{int}{L}_1^{+}\ne \mathrm{\varnothing }\end{array}$$ 76

since ${\tau }_1$ and ${\tau }_2$ cannot both be equal to 0. Combining (76) and (74), we see that $\mathrm{int}{B}_0\cap L_1\cap L_2\ne \mathrm{\varnothing }$ and, in particular, $L_1\cap L_2\ne \mathrm{\varnothing }$. Hence, $a_1$, $a_2$ are linearly independent (otherwise, $L_1=L_2$, which contradicts (76)). Taking any $x\in \mathrm{int}{B}_0\cap L_1\cap L_2$, we obtain $\Vert x\Vert <1$ and $A^{\ast }x=b$, hence $⟨b,{(A^{\ast }HA)}^{-1}b⟩=⟨A^{\ast }x,{(A^{\ast }HA)}^{-1}{A}^{\ast }x⟩\le {\Vert x\Vert }^2<1$, where we have used $A{(A^{\ast }HA)}^{-1}{A}^{\ast }⪯H^{-1}$. $\square$