On the Convergence Analysis of the Optimized Gradient Method

Abstract

This paper considers the problem of unconstrained minimization of smooth convex functions having Lipschitz continuous gradients with known Lipschitz constant. We recently proposed the optimized gradient method for this problem and showed that it has a worst-case convergence bound for the cost function decrease that is twice as small as that of Nesterov’s fast gradient method, yet has a similarly efficient practical implementation. Drori showed recently that the optimized gradient method has optimal complexity for the cost function decrease over the general class of first-order methods. This optimality makes it important to study fully the convergence properties of the optimized gradient method. The previous worst-case convergence bound for the optimized gradient method was derived for only the last iterate of a secondary sequence. This paper provides an analytic convergence bound for the primary sequence generated by the optimized gradient method. We then discuss additional convergence properties of the optimized gradient method, including the interesting fact that the optimized gradient method has two types of worstcase functions: a piecewise affine-quadratic function and a quadratic function. These results help complete the theory of an optimal first-order method for smooth convex minimization.

1 Introduction

We recently proposed the optimized gradient method (OGM) [1] for unconstrained smooth convex minimization problems, building upon Drori and Teboulle [2]. We showed in [1] that OGM has a worst-case cost function convergence bound that is twice as small as that of Nesterov’s fast gradient method (FGM) [3], yet has an efficient implementation that is similar to FGM. In addition, Drori [4] showed that OGM achieves the optimal worst-case convergence bound of the cost function decrease over the general class of first-order methods (for largedimensional problems), making it important to further study the convergence properties of OGM.

The worst-case convergence bound for OGM was derived for only the last iterate of a secondary sequence in [1], and this paper additionally provides an analytic convergence bound for the primary sequence generated by OGM by extending the analysis in [1]. We further discuss convergence properties of OGM, including the interesting fact that OGM has two types of worstcase functions: a piecewise affine-quadratic function and a quadratic function. These results complement our understanding of an optimal first-order method for smooth convex minimization.

2 Problem, Algorithms and Contributions

We consider the unconstrained smooth convex minimization problem

$$\underset{\mathit{x}\in {\mathbb{ℝ}}^d}{\text{min}}f\left(\mathit{x}\right)$$ (M)

with the following two conditions:

• – f : ℝ^d → ℝ is a convex function of the type $C_L^{1,1}\left({\mathbb{ℝ}}^d\right)$, i.e., continuously differentiable with Lipschitz continuous gradient:

  $$\parallel \nabla f\left(\mathit{x}\right)-\nabla f\left(\mathit{y}\right)\parallel \le L\parallel \mathit{x}-\mathit{y}\parallel ,\forall \mathit{x},\mathit{y}\in {\mathbb{ℝ}}^d,$$

  where L > 0 is the Lipschitz constant.

• – The optimal set X_∗(f) = arg min_x∈ℝ^d f (x) is nonempty, i.e., problem (M) is solvable.

We use ℱ_L(ℝ^d) to denote the class of functions that satisfy the above conditions hereafter.

For large-scale optimization problems of type (M) that arise in various fields such as communications, machine learning and signal processing, general first-order algorithms that query only the cost function values and gradients are attractive because of their mild dependence on the problem dimension [5]. For simplicity, we initially focus on the class of fixed-step first-order (FO) algorithms having the following form:

FO updates use weighted sums of current and previous gradients ${\{\nabla f\left({\mathit{x}}_k\right)\}}_{k=0}^i$ with (pre-determined) step sizes ${\left\{h_{i+1,k}\right\}}_{k=0}^i$ and the Lipschitz constant L. Class FO includes the (fixed-step) gradient method (GM), the heavy-ball method [6], Nesterov’s fast gradient method (FGM) [3, 7], and the recently introduced optimized gradient method (OGM) [1]. Those four methods have efficient recursive formulations rather than directly using (1) that would require storing all previous gradients and computing weighted summations every iteration. Among class FO, Nesterov’s FGM has been used widely, since it achieves the optimal rate O(1/N²) for decreasing a cost function in N iterations [8], and has two efficient forms as shown below for smooth convex problems.

Both FGM1 and FGM2 produce identical sequences {y_i} and {x_i}, where the primary sequence {y_i} satisfies the following convergence bound [3, 7] for any 1 ≤ i ≤ N :

$$f\left({\mathit{y}}_i\right)-f\left({\mathit{x}}_{\ast }\right)\le \frac{L{R}^2}{2t_{i-1}^2}\le \frac{2L{R}^2}{{(i+1)}^2}.$$ (2)

In [1], we showed that the secondary sequence {x_i} of FGM satisfies the following convergence bound that is similar to (2) for any 1 ≤ i ≤ N :

$$f\left({\mathit{x}}_i\right)-f\left({\mathit{x}}_{\ast }\right)\le \frac{L{R}^2}{2t_i^2}\le \frac{2L{R}^2}{{(i+2)}^2}.$$ (3)

Taylor et al. [9] demonstrated that the upper bounds (2) and (3) are only asymptotically tight.

When the large-scale condition “d ≥ 2N + 1” holds, Nesterov [8] showed that for any first-order method generating x_N after N iterations there exists a function φ in ℱ_L(ℝ^d) that satisfies the following lower bound:

$$\frac{3L{\parallel {\mathit{x}}_0-{\mathit{x}}_{\ast }\parallel }^2}{32{(N+1)}^2}\le \varphi \left({\mathit{x}}_N\right)-\varphi \left({\mathit{x}}_{\ast }\right).$$ (4)

Although FGM achieves the optimal rate O(1/N²), one can still seek algorithms that improve upon the constant factor in (2) and (3), in light of the gap between the bounds (2), (3) of FGM and the lower complexity bound (4). Building upon Drori and Teboulle (hereafter “DT”)’s approach [2] of seeking FO methods that are faster than Nesterov’s FGM (reviewed in Section 3.3), we recently proposed following two efficient formulations of OGM [1].

OGM1 and OGM2 have computational efficiency comparable to FGM1 and FGM2, and produce identical primary sequence {y_i} and secondary sequence {x_i}. The last iterate x_N of OGM satisfies the following analytical worst-case bound [1, Theorem 2]:

$$f\left({\mathit{x}}_N\right)-f\left({\mathit{x}}_{\ast }\right)\le \frac{L{R}^2}{2{\theta }_N^2}\le \frac{L{R}^2}{(N+1)(N+1+\sqrt{2})},$$ (5)

which is twice as small as those for FGM in (2) and (3). Recently for the condition “d ≥ N + 1”, Drori [4] showed that for any first-order method there exists a function ψ in ℱ_L(ℝ^d) that cannot be minimized faster than the following lower bound:

$$\frac{L{\parallel {\mathit{x}}_0-{\mathit{x}}_{\ast }\parallel }^2}{2{\theta }_N^2}\le \psi \left({\mathit{x}}_N\right)-\psi \left({\mathit{x}}_{\ast }\right),$$ (6)

where x_N is the Nth iterate of any first-order method. This lower complexity bound (6) improves on (4), and exactly matches the bound (5) of OGM, showing that OGM achieves the optimal worst-case bound of the cost function for first-order methods when d ≥ N + 1. What is remarkable about Drori’s result is that OGM was derived by optimizing over the class FO having fixed step sizes, leading to (5), whereas Drori’s lower bound in (6) is for the general class of first-order methods where the step sizes are arbitrary. It is interesting that OGM with its fixed step sizes is optimal over the apparently much broader class.

Because OGM has such optimality, it is desirable to understand its properties thoroughly. For example, analytical bounds for the primary sequence {y_i} of OGM have not been studied previously, although numerical bounds were discussed by Taylor et al. [9]. This paper provides analytical bounds for the primary sequence of OGM, augmenting the convergence analysis of x_N of OGM given in [1]. We also relate OGM to another version of Nesterov’s accelerated first-order method in [10] that has a similar formulation as OGM2.

In [1, Theorem 3], we specified a worst-case function for which OGM achieves the first upper bound in (5) exactly. The corresponding worst-case function is the following piecewise affine-quadratic function:

$$f_1,\text{OGM}(\mathit{x};N)=\{\begin{array}{cc}\frac{LR}{{\theta }_N^2}\parallel \mathit{x}\parallel -\frac{L{R}^2}{2{\theta }_N^4,}\hfill & \text{if}\parallel \mathit{x}\parallel \ge \frac{R}{{\theta }_N^2},\hfill \\ \frac{L}{2}{\parallel \mathit{x}\parallel }^2,\hfill & \text{otherwise}\hfill \end{array}\phantom{\}}$$ (7)

where OGM iterates remain in the affine region with the same gradient value (without overshooting) for all N iterations. Section 5 shows that a simple quadratic function is also a worst-case function for OGM, and describes why it is interesting that the optimal OGM has these two types of worst-case functions.

Section 3 reviews DT’s Performance Estimation Problem (PEP) framework in [2] that enables systematic worst-case performance analysis of optimization methods. Section 4 provides new convergence analysis for the primary sequence of OGM. Section 5 discusses the two types of worst-case functions for OGM, and Section 6 concludes.

3 Prior Work: Performance Estimation Problem (PEP)

Exploring the convergence performance of optimization methods including class FO has a long history. DT [2] were the first to cast the analysis of the worst-case performance of optimization methods into an optimization problem called PEP, reviewed in this section. We also review how we developed OGM [1] that is built upon DT’s PEP.

3.1 Review of PEP

To analyze the worst-case convergence behavior of a method in class FO having given step sizes h = {h_i,k }_0≤k<i≤N , DT’s PEP [2] bounds the decrease of the cost function after N iterations as

$$\begin{array}{cc}\hfill B_{\text{P}}(\mathit{h},N,d,L,R):& =\underset{\begin{array}{c}\hfill f\in {\mathcal{ℱ}}_L\left({\mathbb{ℝ}}^d\right),\hfill \\ \hfill {\mathit{x}}_0,\cdots ,{\mathit{x}}_N\in {\mathbb{ℝ}}^d,\hfill \\ \hfill {\mathit{x}}_{\ast }\in X_{\ast }\left(f\right)\hfill \end{array}}{\text{max}}f\left({\mathit{x}}_N\right)-f\left({\mathit{x}}_{\ast }\right)\hfill \\ \hfill & \text{s.t}\parallel {\mathit{x}}_0-{\mathit{x}}_{\ast }\parallel \le R,\hfill \\ \hfill & {\mathit{x}}_{i+1}={\mathit{x}}_i-\frac{1}{L}\sum _{k=0}^i{h}_{i+1,k}\nabla f\left({\mathit{x}}_k\right),i=0,\dots ,N-1,\hfill \end{array}$$ (P)

for given dimension d, Lipschitz constant L and the distance R between an initial point x₀ and an optimal point x_∗ ∈ X_∗(f).

Since problem (P) is difficult to solve, DT [2] introduced a series of relaxations. Then the upper bound of the worst-case performance was found numerically in [2] by solving a relaxed PEP problem. For some cases, analytical worst-case bounds were revealed in [1, 2], where some of those analytical bounds were even found to be exact despite the relaxations. On the other hand, Taylor et al. [9] studied the numerical tight worst-case bound of (P) by avoiding DT’s one relaxation step that is not guaranteed to be tight and showing the tightness of the rest of DT’s relaxations in [2] (for the condition “d ≥ N + 2”).

To summarize recent PEP studies, DT extended the PEP approach for nonsmooth convex problems [11], Drori’s thesis [12] includes an extension of PEP to projected gradient methods for constrained smooth convex problems, and Taylor et al. [13] studied PEP for various first-order algorithms for solving composite convex problems. Similarly but using different relaxations of (P), Lessard et al. [14] applied the Integral quadratic constraints to (P), leading to simpler computation but slightly looser convergence upper bounds.

The next two sections review relaxations of DT’s PEP and an approach for optimizing the choice of h for FO using PEP in [2].

3.2 Review of DT’s Relaxation on PEP

This section reviews relaxations introduced by DT to make (P) into a simpler semidefinite programming (SDP) problem. DT first relax the functional constraint f ∈ ℱ_L(ℝ^d) by a well-known property of the class of ℱ_L(ℝ^d) functions in [8, Theorem 2.1.5] and then further relax as follows:

$$\begin{array}{cc}\hfill & B_{P1}(\mathit{h},N,d,L,R)≔\underset{\begin{array}{c}\hfill \mathit{G}\in {\mathbb{ℝ}}^{(N+1)d},\hfill \\ \hfill \delta \in {\mathbb{ℝ}}^{N+1}\hfill \end{array}}{\text{max}}L{R}^2{\delta }_N\hfill \\ \hfill & \text{s.t}\frac{1}{2}{\parallel {\mathit{g}}_{i-1}-{\mathit{g}}_i\parallel }^2\le {\delta }_{i-1}-{\delta }_i-\langle {\mathit{g}}_i\sum _{k=0}^{i-1}{h}_{i,k\mathit{g}k}\rangle ,i=1,\dots ,N,\hfill \\ \hfill & \frac{1}{2}{\parallel {\mathit{g}}_i\parallel }^2\le -{\delta }_i-\langle {\mathit{g}}_i,\sum _{j=1}^i\sum _{k=0}^{j-1}{h}_{j,k}{\mathit{g}}_k+\nu \rangle ,i=0,\dots ,N,\hfill \end{array}$$ (P1)

for any given unit vector ν ∈ ℝ^d, where we denote ${\mathit{g}}_i:=\frac{1}{L\parallel {\mathit{x}}_0-{\mathit{x}}_{\ast }\parallel }\nabla f\left({\mathit{x}}_i\right)$ and ${\delta }_i≔\frac{1}{L{\parallel {\mathit{x}}_0-{\mathit{x}}_{\ast }\parallel }^2}(f\left({\mathit{x}}_i\right)-f\left({\mathit{x}}_{\ast }\right))$ for i = 0, … , N, ∗, and define $\mathit{G}={[{\mathit{g}}_0\cdots {\mathit{g}}_N]}^{\text{⊤}}\in {\mathbb{ℝ}}^{(N+1)\times d}$ and $\delta ={[{\delta }_0\cdots {\delta }_N]}^{\text{⊤}}\in {\mathbb{ℝ}}^{N+1}$.

Maximizing relaxed problem (P1) is still difficult, so DT [2] use a duality approach on (P1). Replacing max_G,δ LR²δ_N by min_G,δ −δ_N for convenience, the Lagrangian of the corresponding constrained minimization problem (P1) with dual variables $\lambda ={({\lambda }_1,\cdots ,{\lambda }_N)}^{\text{⊤}}\in {\mathbb{ℝ}}_{+}^N$ and $\tau ={({\tau }_0,\cdots ,{\tau }_N)}^{\text{⊤}}\in {\mathbb{ℝ}}_{+}^{N+1}$ becomes

$$\begin{array}{cc}\hfill \mathcal{ℒ}(\mathit{G},\delta ,\lambda ,\tau ;\mathit{h})≔& -{\delta }_N+\sum _{i=1}^N{\lambda }_i({\delta }_i-{\delta }_{i-1})+\sum _{i=0}^N{\tau }_i{\delta }_i\hfill \\ \hfill & +\text{Tr}\{{\mathit{G}}^T\mathit{S}(\mathit{h},\lambda ,\tau )\mathit{G}+\nu {\tau }^T\mathit{G}\},\hfill \end{array}$$ (8)

where

$$\{\begin{array}{c}\mathit{S}(\mathit{h},\lambda ,\tau )≔\sum _{i=1}^N{\lambda }_i{\mathit{A}}_{i-1,i}\left(h\right)+\sum _{i=0}^N{\tau }_i{D}_i\left(h\right),\hfill \\ {\mathit{A}}_{i-1,i}\left(\mathit{h}\right)≔\frac{1}{2}({\mathit{u}}_{i-1}-{\mathit{u}}_i){({\mathit{u}}_{i-1}-{\mathit{u}}_i)}^{\text{⊤}}+\frac{1}{2}\sum _{k=0}^{i-1}{h}_{i,k}({\mathit{u}}_i{\mathit{u}}_k^{\text{⊤}}+{\mathit{u}}_k{\mathit{u}}_i^{\text{⊤}}),\hfill \\ {\mathit{D}}_i\left(\mathit{h}\right):=\frac{1}{2}{\mathit{u}}_i{\mathit{u}}_i^{\text{⊤}}+\frac{1}{2}\sum _{j=1}^i\sum _{k=0}^{j-1}{h}_{j,k}({\mathit{u}}_i{\mathit{u}}_k^{\text{⊤}}+{\mathit{u}}_k{\mathit{u}}_i^{\text{⊤}}),\hfill \end{array}\phantom{\}}$$ (9)

and ${\mathit{u}}_i=e_{i+1}\in {\mathbb{ℝ}}^{N+1}$ is the (i + 1)th standard basis vector.

Using further derivations of a duality approach on (8) in [2], the dual problem of (P1) becomes the following SDP problem:

$$\begin{array}{cc}\hfill & B_{\text{P}}(\mathit{h},N,d,L,R)\hfill \\ \hfill & \le B_{\text{D}}(\mathit{h},N,L,R)≔\underset{\begin{array}{c}\hfill (\lambda ,\tau )\in \Lambda ,\hfill \\ \hfill \gamma \in \mathbb{ℝ}\hfill \end{array}}{\text{min}}\left\{\frac{1}{2}L{R}^2\gamma :\left(\begin{array}{cc}\hfill \mathit{S}(\mathit{h},\lambda ,\tau )\hfill & \hfill \frac{1}{2}\tau \hfill \\ \hfill \frac{1}{2}{\tau }^T\hfill & \hfill \frac{1}{2}\gamma \hfill \end{array}\right)\succcurlyeq 0\right\},\hfill \end{array}$$ (D)

where

$$\Lambda =\left\{(\lambda ,\tau )\in {\mathbb{ℝ}}_{+}^N\times {\mathbb{ℝ}}_{+}^{N+1}:\underset{{\lambda }_i-{\lambda }_{i+1}+{\tau }_i=0,i=1,\dots ,N-1,}{\overset{{\tau }_0={\lambda }_1,{\lambda }_N+{\tau }_N=1,}{}}\right\}.$$

Then, for given h, the bound B_D(h, N, L, R) (that is not guaranteed to be tight) can be numerically computed using any SDP solver, while analytical upper bounds B_D(h, N, L, R) for some choices of h were found in [1, 2]. Section 4 finds a new analytical upper bound for a modified version of B_D.

3.3 Review of Optimizing the Step Sizes Using PEP

In addition to finding upper bounds for given FO methods, DT [2] searched for the best FO methods with respect to the worst-case performance. Ideally one would like to optimize h over problem (P):

$${\widehat{\mathit{h}}}_{\text{P}}:=\underset{\mathit{h}\in {\mathbb{ℝ}}^{N(N+1)∕2}}{\text{arg min}}{B}_{\text{P}}(\mathit{h},N,d,L,R).$$ (HP)

However, optimizing (HP) directly seems impractical, so DT minimized the dual problem (D) using a SDP solver over the coefficients h as

$${\widehat{\mathit{h}}}_{\text{D}}:=\underset{\mathit{h}\in {\mathbb{ℝ}}^{N(N+1)∕2}}{\text{arg min}}{B}_{\text{D}}(\mathit{h},N,d,L,R).$$ (HD)

Due to relaxations, the computed ${\widehat{\mathit{h}}}_{\text{D}}$ is not guaranteed to be optimal for problem (HP). Nevertheless, we show in [1] that solving (HD) leads to an algorithm (OGM) having a convergence bound that is twice as small as that of FGM. Interestingly, OGM is optimal among first-order methods with d ≥ N + 1 [4], i.e., ${\widehat{\mathit{h}}}_{\text{D}}$ is a solution of both (HP) and (HD) for d ≥ N + 1. An optimal point $(\widehat{\mathit{h}},\widehat{\lambda },\widehat{\tau },\widehat{\gamma })$ of (HD) is given in [1, Lemma 4 and Proposition 3] as follows:

$${\widehat{h}}_{i+1,k}=\{\begin{array}{cc}\frac{{\theta }_i-1}{{\theta }_{i+1}}{\widehat{h}}_{i,k},\hfill & k=0,\dots ,i-2,\hfill \\ \frac{{\theta }_i-1}{{\theta }_{i+1}}({\widehat{h}}_{i,i-1}-1),\hfill & k=i-1,\hfill \\ 1+\frac{2{\theta }_i-1}{{\theta }_{i+1}},\hfill & k=i,\hfill \end{array}\phantom{\}}$$ (10)

$$=\{\begin{array}{cc}\frac{1}{{\theta }_{i+1}}(2{\theta }_k-\sum _{j=k+1}^i{\widehat{h}}_{j,k}),\hfill & k=0,\dots ,i-1,\hfill \\ 1+\frac{2{\theta }_i-1}{{\theta }_{i+1}},\hfill & k=i,\hfill \end{array}\phantom{\}}$$ (11)

$${\widehat{\lambda }}_i=\frac{2{\theta }_{i-1}^2}{{\theta }_N^2},i=1,\dots ,N,{\widehat{\tau }}_i=\{\begin{array}{cc}\frac{2{\theta }_i}{{\theta }_N^2},\hfill & i=0,\dots ,N-1,\hfill \\ \frac{1}{{\theta }_N},\hfill & i=N,\hfill \end{array}\phantom{\}}\widehat{\gamma }=\frac{1}{{\theta }_N^2}.$$ (12)

Thus both OGM1 and OGM2 satisfy the convergence bound (5) [1, Theorem 2, Propositions 4 and 5].

4 New Convergence Analysis for the Primary Sequence of OGM

4.1 Relaxed PEP for the Primary Sequence of OGM

This section applies PEP to an iterate y_N of the following class of fixed-step first-order methods (FO^′), complementing the worst-case performance of x_N in the previous section.

We first replace f (x_N) − f(x_∗) in (P) by f(y_{N +1}) − f (x_∗) as follows:

$$\begin{array}{cc}\hfill B_{\text{P}\prime }(\mathit{h},N,d,L,R)≔& \underset{\begin{array}{c}\hfill f\in {\mathcal{ℱ}}_L\left({\mathbb{ℝ}}^d\right),\hfill \\ \hfill \mathit{x},\cdots ,{\mathit{x}}_N,{\mathit{y}}_{N+1}\in {\mathbb{ℝ}}^d,\hfill \\ \hfill {\mathit{x}}_{\ast }\in X_{\ast }\left(f\right)\hfill \end{array}}{\text{max}}f\left({\mathit{y}}_{N+1}\right)-f\left({\mathit{x}}_{\ast }\right)\hfill \\ \hfill & \text{s.t.}\parallel {\mathit{x}}_0-{\mathit{x}}_{\ast }\parallel \le R,{\mathit{y}}_{N+1}={\mathit{x}}_N-\frac{1}{L}\nabla f\left({\mathit{x}}_N\right),\hfill \\ \hfill & {\mathit{x}}_{i+1}={\mathit{x}}_i-\frac{1}{L}\sum _{k=0}^i{h}_{i+1,k}\nabla f\left({\mathit{x}}_k\right),i=0,\dots ,N-1.\hfill \end{array}$$ (P′)

We could directly repeat relaxations on (P^′) as reviewed in Section 3.2, but we found it difficult to solve a such relaxed problem of (P^′) analytically. Instead, we use the following inequality [8]:

$$f\left(\mathit{x}-\frac{1}{L}\nabla f\left(\mathit{x}\right)\right)\le f\left(\mathit{x}\right)-\frac{1}{2L}{\parallel \nabla f\left(\mathit{x}\right)\parallel }^2,{\forall }_{\mathit{x}}\in {\mathbb{ℝ}}^d.$$ (13)

to relax (P^′), leading to the following bound:

$$\begin{array}{cc}\hfill B_{\text{P}1\prime }(\mathit{h},N,d,L,R):=& \underset{\begin{array}{c}\hfill f\in {\mathcal{ℱ}}_L\left({\mathbb{ℝ}}^d\right),\hfill \\ \hfill {\mathit{x}}_0,\cdots ,{\mathit{x}}_N\in {\mathbb{ℝ}}^d,\hfill \\ \hfill {\mathit{x}}_{\ast }\in X_{\ast }\left(f\right)\hfill \end{array}}{\text{max}}f\left({\mathit{x}}_N\right)-\frac{1}{2L}{\parallel \nabla f\left({\mathit{x}}_N\right)\parallel }^2-f\left({\mathit{x}}_{\ast }\right)\hfill \\ \hfill & \text{s.t.}\parallel {\mathit{x}}_0-{\mathit{x}}_{\ast }\parallel \le R,\hfill \\ \hfill & {\mathit{x}}_{i+1}={\mathit{x}}_i-\frac{1}{L}\sum _{k=0}^i{h}_{i+1,k}\nabla f\left({\mathit{x}}_k\right),i=0,\dots ,N-1.\hfill \end{array}$$ (P1′)

This bound has an additional term $-\frac{1}{2L}{\parallel \nabla f\left({\mathit{x}}_N\right)\parallel }^2$ compared to (P). We later show that the increase of the worst-case upper bound due to this strict relaxation step using (13) is negligible asymptotically.

Similar to relaxing from (P) to (P1) in Section 3.2, we relax (P1^′) to the following bound:

$$\begin{array}{c}\hfill B_{\text{P}2\prime }(\mathit{h},N,d,L,R)≔\underset{\begin{array}{c}\hfill \mathit{G}\in {\mathbb{ℝ}}^{(N+1)d},\hfill \\ \hfill \delta \in {\mathbb{ℝ}}^{N+1}\hfill \end{array}}{\text{max}}L{R}^2\left({\delta }_N-\frac{1}{2}{\parallel {\mathit{g}}_N\parallel }^2\right)\hfill \\ \hfill \text{s.t.}\frac{1}{2}{\parallel {\mathit{g}}_{i-1}-{\mathit{g}}_i\parallel }^2\le {\delta }_{i-1}-{\delta }_i-\langle {\mathit{g}}_i,\sum _{k=0}^{i-1}{h}_{i,k}{g}_k\rangle ,i=1,\dots ,N,\hfill \\ \hfill \frac{1}{2}{\parallel {\mathit{g}}_i\parallel }^2\le -{\delta }_i-\langle {\mathit{g}}_i,\sum _{j=1}^i\sum _{k=0}^{j-1}{h}_{j,k}{\mathit{g}}_k+\nu \rangle ,i=0,\dots ,N,\hfill \end{array}$$ (P2′)

for any given unit vector ν ∈ ℝ^d. Then, as in Section 3.2 and [1, 2], one can show that the dual problem of (P2^′) is the following SDP problem

$$\begin{array}{cc}\hfill & B_{\text{P}\prime }(\mathit{h},N,d,L,R)\hfill \\ \hfill \le & B_{\text{D}\prime }(\mathit{h},N,L,R)≔\underset{\begin{array}{c}\hfill (\lambda ,\tau )\in \Lambda ,\hfill \\ \hfill \gamma \in \mathbb{ℝ}\hfill \end{array}}{\text{min}}\left\{\frac{1}{2}L{R}^2\gamma :\left(\begin{array}{cc}\hfill \mathit{S}(\mathit{h},\lambda ,\tau )+\frac{1}{2}{\mathit{u}}_N{\mathit{u}}_N^{\text{⊤}}\hfill & \hfill \frac{1}{2}\tau \hfill \\ \hfill \frac{1}{2}\tau \hfill & \hfill \frac{1}{2}\gamma \hfill \end{array}\right)\succcurlyeq 0\right\},\hfill \end{array}$$ (D′)

by considering that the Lagrangian of (P2^′) becomes

$$\begin{array}{cc}\hfill {\mathcal{ℒ}}^{\prime }(\mathit{G},\delta ,\lambda ,\tau ;\mathit{h})≔& -{\delta }_N+\sum _{i=1}^N{\lambda }_i({\delta }_i-{\delta }_{i-1})+\sum _{i=0}^N{\tau }_i{\delta }_i\hfill \\ \hfill & +\text{Tr}\left\{{\mathit{G}}^{\text{⊤}}\left(\mathit{G}(\mathit{h},\lambda ,\tau )+\frac{1}{2}{\mathit{u}}_N{\mathit{u}}_N^{\text{⊤}}\right)\mathit{G}+\nu {\tau }^{\text{⊤}}\mathit{G}\right\}\hfill \end{array}$$ (14)

when we replace ${\text{man}}_{\mathit{G},\delta }L{R}^2({\delta }_N-\frac{1}{2}{\parallel {\mathit{g}}_N\parallel }^2)$ in (P2′) by ${\text{min}}_{\mathit{G},\delta }(-{\delta }_N+\frac{1}{2}{\parallel {\mathit{g}}_N\parallel }^2)$ for simplicity as we did for (P1) and (8). The formulation (14) is similar to (8), except the term $\frac{1}{2}{\mathit{u}}_N{\mathit{u}}_N^{\text{⊤}}$. The derivation of (D^′) and (14) is omitted here, since it is almost identical to the derivation of (D) and (8) in [1, 2].

4.2 Convergence Analysis for the Primary Sequence of OGM

To find an upper bound for (D^′), it suffices to specify a feasible point.

Lemma 4.1 The following choice of $({\widehat{\mathit{h}}}^{\prime },{\widehat{\lambda }}^{\prime },{\widehat{\tau }}^{\prime },{\widehat{\gamma }}^{\prime })$ is a feasible point of (D^′):

$${\widehat{h}}_{i+1,k}^{\prime }=\{\begin{array}{cc}\frac{t_i-1}{t_{i+1}}{\widehat{h}}_{i,k}^{\prime },\hfill & k=0,\dots ,i-2,\hfill \\ \frac{t_i-1}{t_{i+1}}({\widehat{h}}_{i,i-1}^{\prime }-1),\hfill & k=i-1,\hfill \\ 1+\frac{2t_i-1}{t_{i+1}},\hfill & k=i,\hfill \end{array}\phantom{\}}$$ (15)

$$=\{\begin{array}{cc}\frac{1}{t_{i+1}}(2t_k-\sum _{j=k+1}^i{\widehat{h}}_{j,k}^{\prime }),\hfill & k=0,\dots ,i-1,\hfill \\ 1+\frac{2t_i-1}{t_{i+1}},\hfill & k=i,\hfill \end{array}\phantom{\}}$$ (16)

$${\widehat{\lambda }}_i^{\prime }=\frac{t_{i-1}^2}{t_N^2},i=1,\dots ,N,{\widehat{\tau }}_i^{\prime }=\frac{t_i}{t_N^2},i=0,\dots ,N,\widehat{\gamma }\prime =\frac{1}{2t_N^2}.$$ (17)

Proof The equivalency between (15) and (16) follows from [1, Proposition 3]. Also, it is obvious that $({\widehat{\lambda }}^{\prime },{\widehat{\tau }}^{\prime })\in \Lambda$ using $t_i^2={\sum }_{k=0}^i{t}_k$.

We next rewrite $\mathit{S}({\widehat{\mathit{h}}}^{\prime },{\widehat{\lambda }}^{\prime },{\widehat{\tau }}^{\prime })$ to show that the choice $({\widehat{\mathit{h}}}^{\prime },{\widehat{\lambda }}^{\prime },{\widehat{\tau }}^{\prime },{\widehat{\gamma }}^{\prime })$ satisfies the positive semidefinite condition in (D^′). For any h and (λ, τ) ∈ Λ, the (i, k)th entry of the symmetric matrix S(h, λ, τ) in (9) can be written as

$$S_{i,k}(\mathit{h},\lambda ,\tau )=\{\begin{array}{cc}\frac{1}{2}\left(({\lambda }_i+{\tau }_i)h_{i,k}+{\tau }_i\sum _{j=k}^{i-1}{h}_{j,k}\right),\hfill & i=2,\dots ,N,k=0,\dots ,i-2,\hfill \\ \frac{1}{2}(({\lambda }_i+{\tau }_i)h_{i,k}-{\lambda }_i),\hfill & i=1,\dots ,N,k=i-1,\hfill \\ {\lambda }_{i+1},\hfill & i=0,\dots ,N-1,k=i,\hfill \\ \frac{1}{2},\hfill & i=N,k=i.\hfill \end{array}\phantom{\}}$$ (18)

Inserting ${\widehat{\mathit{h}}}^{\prime },{\widehat{\lambda }}^{\prime }$ and ${\widehat{\tau }}^{\prime }$ into (18), we get

$$\begin{array}{cc}\hfill & S_{i,k}({\widehat{\mathit{h}}}^{\prime },{\widehat{\lambda }}^{\prime },\widehat{\tau }\prime )+\frac{1}{2}{\mathit{u}}_N{\mathit{u}}_N^{\text{⊤}}\hfill \\ \hfill & =\{\begin{array}{cc}\frac{1}{2}\left(\frac{t_i^2}{t_N^2}\frac{1}{t_i}\left(2t_k-\sum _{j=k+1}^{i-1}{\widehat{h}}_{j,k}^{\prime }\right)+\frac{t_i}{t_N^2}\sum _{j=k+1}^{i-1}{\widehat{h}}_{j,k}^{\prime }\right),\hfill & i=2,\dots ,N,k=0,\dots ,i-2,\hfill \\ \frac{1}{2}\left(\frac{t_i^2}{t_N^2}\left(1+\frac{2t_{t-1}-1}{t_i}\right)-\frac{t_{i-1}^2}{t_N^2}\right),\hfill & i=1,\dots ,N,k=i-1,\hfill \\ \frac{t_i^2}{t_N^2},\hfill & i=0,\dots ,N-1,k=i,\hfill \\ 1,\hfill & i=N,k=i,\hfill \end{array}\phantom{\}}\hfill \\ \hfill & \frac{t_i{t}_k}{t_N^2}\hfill \end{array}$$

where the second equality uses $t_i^2-t_i-t_{i-1}^2=0$.

Finally, using ${\widehat{\gamma }}^{\prime }$, we have

$$\left(\begin{array}{cc}\hfill \mathit{S}({\widehat{\mathit{h}}}^{\prime },{\widehat{\lambda }}^{\prime },{\widehat{\tau }}^{\prime })+\frac{1}{2}{\mathit{u}}_N{\mathit{u}}_N^{\text{⊤}}\hfill & \hfill \frac{1}{2}\widehat{\tau }\prime \hfill \\ \hfill \frac{1}{2}{\widehat{\tau }}^{{\prime }^{\text{⊤}}}\hfill & \hfill \frac{1}{2}{\widehat{\gamma }}^{\prime }\hfill \end{array}\right)=\left(\begin{array}{cc}\hfill \frac{1}{t_N^2}\mathit{t}{\mathit{t}}^{\text{⊤}}\hfill & \hfill \frac{1}{2t_N^2}\mathit{t}\hfill \\ \hfill \frac{1}{2t_N^2}{\mathit{t}}^{\text{⊤}}\hfill & \hfill \frac{1}{4t_N^2}\hfill \end{array}\right)=\frac{1}{t_N^2}\left(\begin{array}{c}\hfill \mathit{t}\hfill \\ \hfill \frac{1}{2}\hfill \end{array}\right){\left(\begin{array}{c}\hfill \mathit{t}\hfill \\ \hfill \frac{1}{2}\hfill \end{array}\right)}^{\text{⊤}}\succcurlyeq 0,$$

where $\mathit{t}={(t_0,\cdots ,t_N)}^{\text{⊤}}$.

Since $\widehat{\mathit{h}}$ (10) and ${\widehat{\mathit{h}}}^{\prime }$ (15) are identical except for the last iteration, the intermediate iterates ${\{{\mathit{x}}_i,\}}_{i=0}^{N-1}$ of FO with both $\widehat{\mathit{h}}$ and ${\widehat{\mathit{h}}}^{\prime }$ are equivalent. We can also easily notice that the sequence ${\{{\mathit{y}}_i,\}}_{i=0}^N$ of FO^′ with both $\widehat{\mathit{h}}$ and ${\widehat{\mathit{h}}}^{\prime }$ are also identical, implying that both the primary sequence {y_i} of OGM and FO^′ with ${\widehat{\mathit{h}}}^{\prime }$ are equivalent.

Using Lemma 4.1, the following theorem provides an analytical convergence bound for the primary sequence {y_i} of OGM.

Theorem 4.1 Let f ∈ ℱ_L(ℝ^d) and let y₀, · · · , y_N ∈ ℝ^d be generated by OGM1 and OGM2. Then for 1 ≤ i ≤ N , the primary sequence for OGM satisfies:

$$f\left({\mathit{y}}_i\right)-f\left({\mathit{x}}_{\ast }\right)\le \frac{L{R}^2}{4t_{i-1}^2}\le \frac{L{R}^2}{{(i+1)}^2}.$$ (19)

Proof The sequence ${\{{\mathit{y}}_i,\}}_{i=0}^N$ generated by FO^′ with ${\widehat{\mathit{h}}}^{\prime }$ is equivalent to that of OGM1 and OGM2 [1, Propositions 4 and 5].

Using ${\widehat{\gamma }}^{\prime }$ (17) and $t_i^2\ge \frac{{(i+2)}^2}{4}$, we have

$$f\left({\mathit{y}}_N\right)-f\left({\mathit{x}}_{\ast }\right)\le B_{{\text{D}}^{\prime }}({\widehat{\mathit{h}}}^{\prime },N-1,L,R)=\frac{1}{2}L{R}^2{\widehat{\gamma }}^{\prime }=\frac{L{R}^2}{4t_{N-1}^2}\le \frac{L{R}^2}{{(N+1)}^2},$$ (20)

based on Lemma 4.1. Since the primary sequence ${\{{\mathit{y}}_i,\}}_{i=0}^N$ of OGM1 and OGM2 does not depend on a given N , we can extend (20) for all 1 ≤ i ≤ N.

Due to a strict relaxation leading to (P1^′), we cannot guarantee that the bound (19) is tight. However, the next proposition shows that bound (19) is asymptotically tight by specifying one particular worst-case function that was conjectured by Taylor et al. [9, Conjecture 4].

Proposition 4.1 For the following function in ℱ_L(ℝ^d):

$$f_{1,{\text{OGM}}^{\prime }}(\mathit{x};N)=\{\begin{array}{cc}\frac{LR}{2t_{N-1}^2+1}\parallel \mathit{x}\parallel -\frac{L{R}^2}{2{(2t_{N-1}^2+1)}^2},\hfill & if\parallel \mathit{x}\parallel \ge \frac{R}{2t_{N-1}^2+1},\hfill \\ \frac{L}{2}{\parallel \mathit{x}\parallel }^2,\hfill & otherwise,\hfill \end{array}\phantom{\}}$$ (21)

the iterate y_N generated by OGM1 and OGM2 provides the following lower bound:

$$\frac{L{R}^2}{4t_{N-1}^2+2}=f_{1,{\text{OGM}}^{\prime }}({\mathit{y}}_N;N)-f_{1,{\text{OGM}}^{\prime }}({\mathit{x}}_{\ast };N)\le \underset{\begin{array}{c}\hfill f\in {\mathcal{ℱ}}_L\left({\mathbb{ℝ}}^d\right),\hfill \\ \hfill {\mathit{x}}_{\ast }\in X_{\ast }\left(f\right)\hfill \end{array}}{\text{max}}f\left({\mathit{y}}_N\right)-f\left({\mathit{x}}_{\ast }\right).$$ (22)

Proof Starting from x₀ = Rν, where ν is a unit vector, and using the following property of the coefficients ${\widehat{\mathit{h}}}^{\prime }$[1, Equation (8.2)]:

$$\sum _{j=1}^i\sum _{k=0}^{j-1}{\widehat{h}}_{j,k}^{\prime }=t_i^2-1,i=1,\dots ,N,$$ (23)

the primary iterates of OGM1 and OGM2 are as follows

$$\begin{array}{cc}\hfill {\mathit{y}}_i& ={\mathit{x}}_{i-1}-\frac{1}{L}\nabla f_{1,{\text{OGM}}^{\prime }}({\mathit{x}}_{i-1};N)\hfill \\ \hfill & ={\mathit{x}}_0-\frac{1}{L}\sum _{j=1}^{i-1}\sum _{k=0}^{j-1}{\widehat{h}}_{j,k}^{\prime }\nabla f_{1,{\text{OGM}}^{\prime }}({\mathit{x}}_k;N)-\frac{1}{L}\nabla f_{1,{\text{OGM}}^{\prime }}({\mathit{x}}_{i-1};N)\hfill \\ \hfill & =\left(1-\frac{t_{i-1}^2}{2t_{N-1}^2+1}\right)R\nu ,i=1,\dots ,N,\hfill \end{array}$$

where the corresponding sequence $\{{\mathit{x}}_0,\cdots ,{\mathit{x}}_{N-1},{\mathit{y}}_1,\cdots ,{\mathit{y}}_N\}$ stays in the affine region of the function f_1,OGM′ (x; N) with the same gradient value:

$$\nabla f_{1,{\text{OGM}}^{\prime }}({\mathit{x}}_i;N)=\nabla f_{1,{\text{OGM}}^{\prime }}({\mathit{y}}_{i+1};N)=\frac{LR}{2t_{N-1}^2+1}\nu ,i=0,\dots ,N-1.$$

Therefore, after N iterations of OGM1 and OGM2, we have

$$f_{1,{\text{OGM}}^{\prime }}({\mathit{y}}_N;N)-f_{1,{\text{OGM}}^{\prime }}({\mathit{x}}_{\ast };N)=f_{1,{\text{OGM}}^{\prime }}\left(\frac{t_{N-1}^2+1}{2t_{N-1}^2+1}R\nu ;N\right)=\frac{L{R}^2}{2(2t_{N-1}^2+1)},$$

exactly matching the lower bound (22).

The lower bound (22) matches the tight numerical worst-case bound in [9] (see Table 1). While Taylor et al. [9] provide numerical evidence about the tight bound of the primary sequence of OGM, our (22) provides an analytical bound that suffices for asymptotically tight worst-case analysis.

Table 1.

Exact numerical cost function bound $\frac{1}{L{R}^2}(f(\cdot )-f\left({\mathit{x}}_{\ast }\right))$ of the last primary iterate y_N and the last secondary iterate x_N of FGM, OGM and OGM′

N	FGM(prim.)	FGM(sec.)	OGM(prim.)	OGM(sec.)	OGM′(sec)
1	1/6.00	1/6.00	1/6.00	1/8.00	1/5.24
2	1/10.00	1/11.13	1/12.47	1/16.16	1/9.62
3	1/15.13	1/17.35	1/21.25	1/26.53	1/15.12
4	1/21.35	1/24.66	1/32.25	1/39.09	1/21.71
5	1/28.66	1/33.03	1/45.42	1/53.80	1/29.38
10	1/81.07	1/90.69	1/143.23	1/159.07	1/83.54
20	1/263.65	1/283.55	1/494.68	1/525.09	1/269.56
40	1/934.89	1/975.10	1/1810.08	1/1869.22	1/947.55
80	1/3490.22	1/3570.75	1/6866.93	1/6983.13	1/3516.00

4.3 New Formulations of OGM

Using [1, Propositions 4 and 5], Algorithm FO^′ with the coefficients ${\widehat{\mathit{h}}}^{\prime }$ (15) and (16) can be implemented efficiently as follows:

The OGM^′ is very similar to OGM, because it generates same primary and secondary sequence; only the last iterate of the secondary sequence differs. Therefore, the bound (19) applies to the primary sequence {y_i} of both OGM and OGM^′, as summarized in the following corollary.

Corollary 4.1 Let f ∈ ℱ_L(ℝ^d) and let y₀, · · · , y_N ∈ ℝ^d be generated by OGM1^′ and OGM2^′. Then for 1 ≤ i ≤ N ,

$$f\left({\mathit{y}}_i\right)-f\left({\mathit{x}}_{\ast }\right)\le \frac{L{R}^2}{4t_{i-1}^2}\le \frac{L{R}^2}{{(i+1)}^2}.$$ (24)

4.4 Comparing Tight Worst-case Bounds of FGM, OGM and OGM′

While some analytical upper bounds of FGM, OGM and OGM^′ such as (2), (3) (5), (19) and (24) are available for comparison, some of those are tight only asymptotically or some bounds for such algorithms are even unknown analytically. Therefore, we used the code of Taylor et al. [9] for tight (numerical) comparison of algorithms of interest for some given N. Table 1 provides tight numerical bounds of the primary and secondary sequence of FGM, OGM and OGM^′. Interestingly, the worst-case performance of the secondary sequence of OGM^′ is worse than that of FGM sequences, whereas the primary sequence of OGM (and OGM^′) is roughly twice better.

The following proposition uses a quadratic function to define a lower bound on the worst-case performance of OGM1^′ and OGM2^′.

Proposition 4.2 For the following quadratic function in ℱ_L(ℝ^d):

$$f_2\left(\mathit{x}\right)=\frac{L}{2}{\parallel \mathit{x}\parallel }^2$$ (25)

both OGM1^′ and OGM2^′ provide the following lower bound:

$$\frac{L{R}^2}{2t_i^2}=f_2\left({\mathit{x}}_i\right)-f_2\left({\mathit{x}}_{\ast }\right)\le \underset{\begin{array}{c}\hfill f\in {\mathcal{ℱ}}_L\left({\mathbb{ℝ}}^d\right),\hfill \\ \hfill {\mathit{x}}_{\ast }\in X_{\ast }\left(f\right)\hfill \end{array}}{\text{max}}f\left({\mathit{x}}_i\right)-f\left({\mathit{x}}_{\ast }\right),$$ (26)

Proof We use induction to show that the following iterates:

$${\mathit{x}}_i={(-1)}^i\frac{1}{t_i}R\nu ,i=0,\dots ,N,$$ (27)

correspond to the iterates of OGM1^′ and OGM2^′ applied to f₂(x). Starting from x₀ = Rν, where ν is a unit vector, and assuming that (27) holds for i < N, we have

$$\begin{array}{cc}\hfill {\mathit{x}}_{i+1}& ={\mathit{x}}_i-\frac{1}{L}\sum _{k=0}^i{\widehat{h}}_{i+1,k}^{\prime }\nabla f_2\left({\mathit{x}}_k\right)\hfill \\ \hfill & =\left({\mathit{x}}_i-\frac{1}{L}{\widehat{h}}_{i+1,i}^{\prime }\nabla f_2\left({\mathit{x}}_i\right)\right)-\frac{1}{L}\sum _{k=0}^{i-1}\frac{t_i-1}{t_{i+1}}{\widehat{h}}_{i,k}^{\prime }\nabla f_2\left({\mathit{x}}_k\right)+\frac{1}{L}\frac{t_i-1}{t_{i+1}}\nabla f_2\left({\mathit{x}}_{i-1}\right)\hfill \\ \hfill & =\frac{1-2t_i}{t_{i+1}}{\mathit{x}}_i+\frac{t_i-1}{t_{i+1}}({\mathit{x}}_i-{\mathit{x}}_{i-1})+\frac{t_i-1}{t_{i+1}}{\mathit{x}}_{i-1}=-\frac{t_i}{t_{i+1}}{\mathit{x}}_i\hfill \\ \hfill & ={(-1)}^{i+1}\frac{1}{t_{i+1}}R\nu ,\hfill \end{array}$$ (31)

where the second and third equalities use (1) and (15). Therefore, we have

$$f_2\left({\mathit{x}}_N\right)-f_2\left({\mathit{x}}_{\ast }\right)=f_2\left({(-1)}^N\frac{1}{t_N}R\nu \right)=\frac{L{R}^2}{2t_N^2},$$

after N iterations of OGM1^′ and OGM2^′, which is equivalent to the lower bound (26).

Since the analytical lower bound (26) matches the numerical tight bound in Table 1, we conjecture that the quadratic function f₂(x) is the worst-case function for the secondary sequence of OGM^′ and thus (26) is the tight worst-case bound. Whereas FGM has similar worst-case bounds (and behavior as conjectured by Taylor et al. [9, Conjectures 4 and 5]) for both its primary and secondary sequence, the two sequences of OGM^′ (or intermediate iterates of OGM) have two different worst-case behaviors, as discussed further in Section 5.2.

4.5 Related Work

Nesterov’s Accelerated First-order Method in [10] Interestingly, an algorithm in [10, Section 4] is similar to OGM2^′ and satisfies same convergence bound (19) for the primary sequence {y_i}, which we call Nes13 in this paper for convenience.¹

The only difference between OGM2^′ and Nes13 is the gradient used for the update of z_i. While both algorithms achieve same bound (19), Nes13 is less attractive in practice since it requires computing gradients at two different points x_i and y_i+1 at each ith iteration.

Similar to Proposition 4.1, the following proposition shows that the bound (19) is asymptotically tight for Nes13.

Proposition 4.3 For the function f_1,OGM′ (x; N) (21) in ℱ_L(ℝ^d), the iterate y_N generated by Nes13 achieves the lower bound (22).

Proof See the proof of Proposition 4.1.

5 Two Worst-case Functions for an Optimal Fixed-step GM and OGM

This section discusses two algorithms, an optimal fixed-step GM and OGM, in class FO that have a piecewise affine-quadratic function and a quadratic function as two worst-case functions. Considering that OGM is optimal among first-order methods (for d ≥ N +1), it is interesting that OGM has two different types of worst-case functions, because this property resembles the (numerical) analysis of the optimal fixed-step GM in [9] (reviewed below).

5.1 Two Worst-case Functions for an Optimal Fixed-step GM

The following is GM with a constant step size h.

For GM with 0 < h < 2, both [9] and [2] conjecture the following tight convergence bound:

$$f\left({\mathit{x}}_N\right)-f\left({\mathit{x}}_{\ast }\right)\le \frac{L{R}^2}{2}\text{max}\left(\frac{1}{2Nh+1},{(1-h)}^{2N}\right).$$ (28)

The proof of the bound (28) for 0 < h ≤ 1 is given in [2], while the proof for 1 < h < 2 is still unknown but strong numerical evidence is given in [9]. In other words, at least one of the two functions specified below is conjectured to be a worst-case for GM with a constant step size 0 < h < 2. Such functions are a piecewise affine-quadratic function

$$f_{1,\text{GM}}(\mathit{x};h,N)=\{\begin{array}{cc}\frac{LR}{2Nh+1}\parallel \mathit{x}\parallel -\frac{L{R}^2}{2{(2Nh+1)}^2},\hfill & \text{if}\parallel \mathit{x}\parallel \ge \frac{R}{2Nh+1},\hfill \\ \frac{L}{2}{\parallel \mathit{x}\parallel }^2,\hfill & \text{otherwise},\hfill \end{array}\phantom{\}}$$ (29)

and a quadratic function f₂(x) (25), where f_1,GM(x; h, N) and f₂(x) contribute to the factors $\frac{1}{2Nh+1}$ and (1−h)^2N respectively in (28). Here, f_1,GM(x; h, N) is a worst-case function where the GM iterates approach the optimum slowly, whereas f₂(x) is a worst-case function where the iterates overshoot the optimum. (See Fig. 1.)

Fig. 1.

The worst-case performance of the sequence ${\left\{{\mathit{x}}_i\right\}}_{i=0}^N$ of GM with an optimal fixed-step h_opt(N) for N = 2, 5 and d = L = R = 1. The numerically optimized fixed-step sizes for N = 2, 5 are h_opt(2) = 1.6058 and h_opt(5) = 1.7471 [9].

Assuming that the above conjecture for a fixed-step GM holds, Taylor et al. [9] searched (numerically) for the optimal fixed-step size 0 < h_opt(N) < 2 for given N that minimizes the bound (28):

$$h_{\text{opt}}\left(N\right)≔\underset{0<h<2}{\text{arg min}}\text{max}\left(\frac{1}{2Nh+1},{(1-h)}^{2N}\right).$$ (30)

GM with the step h_opt(N) has two worst-case functions f_1,GM(x; h, N) and f₂(x), and must compromise between two extreme cases. On the other hand, the case 0 < h < h_opt(N) has only f_1,GM(x; h, N) as the worst-case and the case h_opt(N) < h < 2 has only f₂(x) as the worst-case. We believe this compromise is inherent to optimizing the worst-case performance of FO methods. The next section shows that the optimal OGM also has this desirable property.

For the special case of N = 1, the optimal OGM reduces to GM with a fixed-step h = 1.5, and this confirms the conjecture in [9] that the step h_opt(1) = 1.5 (30) is optimal for a fixed-step GM with N = 1. However, proving the optimality of h_opt(N) (30) for the fixed-step GM for N > 1 is left as future work.

Fig. 1 visualizes the worst-case performance of GM with the optimal fixed-step h_opt(N) for N = 2 and N = 5. As discussed, for the two worst-case function in Fig. 1, the final iterates reach the same cost function value, where the iterates approach the optimum slowly for f_1,GM(x; h, N), and overshoot for f₂(x).

5.2 Two Worst-case Functions for the Last Iterate x_N of OGM

[1, Theorem 3] showed that f_1,OGM(x; N) (7) is a worst-case function for the last iterate x_N of OGM. The following theorem shows that a quadratic function f₂(x) (25) is also a worst-case function for the last iterate of OGM.

Theorem 5.1 For the quadratic function $f_2\left(\mathit{x}\right)=\frac{L}{2}{\parallel \mathit{x}\parallel }^2$ (25) in ℱ_L(ℝ^d), both OGM1 and OGM2 exactly achieve the convergence bound (5), i.e.,

$$f_2\left({\mathit{x}}_N\right)-f_2\left({\mathit{x}}_{\ast }\right)=\frac{L{R}^2}{2{\theta }_N^2}.$$

Proof We use induction to show that the following iterates:

$${\mathit{x}}_i={(-1)}^i\frac{1}{{\theta }_i}R\nu ,i=0,\dots ,N,$$

correspond to the iterates of OGM1 and OGM2 applied to f₂(x).

Starting from x₀ = Rν, where ν is a unit vector, and assuming that (31) holds for i < N , we have

$$\begin{array}{cc}\hfill {\mathit{x}}_{i+1}& ={\mathit{x}}_i-\frac{1}{L}\sum _{k=0}^i{\widehat{h}}_{i+1,k}\nabla f_2\left({\mathit{x}}_k\right)\hfill \\ \hfill & =\left({\mathit{x}}_i-\frac{1}{L}{\widehat{h}}_{i+1,i}\nabla f_2\left({\mathit{x}}_i\right)\right)-\frac{1}{L}\sum _{k=0}^{i-1}\frac{{\theta }_i-1}{{\theta }_{i+1}}{\widehat{h}}_{i,k}\nabla f_2\left({\mathit{x}}_k\right)+\frac{1}{L}\frac{{\theta }_i-1}{{\theta }_{i+1}}{f}_2\left({\mathit{x}}_{i-1}\right)\hfill \\ \hfill & =\frac{1-2{\theta }_i}{{\theta }_{i+1}}{\mathit{x}}_i+\frac{{\theta }_i-1}{{\theta }_{i+1}}({\mathit{x}}_i-{\mathit{x}}_{i-1})+\frac{{\theta }_i-1}{{\theta }_{i+1}}{\mathit{x}}_{i-1}=-\frac{{\theta }_i}{{\theta }_{i+1}}{\mathit{x}}_i\hfill \\ \hfill & ={(-1)}^{i+1}\frac{1}{{\theta }_{i+1}}R\nu ,\hfill \end{array}$$

where the second and third equalities use (1) and (10). Therefore, we have

$$f_2\left({\mathit{x}}_N\right)-f_2\left({\mathit{x}}_{\ast }\right)=f_2\left({(-1)}^N\frac{1}{{\theta }_N}R\nu \right)=\frac{L{R}^2}{2{\theta }_N^2}$$

after N iterations of OGM1 and OGM2, exactly matching the bound (5).

Thus the last iterate x_N of OGM has two worst case functions: f_1,OGM(x; N) and f₂(x), similar to an optimal fixed-step GM in Section 5.1. Fig. 2 illustrates behavior of OGM for N = 2 and N = 5, where OGM reaches same worst-case cost function value for two different functions f_1,OGM(x; N) and f₂(x) after N iterations.

Fig. 2.

The worst-case performance of the secondary sequence ${\left\{{\mathit{x}}_i\right\}}_{i=0}^N$ of OGM for N = 2, 5 and d = L = R = 1.

In [9, Conjecture 4] and Section 4.2, the primary sequence of OGM is conjectured to have f_1,OGM′ (x; N) as a worst-case function, whereas the quadratic function f₂(x) becomes the best-case as the first primary iterate of OGM reaches the optimum just in one step. On the other hand, Section 4.4 conjectured that f₂(x) is a worst-case function for the secondary sequence of OGM prior to the last iterate. Apparently the primary and secondary sequences of OGM have two extremely different worst-case analyses, whereas the last iterate x_N of OGM compromises between the two worst-case behaviors, making the worst-case behavior of the optimal OGM interesting.

6 Conclusions

We provided an analytical convergence bound for the primary sequence of OGM1 and OGM2, augmenting the bounds of the last iterate of the secondary sequence of OGM in [1]. The corresponding convergence bound is twice as small as that of Nesterov’s FGM, showing that the primary sequence of OGM is faster than FGM. However, interestingly the intermediate iterates of the secondary sequence of OGM were found to be slower than FGM in the worstcase.

We proposed two new formulations of OGM, called OGM1^′ and OGM2^′ that are related closely to Nesterov’s accelerated first-order methods in [10] (originally developed for nonsmooth composite convex functions and differing from FGM in [3, 7]). For smooth problems, OGM and OGM^′ provide faster convergence speed than [10] considering the number of gradient computations required per iteration.

We showed that the last iterate of the secondary sequence of OGM has two types of worst-case functions, a piecewise affine-quadratic function and a quadratic function. In light of the optimality of OGM (for d ≥ N + 1) in [4], it is interesting that OGM has these two types of worst-case functions. Because the optimal fixed-step GM also appears to have two such worst-case functions, one might conjecture that this behavior is a general characteristic of optimal fixed-step first-order methods.

In addition to the optimality of fixed-step first-order methods for the cost function value, studying the optimality for an alternative criteria such as the gradient (||∇f (x_N )||) is an interesting research direction. Just as Nesterov’s FGM was extended for solving nonsmooth composite convex functions [10, 15], it would be interesting to extend OGM to such problems; recently this was numerically studied by Taylor et al. [13]. Incorporating a line-search scheme in [10, 15] to OGM would be also worth investigating, since computing the Lipschitz constant L is sometimes expensive in practice.