ANOTHER LOOK AT THE FAST ITERATIVE SHRINKAGE/THRESHOLDING ALGORITHM (FISTA)

Abstract

This paper provides a new way of developing the “Fast Iterative Shrinkage/Thresholding Algorithm (FISTA)” [3] that is widely used for minimizing composite convex functions with a nonsmooth term such as the ℓ₁ regularizer. In particular, this paper shows that FISTA corresponds to an optimized approach to accelerating the proximal gradient method with respect to a worst-case bound of the cost function. This paper then proposes a new algorithm that is derived by instead optimizing the step coefficients of the proximal gradient method with respect to a worst-case bound of the composite gradient mapping. The proof is based on the worst-case analysis called Performance Estimation Problem in [11].

1. Introduction

The “Fast Iterative Shrinkage/Thresholding Algorithm” (FISTA) [3], also known as a fast proximal gradient method (FPGM) in general, is a very widely used fast first-order method. FISTA’s speed arises from Nesterov’s accelerating technique in [23, 24] that improves the O(1/N) cost function worst-case bound of a proximal gradient method (PGM) to the optimal O(1/N²) rate where N denotes the number of iterations [3].

This paper first provides a new way to develop Nesterov’s acceleration approach, i.e., FISTA (FPGM). In particular, we show that FPGM corresponds to an optimized approach to accelerating PGM with respect to a worst-case bound of the cost function. We then propose a new fast algorithm that is derived from PGM by instead optimizing a worst-case bound of the composite gradient mapping. We call this new method FPGM-OCG (OCG for optimized over composite gradient mapping). This new method provides the best known analytical worst-case bound for decreasing the composite gradient mapping with rate $O(1/N^{\frac{3}{2}})$ among fixed-step first-order methods. The proof is based on the worst-case bound analysis called Performance Estimation Problem (PEP) in [11], which we briefly review next.

Drori and Teboulle’s PEP [11] casts a worst-case analysis for a given optimization method and a given class of optimization problems into a meta-optimization problem. The original PEP has been intractable to solve exactly, so [11] introduced a series of tractable relaxations, focusing on first-order methods and smooth convex minimization problems; this PEP and its relaxations were studied for various algorithms and minimization problem classes in [12, 16, 17, 18, 19, 29, 30]. Drori and Teboulle [11] further proposed to optimize the step coefficients of a given class of optimization methods using a PEP. This approach was studied for first-order methods on unconstrained smooth convex minimization problems in [11], and the authors [17] derived a new first-order method, called an optimized gradient method (OGM) that has an analytic worst-case bound on the cost function that is twice smaller than the previously best known bounds of [23, 24]. Recently, Drori [10] showed that the OGM exactly achieves the optimal cost function worst-case bound among first-order methods for smooth convex minimization (in high-dimensional problems).

Building upon [11] and its successors, Taylor et al. [29] expanded the use of PEP to first-order (proximal gradient) methods for minimizing nonsmooth composite convex functions. They used a tight relaxation¹ for PEP and studied the tight (exact) numerical worst-case bounds of FPGM, a proximal gradient version of OGM, and some variants versus number of iterations N. Their numerical results suggest that there exists an OGM-type acceleration of PGM that has a worst-case cost function bound that is about twice smaller than that of FPGM, showing room for improvement in accelerating PGM. However, it is difficult to derive an analytical worst-case bound for the tightly relaxed PEP in [29], so optimizing the step coefficients of PGM remains an open problem, unlike [11, 17] for smooth convex minimization.

Different from the tightly relaxed PEP in [29], this paper suggests a new (looser) relaxation of a cost function form of PEP for nonsmooth composite convex minimization that simplifies analysis and optimization of step coefficients of PGM, although yields loose worst-case bounds. Interestingly, the resulting optimized PGM numerically appears to be the FPGM. Then, we further provide a new generalized version of FPGM using our relaxed PEP that extends our understanding of the FPGM variants.

This paper next extends the PEP analysis of the gradient norm in [29, 30]. For unconstrained smooth convex minimization, the authors [16] used such PEP to optimize the step coefficients with respect to the gradient norm. The corresponding optimized algorithm can be useful particularly when dealing with dual problems where the gradient norm decrease is important in addition to the cost function minimization (see e.g., [9, 22, 26]). By extending [16], this paper optimizes the step coefficients of the PGM for the composite gradient mapping form of PEP for nonsmooth composite convex minimization. The resulting optimized algorithm differs somewhat from Nesterov’s acceleration and turns out to belong to the proposed generalized FPGM class.

Sec. 2 describes a nonsmooth composite convex minimization problem and first-order (proximal gradient) methods. Sec. 3 proposes a new relaxation of PEP for nonsmooth composite convex minimization problems and the proximal gradient methods, and suggests that the FPGM (FISTA) [3] is the optimized method of the cost function form of this relaxed PEP. Sec. 3 further proposes a generalized version of FPGM using the relaxed PEP. Sec. 4 studies the composite gradient mapping form of the relaxed PEP and describes a new optimized method for decreasing the norm of composite gradient mapping. Sec. 5 compares the various algorithms considered, and Sec. 6 concludes.

2. Problem, methods, and contribution

We consider first-order algorithms for solving the nonsmooth composite convex minimization problem:

$$\underset{\mathit{x}\in {ℝ}^d}{\text{min}}\{F(\mathit{x})≔f(\mathit{x})+\varphi (\mathit{x})\},$$ (M)

under the following assumptions:

• f : ℝ^d → ℝ is is a convex function of the type ${𝒞}_L^{1,1}({ℝ}^d)$, i.e., continuously differentiable with Lipschitz continuous gradient:

  $$\Vert \nabla f(\mathit{x})-\nabla f(\mathit{y})\Vert \le L\Vert \mathit{x}-\mathit{y}\Vert ,\forall \mathit{x},\mathit{y}\in {ℝ}^d,$$ (1)

  where L > 0 is the Lipschitz constant.

• ϕ : ℝ^d → ℝ is proper, closed, convex and “proximal-friendly” [6].

• The optimal set X_*(F) = arg min_x∈ℝ^d F(x) is nonempty, i.e., the problem (M) is solvable.

We use ℱ_L(ℝ^d) to denote the class of functions F that satisfy the above conditions. We additionally assume that the distance between the initial point x₀ and an optimal solution x_* ∈ X(F) is bounded by R > 0, i.e., ‖x₀ − x_*‖ ≤ R.

PGM is a standard first-order method for solving the problem (M) [3, 6], particularly when the following proximal gradient update (that consists of a gradient descent step and a proximal operation [6]) is relatively simple:

$${\mathbf{p}}_L(\mathit{y})≔\underset{\mathit{x}}{\text{arg min}}\{f(\mathit{y})+\langle \mathit{x}-\mathit{y},\nabla f(\mathit{y})\rangle +\frac{L}{2}{\Vert \mathit{x}-\mathit{y}\Vert }^2+\varphi (\mathit{x})\}=\underset{\mathit{x}}{\text{arg min}}\{\frac{L}{2}{\Vert \mathit{x}-(\mathit{y}-\frac{1}{L}\nabla f(\mathit{y}))\Vert }^2+\varphi (\mathit{x})\}.$$ (2)

For ϕ(x) = ‖x‖₁, the update (2) becomes a simple shrinkage/thresholding update, and PGM reduces to an iterative shrinkage/thresholding algorithm (ISTA) [8]. (See [6, Table 10.2] for more functions ϕ(x) that lead to simple proximal operations.) PGM has the following bound on the cost function [3, Thm. 3.1] for any N ≥ 1:

$$F({\mathit{x}}_N)-F({\mathit{x}}_{\ast })\le \frac{{\mathit{\text{LR}}}^2}{2N}.$$ (3)

Algorithm PGM

Input: f ∈ ℱL(ℝd), x0 ∈ ℝd.

For i = 0, …, N − 1

$${\mathit{x}}_{i+1}={\mathbf{p}}_L({\mathit{x}}_i)$$

For simplicity in later derivations, we use the following definition of the composite gradient mapping [27]:

$${\stackrel{\sim }{\nabla }}_{L}F(\mathit{x})≔-L({\mathbf{p}}_L(\mathit{x})-\mathit{x}).$$ (4)

The composite gradient mapping reduces to the usual function gradient ∇f(x) when ϕ(x) = 0. We can then rewrite the PGM update in the following form reminiscent of a gradient method:

$${\mathit{x}}_{i+1}={\mathbf{p}}_L({\mathit{x}}_i)={\mathit{x}}_i-\frac{1}{L}{\stackrel{\sim }{\nabla }}_{L}F({\mathit{x}}_i),$$ (5)

where each update guarantees the following monotonic cost function descent [27, Thm. 1]:

$$F({\mathit{x}}_i)-F({\mathit{x}}_{i+1})\ge \frac{1}{2L}{\Vert {\stackrel{\sim }{\nabla }}_{L}F({\mathit{x}}_i)\Vert }^2.$$ (6)

For any x ∈ ℝ^d, there exists a subgradient ϕ′(p_L(x)) ∈ ∂ϕ(p_L(x)) that satisfies the following equality [3, Lemma 2.2]:

$${\stackrel{\sim }{\nabla }}_{L}F(\mathit{x})=\nabla f(\mathit{x})+\varphi \prime ({\mathbf{p}}_L(\mathit{x})).$$ (7)

This equality implies that any point x̄ with a zero composite gradient mapping (∇̃_LF(x̄) = 0, i.e., x̄ = p_L(x̄)) satisfies 0 ∈ ∂F(x̄) and is a minimizer of (M). As discussed, minimizing the composite gradient mapping is noteworthy in addition to decreasing the cost function. This property becomes particularly important when dealing with dual problems. In particular, it is known that the norm of the dual (sub)gradient is related to the primal feasibility (see e.g., [9, 22, 26]). Furthermore, the norm of the subgradient is upper bounded by the norm of the composite gradient mapping, i.e., for any given subgradients ϕ′(p_L(x)) in (7) and F′(p_L(x)) ≔ ∇f(p_L(x)) + ϕ′(p_L(x)) ∈ ∂F(p_L(x)), we have

$$\Vert F\prime ({\mathbf{p}}_L(\mathit{x}))\Vert \le \Vert \nabla f(\mathit{x})-\nabla f({\mathbf{p}}_L(\mathit{x}))\Vert +\Vert \nabla f(\mathit{x})+\varphi \prime ({\mathbf{p}}_L(\mathit{x}))\Vert \le 2L\Vert \mathit{x}-{\mathbf{p}}_L(\mathit{x})\Vert =2\Vert {\stackrel{\sim }{\nabla }}_{L}F({\mathbf{p}}_L(\mathit{x}))\Vert ,$$ (8)

where the first inequality uses the triangle inequality and the second inequality uses (1) and (7). This inequality provides a close relationship between the primal feasibility and the dual composite gradient mapping. Therefore, we next analyze the worst-case bound of the composite gradient mapping of PGM; Sec. 4 below discusses a first-order algorithm that is optimized with respect to the composite gradient mapping.²

The following lemma shows that PGM monotonically decreases the norm of the composite gradient mapping.

Lemma 1

The PGM monotonically decreases the norm of composite gradient mapping, i.e., for all x:

$$\Vert {\stackrel{\sim }{\nabla }}_{L}F({\mathbf{p}}_L(\mathit{x}))\Vert \le \Vert {\stackrel{\sim }{\nabla }}_{L}F(\mathit{x})\Vert .$$ (9)

Proof

The proof in [22, Lemma 2.4] can be easily extended to prove (9) using the nonexpansiveness of the proximal mapping (proximity operator) [6].

The following theorem provides a O(1/N) bound on the norm of composite gradient mapping for the PGM, using the idea in [26] and Lemma 1.

Theorem 2

Let F : ℝ^d → ℝ be in ℱ_L(ℝ^d) and let x₀, ⋯, x_N ∈ ℝ^d be generated by PGM. Then for N ≥ 2,

$$\underset{i\in \{0,\dots ,N\}}{\text{min}}\Vert {\stackrel{\sim }{\nabla }}_{L}F({\mathit{x}}_i)\Vert =\Vert {\stackrel{\sim }{\nabla }}_{L}F({\mathit{x}}_N)\Vert \le \frac{2\mathit{\text{LR}}}{\sqrt{(N-1)(N+2)}}.$$ (10)

Proof

Let $m=\lfloor \frac{N}{2}\rfloor$, and we have

$$\frac{{\mathit{\text{LR}}}^2}{2m}\stackrel{(3)}{\ge }F({\mathit{x}}_m)-F({\mathit{x}}_{\ast })\stackrel{(6)}{\ge }F({\mathit{x}}_{N+1})-F({\mathit{x}}_{\ast })+\frac{1}{2L}\sum _{i=m}^N{\Vert {\stackrel{\sim }{\nabla }}_{L}F({\mathit{x}}_i)\Vert }^2\stackrel{(9)}{\ge }\frac{N-m+1}{2L}{\Vert {\stackrel{\sim }{\nabla }}_{L}F({\mathit{x}}_N)\Vert }^2,$$

which is equivalent to (10) using $m\ge \frac{N-1}{2}$ and $N-m\ge \frac{N}{2}$.

Despite its inexpensive per-iteration computational cost, PGM suffers from the slow rate O(1/N) for decreasing both the cost function and the norm of composite gradient mapping.³ Therefore for acceleration, this paper considers the following class of fixed-step first-order methods (FSFOM), where the (i + 1)th iteration consists of one proximal gradient evaluation, just like PGM, and a weighted summation of previous and current proximal gradient updates ${\{{\mathit{x}}_{k+1}-{\mathit{y}}_k\}}_{k=0}^i$ with step coefficients ${\{h_{i+1,k}\}}_{k=0}^i$.

Algorithm Class FSFOM

Input: f ∈ ℱL(ℝd), x0 ∈ ℝd, y0 = x0.

For i = 0, …, N − 1

$${\mathit{x}}_{i+1}={\mathbf{p}}_L({\mathit{y}}_i)={\mathit{y}}_i-\frac{1}{L}{\stackrel{\sim }{\nabla }}_{L}F({\mathit{y}}_i)$$

$${\mathit{y}}_{i+1}={\mathit{y}}_i+\sum _{k=0}^i{h}_{i+1,k}({\mathit{x}}_{k+1}-{\mathit{y}}_k)={\mathit{y}}_i-\frac{1}{L}\sum _{k=0}^i{h}_{i+1,k}{\stackrel{\sim }{\nabla }}_{L}F({\mathit{y}}_k).$$

Although the weighted summation in FSFOM seems at first to be inefficient both computationally and memory-wise, the optimized FSFOM presented in this paper have equivalent recursive forms that have memory and computation requirements that are similar to PGM. Note that this class FSFOM includes PGM but excludes accelerated algorithms in [13, 25, 27] that combine the proximal operations and the gradient steps in other ways.

Among FSFOM⁴, FISTA [3], also known as FPGM, is widely used since it has computation and memory requirements that are similar to PGM yet it achieves the optimal O(1/N²) worst-case rate for decreasing the cost function, using Nesterov’s acceleration technique [23, 24].

Algorithm FPGM (FISTA)

Input: f ∈ ℱL(ℝd), x0 ∈ ℝd, y0 = x0, t0 = 1.

For i = 0, …, N − 1

$${\mathit{x}}_{i+1}={\mathbf{p}}_L({\mathit{y}}_i)$$

$$t_{i+1}=\frac{1+\sqrt{1+4t_i^2}}{2}$$ (11)

$${\mathit{y}}_{i+1}={\mathit{x}}_{i+1}+\frac{t_i-1}{t_{i+1}}({\mathit{x}}_{i+1}-{\mathit{x}}_i)$$

FPGM has the following bound for the cost function [3, Thm. 4.4] for any N ≥ 1:

$$F({\mathit{x}}_N)-F({\mathit{x}}_{\ast })\le \frac{{\mathit{\text{LR}}}^2}{2t_{N-1}^2}\le \frac{2{\mathit{\text{LR}}}^2}{{(N+1)}^2},$$ (12)

where the parameters t_i (11) satisfy

$$t_i^2=\sum _{l=0}^i{t}_l\text{ and }{t}_i\ge \frac{i+2}{2}.$$ (13)

Sec. 3 provides a new proof of the cost function bound (12) of FPGM using a new relaxation of PEP, and illustrates that this particular acceleration of PGM results from optimizing a relaxed version of the cost function form of PEP. In addition, it is shown in [3, 5] that FPGM and its bound (12) generalize to any t_i such that t₀ = 1 and $t_i^2\le t_{i-1}^2+t_i$ for all i ≥ 1 with corresponding bound for any N ≥ 1:

$$F({\mathit{x}}_N)-F({\mathit{x}}_{\ast })\le \frac{{\mathit{\text{LR}}}^2}{2t_{N-1}^2},$$ (14)

which includes the choice $t_i=\frac{i+a}{a}$ for any a ≥ 2. Using our relaxed PEP, Sec. 3 further describes similar but different generalizations of FPGM that complement our understanding of FPGM.

We are often interested in the worst-case analysis of the norm of the (composite) gradient (mapping) in addition to that of the cost function, particularly when dealing with dual problems. To improve the rate O(1/N) of the gradient norm bound of a gradient method, Nesterov [26] suggested performing his fast gradient method (FGM) [23, 24], a non-proximal version of FPGM, for the first m iterations and a gradient method for remaining N − m iterations for smooth convex problems (when ϕ(x) = 0). Here we extend this idea to the nonsmooth composite convex problem (M) and use FPGM-m to denote the resulting algorithm. The following theorem provides a $O(1/N^{\frac{3}{2}})$ worst-case bound for the norm of composite gradient mapping of FPGM-m, using the idea in [26] and Lemma 1.

Algorithm FPGM-m

Input: f ∈ ℱL(ℝd), x0 ∈ ℝd, y0 = x0, t0 = 1.

For i = 0, …, N − 1

$${\mathit{x}}_{i+1}={\mathbf{p}}_L({\mathit{y}}_i)$$

$$t_{i+1}=\frac{1+\sqrt{1+4t_i^2}}{2},i\le m-1$$

$${\mathit{y}}_{i+1}=\{\begin{array}{cc}{\mathit{x}}_{i+1}+\frac{t_i-1}{t_i+1}({\mathit{x}}_{i+1}-{\mathit{x}}_i),\hfill & i\le m-1,\hfill \\ {\mathit{x}}_{i+1},\hfill & \text{otherwise}.\hfill \end{array}$$

Theorem 3

Let F : ℝ^d → ℝ be in ℱ_L(ℝ^d) and let x₀, ⋯, x_N ∈ ℝ^d be generated by FPGM-m for 1 ≤ m ≤ N. Then for N ≥ 1,

$$\underset{i\in \{0,\dots ,N\}}{\text{min}}\Vert {\stackrel{\sim }{\nabla }}_{L}F({\mathit{x}}_i)\Vert \le \Vert {\stackrel{\sim }{\nabla }}_{L}F({\mathit{x}}_N)\Vert \le \frac{2\mathit{\text{LR}}}{(m+1)\sqrt{N-m+1}}.$$ (15)

Proof

We have

$$\frac{2{\mathit{\text{LR}}}^2}{{(m+1)}^2}\stackrel{(12)}{\ge }F({\mathit{x}}_m)-F({\mathit{x}}_{\ast })\stackrel{(6)}{\ge }F({\mathit{x}}_{N+1})-F({\mathit{x}}_{\ast })+\frac{1}{2L}\sum _{i=m}^N{\Vert {\stackrel{\sim }{\nabla }}_{L}F({\mathit{x}}_i)\Vert }^2\stackrel{(9)}{\ge }\frac{N-m+1}{2L}{\Vert {\stackrel{\sim }{\nabla }}_{L}F({\mathit{x}}_N)\Vert }^2,$$

which is equivalent to (15).

As noticed by a reviewer, when $m=\lfloor \frac{2N}{3}\rfloor$, the worst-case bound (15) of the composite gradient mapping roughly has its smallest constant $3\sqrt{3}$ for the rate $O(1/N^{\frac{3}{2}})$, which is better than the choice $m=\lfloor \frac{N}{2}\rfloor$ in [26].

Monteiro and Svaiter [21] considered a variant of FPGM that replaces p_L(·) of FPGM by p_L/σ²(·) for 0 < σ < 1; that variant, which we denote FPGM-σ, satisfies the $O(1/N^{\frac{3}{2}})$ rate for the composite gradient mapping. This FPGM-σ algorithm satisfies the following cost function and composite gradient mapping worst-case bounds⁵ [21, Prop. 5.2] for N ≥ 1:

$$F({\mathit{x}}_N)-F({\mathit{x}}_{\ast })\le \frac{2{\mathit{\text{LR}}}^2}{{\sigma }^2{N}^2},$$ (16)

$$\underset{i\in \{0,\dots ,N\}}{\text{min}}\Vert {\stackrel{\sim }{\nabla }}_{L/{\sigma }^2}F({\mathit{y}}_i)\Vert \le \frac{2\sqrt{3}}{\sigma }\sqrt{\frac{1+\sigma }{1-\sigma }}\frac{\mathit{\text{LR}}}{N^{\frac{3}{2}}}.$$ (17)

The worst-case bound (17) of the composite gradient mapping has its smallest constant $\frac{2\sqrt{3}}{{\sigma }^2}\sqrt{\frac{1+\sigma }{1-\sigma }}\approx 16.2$ when $\sigma =\frac{\sqrt{17}-1}{4}\approx 0.78$, which makes the bound (17) about $\frac{16}{3\sqrt{3}}$ ≈ 3-times larger than the bound (15) of $\text{FPGM}-(m=\lfloor \frac{2N}{3}\rfloor )$ at best. However, since FPGM-σ does not require one to select the number of total iterations N in advance unlike FPGM-m, the FPGM-σ algorithm could be useful in practice, as discussed further in Sec. 4.4. Ghadimi and Lan [13] also showed the $O(1/N^{\frac{3}{2}})$ rate for a composite gradient mapping worst-case bound of another variant of FPGM, but the corresponding algorithm in [13] requires two proximal gradient updates per iteration, combining the proximal operations and the gradient steps in a way that differs from the class FSFOM and could be less attractive in terms of the per-iteration computational complexity.

FPGM has been used in dual problems [1, 2, 4, 14]; using FPGM-m and the algorithms in [13, 21] that guarantee $O(1/N^{\frac{3}{2}})$ rate for minimizing the norm of the composite gradient mapping could be potentially useful for solving dual problems. (Using F(P)GM-m for (dual) smooth convex problems was discussed in [9, 22, 26].) However, FPGM-m and the algorithms in [13, 21] are not necessarily the best possible methods with respect to the worst-case bound of the norm of the composite gradient mapping. Therefore, Sec. 4 seeks to optimize the step coefficients of FSFOM for minimizing the norm of the composite gradient mapping using a relaxed PEP.

The next section first provides a new proof of FPGM using our new relaxation on PEP, and proposes the new generalized FPGM.

3. Relaxation and optimization of the cost function form of PEP

3.1. Relaxation for the cost function form of PEP

For FSFOM with given step-size coefficients h ≔ {h_i+1,k}, in principle the worst-case bound on the cost function after N iterations corresponds to the solution of the following PEP problem [11]:

$$\begin{array}{cc}{ℬ}_{\mathrm{P}}(\mathit{h},N,d,L,R)≔\hfill & \underset{\begin{array}{c}\hfill F\in {ℱ}_L({ℝ}^d),\hfill \\ \hfill {\mathit{x}}_0,\cdots ,{\mathit{x}}_N\in {ℝ}^d,{\mathit{x}}_{\ast }\in X_{\ast }(F)\hfill \\ \hfill {\mathit{y}}_0,\cdots ,{\mathit{y}}_{N-1}\in {ℝ}^d\hfill \end{array}}{\text{max}}F({\mathit{x}}_N)-F({\mathit{x}}_{\ast })\hfill \\ \hfill & \mathrm{s}.\mathrm{t}.{\mathit{x}}_{i+1}={\mathbf{p}}_L({\mathit{y}}_i),i=0,\dots ,N-1,\Vert {\mathit{x}}_0-{\mathit{x}}_{\ast }\Vert \le R,\hfill \\ \hfill & \hfill {\mathit{y}}_{i+1}={\mathit{y}}_i+\sum _{k=0}^i{h}_{i+1,k}({\mathit{x}}_{k+1}-{\mathit{y}}_k),i=0,\dots ,N-2.\end{array}$$ (P)

Since (non-relaxed) PEP problems like (P) are difficult to solve due to the (infinite-dimensional) functional constraint on F, Drori and Teboulle [11] suggested (for smooth convex problems) replacing the functional constraint by a property of F related to the update such as p_L(·) in (P). Taylor et al. [29, 30] discussed properties of F that can replace the functional constraint of PEP without strictly relaxing (P), and provided tight numerical worst-case analysis for any given N. However, analytical solutions remain unknown for (P) and most PEP problems.

Instead, this paper proposes an alternate relaxation that is looser than that in [29, 30] but provides tractable and useful analytical results. We consider the following property of F involving the proximal gradient update p_L(·) [3, Lemma 2.3]:

$$F(\mathit{x})-F({\mathbf{p}}_L(\mathit{y}))\le \frac{L}{2}{\Vert {\mathbf{p}}_L(\mathit{y})-\mathit{y}\Vert }^2+L\langle \mathit{y}-\mathit{x},{\mathbf{p}}_L(\mathit{y})-\mathit{y}\rangle {\forall }_{\mathit{x},\mathit{y}}\in {ℝ}^d$$ (18)

to replace the functional constraint on F. In particular, we use the following property:

$$\frac{L}{2}{\Vert {\mathbf{p}}_L(\mathit{y})-\mathit{y}\Vert }^2-L\langle {\mathbf{p}}_L(\mathit{x})-\mathit{x},{\mathbf{p}}_L(\mathit{y})-\mathit{y}\rangle \le F({\mathbf{p}}_L(\mathit{x}))-F({\mathbf{p}}_L(\mathit{y}))+L\langle {\mathbf{p}}_L(\mathit{y})-\mathit{y},\mathit{x}-\mathit{y}\rangle ,{\forall }_{\mathit{x},\mathit{y}}\in {ℝ}^d$$ (19)

that results from replacing x in (18) by p_L(x). When ϕ(x) = 0, the property (19) reduces to

$$\frac{1}{2L}{\Vert \nabla f(\mathit{y})\Vert }^2-\frac{1}{L}\langle \nabla f(\mathit{x}),\nabla f(\mathit{y})\rangle \le f(\mathit{x}-\frac{1}{L}\nabla f(\mathit{x}))-f(\mathit{y}-\frac{1}{L}\nabla f(\mathit{y}))-\langle \nabla f(\mathit{y}),\mathit{x}-\mathit{y}\rangle ,{\forall }_{\mathit{x},\mathit{y}}\in {ℝ}^d.$$ (20)

Note that the relaxation of PEP in [11, 16, 17, 18, 30] for unconstrained smooth convex minimization (ϕ(x) = 0) uses a well-known property of f in [24, Thm. 2.1.5] that differs from (20) and does not strictly relax the PEP as discussed in [30], whereas our relaxation using (19) and (20) does not guarantee a tight relaxation of (P). Finding a tight relaxation that leads to useful (or even optimal) algorithms remains an open problem for nonsmooth composite convex problems.

Similar to [11, Problem (Q′)], we (strictly) relax problem (P) as follows using a set of constraint inequalities (19) at the points (x, y) = (y_i−1, y_i) for i = 1, …, N − 1 and (x, y) = (x_*, y_i) for i = 0, …, N − 1:

$$\begin{array}{cc}{ℬ}_{\mathrm{P}1}(\mathit{h},N,d,L,R)≔\hfill & \underset{\begin{array}{c}\hfill \mathit{G}\in {ℝ}^{N\times d},\hfill \\ \hfill \mathit{\delta }\in {ℝ}^N\hfill \end{array}}{\text{max}}{\mathit{\text{LR}}}^2{\delta }_{N-1}\hfill \\ \hfill & \mathrm{s}.\mathrm{t}.\mathtt{\text{Tr}}\{{\mathit{G}}^{\top }{\check{\mathit{A}}}_{i-1,i}(\mathit{h})\mathit{G}\}\le {\delta }_{i-1}-{\delta }_i,i=1,\dots ,N-1,\hfill \\ \hfill & \mathtt{\text{Tr}}\{{\mathit{G}}^{\top }{\check{\mathit{D}}}_i(\mathit{h})\mathit{G}+\mathit{\nu }{\mathit{u}}_i^{\top }\mathit{G}\}\le -{\delta }_i,i=0,\dots ,N-1,\hfill \end{array}$$ (P1)

for any given unit vector ν ∈ ℝ^d, by defining the (i + 1)th standard basis vector u_i = e_i+1 ∈ ℝ^N, the matrix G = [g₀, ⋯, g_{N − 1}]^⊤ ∈ ℝ^{N × d} and the vector δ = [δ₀, ⋯, δ_{N − 1}]^⊤ ∈ ℝ^N, where

$$\{\begin{array}{c}{\mathit{g}}_i≔-\frac{1}{\Vert {\mathit{y}}_0-{\mathit{x}}_{\ast }\Vert }({\mathbf{p}}_L({\mathit{y}}_i)-{\mathit{y}}_i)=\frac{1}{L\Vert {\mathit{y}}_0-{\mathit{x}}_{\ast }\Vert }{\stackrel{\sim }{\nabla }}_{L}F({\mathit{y}}_i),\hfill \\ {\delta }_i≔\frac{1}{L{\Vert {\mathit{y}}_0-{\mathit{x}}_{\ast }\Vert }^2}(F({\mathbf{p}}_L({\mathit{y}}_i))-F({\mathit{x}}_{\ast })),\hfill \end{array}$$ (21)

for i = 0, …, N − 1, *. Note that g_* = [0, ⋯, 0]^⊤, δ_* = 0 and $\mathtt{\text{Tr}}\{{\mathit{G}}^{\top }{\mathit{u}}_i{\mathit{u}}_j^{\top }\mathit{G}\}=\langle {\mathit{g}}_i,{\mathit{g}}_j\rangle$ by definition. The matrices Ă_i−1,i(h) and Ď_i(h) are defined as

$$\{\begin{array}{c}{\check{\mathit{A}}}_{i-1,i}(\mathit{h})≔\frac{1}{2}{\mathit{u}}_i{\mathit{u}}_i^{\top }-\frac{1}{2}{\mathit{u}}_{i-1}{\mathit{u}}_i^{\top }-\frac{1}{2}{\mathit{u}}_i{\mathit{u}}_{i-1}^{\top }+\frac{1}{2}{\sum }_{k=0}^{i-1}{h}_{i,k}({\mathit{u}}_i{\mathit{u}}_k^{\top }+{\mathit{u}}_k{\mathit{u}}_i^{\top }),\hfill \\ {\check{\mathit{D}}}_i(\mathit{h})≔\frac{1}{2}{\mathit{u}}_i{\mathit{u}}_i^{\top }+\frac{1}{2}{\sum }_{j=1}^i{\sum }_{k=0}^{j-1}{h}_{j,k}({\mathit{u}}_i{\mathit{u}}_k^{\top }+{\mathit{u}}_k{\mathit{u}}_i^{\top }),\hfill \end{array}$$ (22)

which results from the inequalities (19) at the points (x, y) = (y_i−1, y_i) and (x, y) = (x_*, y_i) respectively.

As in [11, Problem (DQ′)], problem (P1) has a dual formulation that one can solve numerically for any given N using a semidefinite program (SDP) to determine an upper bound on the cost function worst-case bound for any FSFOM:⁶

$$F({\mathit{x}}_N)-F({\mathit{x}}_{\ast })\le {ℬ}_{\mathrm{P}}(\mathit{h},N,d,L,R)\le {ℬ}_{\mathrm{D}}(\mathit{h},N,L,R)≔\underset{\begin{array}{c}\hfill (\mathit{\lambda },\mathit{\tau })\in \mathrm{\Lambda },\hfill \\ \hfill \gamma \in ℝ\hfill \end{array}}{\text{max}}\{\frac{1}{2}{\mathit{\text{LR}}}^2\gamma :(\begin{array}{cc}\hfill \mathit{S}(\mathit{h},\mathit{\lambda },\mathit{\tau })\hfill & \hfill \frac{1}{2}\mathit{\tau }\hfill \\ \hfill \frac{1}{2}{\mathit{\tau }}^{\top }\hfill & \hfill \frac{1}{2}\gamma \hfill \end{array})⪰0\},$$ (D)

where $\mathit{\lambda }={[{\lambda }_1,\cdots ,{\lambda }_{N-1}]}^{\top }\in {ℝ}_{+}^{N-1},\mathit{\tau }={[{\tau }_0,\cdots ,{\tau }_{N-1}]}^{\top }\in {ℝ}_{+}^N$, and

$$\mathrm{\Lambda }=\{(\mathit{\lambda },\mathit{\tau })\in {ℝ}_{+}^{2N-1}:\begin{array}{c}{\tau }_0={\lambda }_1,{\lambda }_{N-1}+{\tau }_{N-1}=1,\hfill \\ {\lambda }_i-{\lambda }_{i+1}+{\tau }_i=0,i=1,\dots ,N-2,\hfill \end{array}\},$$ (23)

$$\mathit{S}(\mathit{h},\mathit{\lambda },\mathit{\tau })=\sum _{i=1}^{N-1}{\lambda }_i{\check{\mathit{A}}}_{i-1,i}(\mathit{h})+\sum _{i=0}^{N-1}{\tau }_i{\check{\mathit{D}}}_i(\mathit{h}).$$ (24)

This means that one can compute a valid upper bound (D) of (P) for given step coefficients h using a SDP. The next two sections provide an analytical solution to (D) for FPGM and similarly for our new generalized FPGM, superseding the use of numerical SDP solvers.

3.2. Generalized FPGM

We specify a feasible point of (D) that leads to our new generalized form of FPGM.

Lemma 4

For the following step coefficients:

$$h_{i+1,k}=\{\begin{array}{cc}\frac{t_{i+1}}{T_{i+1}}(t_k-{\sum }_{j=k+1}^i{h}_{j,k}),\hfill & k=0,\dots ,i-1,\hfill \\ 1+\frac{(t_i-1)t_{i+1}}{T_{i+1}},\hfill & k=i,\hfill \end{array}$$ (25)

the choice of variables:

$${\lambda }_i=\frac{T_{i-1}}{T_{N-1}},i=1,\dots ,N-1,{\tau }_i=\frac{t_i}{T_{N-1}},i=0,\dots ,N-1,\gamma =\frac{1}{T_{N-1}},$$ (26)

is a feasible point of (D) for any choice of t_i such that

$$t_0=1,t_i>0,\mathit{\text{ and }}{t}_i^2\le T_i≔\sum _{l=0}^i{t}_l.$$ (27)

Proof

It is obvious that (λ, τ) in (26) with (27) is in Λ (23). Using (22), the (i, k)th entry of the symmetric matrix S(h, λ, τ) in (24) can be written as

$$S_{i,k}(\mathit{h},\mathit{\lambda },\mathit{\tau })=\{\begin{array}{cc}\frac{1}{2}(({\lambda }_i+{\tau }_i)h_{i,k}+{\tau }_i{\sum }_{j=k+1}^{i-1}{h}_{j,k}),\hfill & i=2,\dots ,N-1,k=0,\dots ,i-2,\hfill \\ \frac{1}{2}(({\lambda }_i+{\tau }_i)h_{i,i-1}-{\lambda }_i),\hfill & i=1,\dots ,N-1,k=i-1,\hfill \\ \frac{1}{2}{\lambda }_{i+1},\hfill & i=0,\dots ,N-2,k=i,\hfill \\ \frac{1}{2},\hfill & i=N-1,k=i,\hfill \end{array}$$

where each element S_i,k(h, λ, τ) corresponds to the coefficient of the term ${\mathit{u}}_i{\mathit{u}}_k^{\top }$ of S(h, λ, τ) in (24). Then, inserting (25) and (26) to the above yields

$$S_{i,k}(\mathit{h},\mathit{\lambda },\mathit{\tau })=\{\begin{array}{cc}\frac{1}{2}\left(\frac{T_i}{T_{N-1}}\frac{t_i}{T_i}\right(t_k-{\sum }_{j=k+1}^{i-1}{h}_{j,k})+\frac{t_i}{T_{N-1}}{\sum }_{j=k+1}^{i-1}{h}_{j,k}),\hfill & i=2,\dots ,N-1,k=0,\dots ,i-2,\hfill \\ \frac{1}{2}\left(\frac{T_i}{T_{N-1}}\right(1+\frac{(t_{i-1}-1)t_i}{T_i})-\frac{T_{i-1}}{T_{N-1}}),\hfill & i=1,\dots ,N-1,k=i-1,\hfill \\ \frac{T_i}{2T_{N-1}},\hfill & i=0,\dots ,N-1,k=i.\hfill \end{array}=\{\begin{array}{cc}\frac{t_i{t}_k}{2T_{N-1}},\hfill & i=1,\dots ,N-1,k=0,\dots ,i-1,\hfill \\ \frac{T_i}{2T_{N-1}},\hfill & i=0,\dots ,N-1,k=i.\hfill \end{array}$$

Then, using (26) and (27), we finally show the feasibility condition of (D):

$$\left(\begin{array}{cc}\mathit{S}(\mathit{h},\mathit{\lambda },\mathit{\tau })\hfill & \frac{1}{2}\mathit{\tau }\hfill \\ \frac{1}{2}{\mathit{\tau }}^{\top }\hfill & \frac{1}{2}\gamma \hfill \end{array}\right)=\frac{1}{2T_{N-1}}(\mathtt{\text{diag}}\{\mathit{T}-{\mathit{t}}^2\}+{\mathit{\text{tt}}}^{\top })⪰0,$$

where t = (t₀, ⋯, t_N−1, 1)^⊤ and T = (T₀, ⋯, T_N−1, 1)^⊤.

FSFOM with the step coefficients (25) would be both computationally and memory-wise inefficient, so we next present an equivalent recursive form of FSFOM with (25), named Generalized FPGM (GFPGM).

Algorithm GFPGM

Input: f ∈ ℱL(ℝd), x0 ∈ ℝd, y0 = x0, t0 = T0 = 1.

For i = 0, …, N − 1

$${\mathit{x}}_{i+1}={\mathbf{p}}_L({\mathit{y}}_i)$$

$$\text{Choose }{t}_{i+1}\mathrm{s}.\mathrm{t}.t_{i+1}>0\text{ and }{t}_{i+1}^2\le T_{i+1}≔\sum _{l=0}^{i+1}{t}_l$$

$${\mathit{y}}_{i+1}={\mathit{x}}_{i+1}+\frac{(T_i-t_i)t_{i+1}}{t_i{T}_{i+1}}({\mathit{x}}_{i+1}-{\mathit{x}}_i)+\frac{(t_i^2-T_i)t_{i+1}}{t_i{T}_{i+1}}({\mathit{x}}_{i+1}-{\mathbf{y}}_i)$$

Proposition 5

The sequence {x₀, ⋯, x_N} generated by FSFOM with step sizes (25) is identical to the corresponding sequence generated by GFPGM.

Proof

See Appendix B.

Using Lemma 4, the following theorem bounds the cost function for the GFPGM iterates.

Theorem 6

Let F : ℝ^d → ℝ be in ℱ_L(ℝ^d) and let x₀, ⋯, x_N ∈ ℝ^d be generated by GFPGM. Then for N ≥ 1,

$$F({\mathit{x}}_N)-F({\mathit{x}}_{\ast })\le \frac{{\mathit{\text{LR}}}^2}{2T_{N-1}}.$$ (28)

Proof

Using (D), Lemma 4 and Prop. 5, we have

$$F({\mathit{x}}_N)-F({\mathit{x}}_{\ast })\le {ℬ}_{\mathrm{D}}(\mathit{h},N,L,R)=\frac{1}{2}{\mathit{\text{LR}}}^2\gamma =\frac{{\mathit{\text{LR}}}^2}{2T_{N-1}}.$$ (29)

The GFPGM and Thm. 6 reduce to FPGM and (12) when $t_i^2=T_i$ for all i, and Sec. 3.4 describes that FPGM results from optimizing the step coefficients of FSFOM with respect to the cost function form of the relaxed PEP (D). This GFPGM also includes the choice $t_i=\frac{i+a}{a}$ for any a ≥ 2 as used in [5], which we denote as FPGM-a that differs from the algorithm in [5]. The following corollary provides a cost function worst-case bound for FPGM-a.

Corollary 7

Let F : ℝ^d → ℝ be in ℱ_L(ℝ^d) and let x₀, ⋯, x_N ∈ ℝ^d be generated by GFPGM with $t_i=\frac{i+a}{a}$ (FPGM-a) for any a ≥ 2. Then for N ≥ 1,

$$F({\mathit{x}}_N)-F({\mathit{x}}_{\ast })\le \frac{{\mathit{\text{aLR}}}^2}{N(N+2a-1)}.$$ (30)

Proof

Thm. 6 implies (30), since $t_i=\frac{i+a}{a}$ satisfies (27), i.e.,

$$T_i-t_i^2=\frac{(i+1)(i+2a)}{2a}-\frac{{(i+a)}^2}{a^2}=\frac{(a-2)i^2+a(2a-3)i}{2a^2}\ge 0$$ (31)

for any a ≥ 2 and all i ≥ 0.

3.3. Related work of GFPGM

This section shows that the GFPGM has a close connection to the accelerated algorithm in [25] that was developed specifically for a constrained smooth convex problem with a closed convex set Q, i.e.,

$$\varphi (\mathit{x})={\mathrm{I}}_Q(\mathit{x})≔\{\begin{array}{cc}0,\hfill & \mathit{x}\in Q,\hfill \\ \infty ,\hfill & \text{otherwise}.\hfill \end{array}$$ (32)

The projection operator P_Q(x) ≔ arg min_y∈Q ‖x − y‖ is used for the proximal gradient update (2).

We show that the GFPGM can be written in the following equivalent form, named GFPGM′, which is similar to that of the accelerated algorithm in [25] shown below. Note that the accelerated algorithm in [25] satisfies the bound (28) of the GFPGM in [25, Thm. 2] when ϕ(x) = I_Q(x).

Algorithm GFPGM′

Input: f ∈ ℱL(ℝd), x0 ∈ ℝd, y0 = x0, t0 = T0 = 1.

For i = 0, …, N − 1

$${\mathit{x}}_{i+1}={\mathbf{p}}_L({\mathit{y}}_i)={\mathit{y}}_i-\frac{1}{L}{\stackrel{\sim }{\nabla }}_{L}F({\mathit{y}}_i)$$

$${\mathit{z}}_{i+1}={\mathit{y}}_0-\frac{1}{L}\sum _{k=0}^i{t}_k{\stackrel{\sim }{\nabla }}_{L}F({\mathit{y}}_k)$$

$$\text{Choose }{t}_{i+1}\mathrm{s}.\mathrm{t}.t_{i+1}>0\text{ and }{t}_{i+1}^2\le T_{i+1}=\sum _{l=0}^{i+1}{t}_l$$

$${\mathit{y}}_{i+1}=(1-\frac{t_{i+1}}{T_{i+1}}){\mathit{x}}_{i+1}+\frac{t_{i+1}}{T_{i+1}}{\mathit{z}}_{i+1}$$

Algorithm [25] for ϕ(x) = IQ(x)

Input: f ∈ ℱL(ℝd), x0 ∈ ℝd, y0 = x0, t0 = T0 = 1.

For i = 0, …, N − 1

$${\mathit{x}}_{i+1}={\mathbf{p}}_L({\mathit{y}}_i)={\mathrm{P}}_Q({\mathit{y}}_i-\frac{1}{L}\nabla f({\mathit{y}}_i))$$

$${\mathit{z}}_{i+1}={\mathrm{P}}_Q({\mathit{y}}_0-\frac{1}{L}\sum _{k=0}^i{t}_k\nabla f({\mathit{y}}_k))$$

$$\text{Choose }{t}_{i+1}\mathrm{s}.\mathrm{t}.t_{i+1}>0\text{ and }{t}_{i+1}^2\le T_{i+1}=\sum _{l=0}^{i+1}{t}_l$$

$${\mathit{y}}_{i+1}=(1-\frac{t_{i+1}}{T_{i+1}}){\mathit{x}}_{i+1}+\frac{t_{i+1}}{T_{i+1}}{\mathit{z}}_{i+1}$$

Proposition 8

The sequence {x₀, ⋯, x_N} generated by GFPGM is identical to the corresponding sequence generated by GFPGM′.

Proof

See Appendix C.

Clearly GFPGM′ and the accelerated algorithm in [25] are equivalent for the unconstrained smooth convex problem (Q = ℝ^d). However, when the operation P_Q(x) is relatively expensive, our GFPGM and GFPGM′ that use one projection per iteration could be preferred over the accelerated algorithm in [25] that uses two projections per iteration.

3.4. Optimizing step coefficients of FSFOM using the cost function form of PEP

To find the step coefficients in the class FSFOM that are optimal in terms of the cost function form of PEP, we would like to solve the following problem:

$${\hat{\mathit{h}}}_{\mathrm{P}}≔\underset{\mathit{h}\in {ℝ}^{N(N+1)/2}}{\text{arg min}}{ℬ}_{\mathrm{P}}(\mathit{h},N,d,L,R).$$ (HP)

Because (HP) seems intractable, we instead optimize the step coefficients using the relaxed bound in (D):

$${\hat{\mathit{h}}}_{\mathrm{D}}≔\underset{\mathit{h}\in {ℝ}^{N(N+1)/2}}{\text{arg min}}{ℬ}_{\mathrm{D}}(\mathit{h},N,L,R).$$ (HD)

The problem (HD) is bilinear, and a convex relaxation technique in [11, Thm. 3] makes it solvable using numerical methods. We optimized (HD) numerically for many choices of N using a SDP solver [7, 15] and based on our numerical results (not shown) we conjecture that the feasible point in Lemma 4 with $t_i^2=T_i$ that corresponds to FPGM (FISTA) is a global minimizer of (HD). It is straightforward to show that the step coefficients in Lemma 4 with $t_i^2=T_i$ give the smallest bound of (D) and (28) among all feasible points in Lemma 4, but showing optimality among all possible feasible points of (HD) may require further derivations as in [17, Lemma 3] using KKT conditions, which we leave as future work.

This section has provided a new worst-case bound proof of FPGM using the relaxed PEP, and suggested that FPGM corresponds to FSFOM with optimized step coefficients using the cost function form of the relaxed PEP. The next section provides a different optimization of the step coefficients of FSFOM that targets the norm of the composite gradient mapping, because minimizing the norm of the composite gradient mapping is important in dual problems (see [9, 22, 26] and (8)).

4. Relaxation and optimization of the composite gradient mapping form of PEP

4.1. Relaxation for the composite gradient mapping form of PEP

To form a worst-case bound on the norm of the composite gradient mapping for a given h of FSFOM, we use the following PEP that replaces F(x_N)−F(x_*) in (P) by the norm squared of the composite gradient mapping. Here, we consider the smallest composite gradient mapping norm squared among all iterates⁷ (min_x∈ΩN ‖L(p_L(x) − x)‖² = min_x∈ΩN ‖∇̃_LF(x)‖² where Ω_N ≔ {y₀, ⋯, y_N−1, x_N}) as follows:

$$\begin{array}{cc}{ℬ}_{\mathrm{P}\prime }(\mathit{h},N,d,L,R)≔\hfill & \underset{\begin{array}{c}\hfill F\in {ℱ}_L({ℝ}^d),\hfill \\ \hfill {\mathit{x}}_0,\cdots ,{\mathit{x}}_N\in {ℝ}^d,{\mathit{x}}_{\ast }\in X_{\ast }(F),\hfill \\ \hfill {\mathit{y}}_0,\cdots ,{\mathit{y}}_{N-1}\in {ℝ}^d\hfill \end{array}}{\text{max}}\underset{\mathit{x}\in {\mathrm{\Omega }}_N}{\text{min}}{\Vert L({\mathbf{p}}_L(\mathit{x})-\mathit{x})\Vert }^2\hfill \\ \hfill & \mathrm{s}.\mathrm{t}.{\mathit{x}}_{i+1}={\mathbf{p}}_L({\mathit{y}}_i),i=0,\dots ,N-1,\Vert {\mathit{x}}_0-{\mathit{x}}_{\ast }\Vert \le R,\hfill \\ \hfill & \hfill {\mathit{y}}_{i+1}={\mathit{y}}_i+\sum _{k=0}^i{h}_{i+1,k}({\mathit{x}}_{k+1}-{\mathit{y}}_k),i=0,\dots ,N-2.\end{array}$$ (P′)

Because this infinite-dimensional max-min problem appears intractable, similar to the relaxation from (P) to (P1), we relax (P′) to a finite-dimensional problem with an additional constraint resulting from (6) that is equivalent to

$$\frac{L}{2}{\Vert {\mathbf{p}}_L({\mathit{x}}_N)-{\mathit{x}}_N\Vert }^2\le F({\mathit{x}}_N)-F({\mathit{x}}_{\ast })$$ (33)

and conditions that are equivalent to α ≤ ‖L(p_L(x) − x)‖² for all x ∈ Ω_N after replacing min_x∈ΩN ‖L(p_L(x) − x)‖² by α as in [30].⁸ This relaxation leads to

$$\begin{array}{cc}{ℬ}_{\mathrm{P}1\prime }(\mathit{h},N,d,L,R)≔\hfill & \underset{\begin{array}{c}\hfill \overline{\mathit{G}}\in {ℝ}^{(N+1)\times d},\hfill \\ \hfill \mathit{\delta }\in {ℝ}^N,\alpha \in ℝ\hfill \end{array}}{\text{max}}{L}^2{R}^2\alpha \hfill \\ \hfill & \mathrm{s}.\mathrm{t}.\mathtt{\text{ Tr}}\{{\overline{\mathit{G}}}^{\top }{\overline{\mathit{A}}}_{i-1,i}(\mathit{h})\overline{\mathit{G}}\}\le {\delta }_{i-1}-{\delta }_i,i=1,\dots ,N-1,\hfill \\ \hfill & \mathtt{\text{Tr}}\{{\overline{\mathit{G}}}^{\top }{\overline{\mathit{D}}}_i(\mathit{h})\overline{\mathit{G}}+\mathit{\nu }{\overline{\mathit{u}}}_i^{\top }\overline{\mathit{G}}\}\le -{\delta }_i,i=0,\dots ,N-1,\hfill \\ \hfill & \mathtt{\text{Tr}}\left\{\frac{1}{2}{\overline{\mathit{G}}}^{\top }{\overline{\mathit{u}}}_N{\overline{\mathit{u}}}_N^{\top }\overline{\mathit{G}}\right\}\le {\delta }_{N-1},\hfill \\ \hfill & \mathtt{\text{Tr}}\{-{\overline{\mathit{G}}}^{\top }{\overline{\mathit{u}}}_i{\overline{\mathit{u}}}_i^{\top }\overline{\mathit{G}}\}\le -\alpha ,i=0,\dots ,N,\hfill \end{array}$$ (P1′)

for any given unit vector ν ∈ ℝ^d, by defining the (i + 1)th standard basis vector ū_i = e_i+1 ∈ ℝ^N+1, the matrices

$${\overline{\mathit{A}}}_{i-1,i}(\mathit{h})=\left(\begin{array}{cc}\hfill {\check{\mathit{A}}}_{i-1,i}(\mathit{h})\hfill & \hfill \mathbf{0}\hfill \\ \hfill {\mathbf{0}}^{\top }\hfill & \hfill 0\hfill \end{array}\right),{\overline{\mathit{D}}}_i(\mathit{h})=\left(\begin{array}{cc}\hfill {\check{\mathit{D}}}_i(\mathit{h})\hfill & \hfill \mathbf{0}\hfill \\ \hfill {\mathbf{0}}^{\top }\hfill & \hfill 0\hfill \end{array}\right)$$ (34)

where 0 = [0, …, 0]^⊤ ∈ ℝ^N, and the matrix Ḡ = [G^⊤, ḡ_N]^⊤ ∈ ℝ^(N+1)×d where

$${\overline{\mathit{g}}}_N≔-\frac{1}{\Vert {\mathit{y}}_0-{\mathit{x}}_{\ast }\Vert }({\mathbf{p}}_L({\mathit{x}}_N)-{\mathit{x}}_N)=\frac{1}{L\Vert {\mathit{y}}_0-{\mathit{x}}_{\ast }\Vert }{\stackrel{\sim }{\nabla }}_{L}F({\mathit{x}}_N).$$ (35)

Similar to (D) and [16, Problem (D″)], we have the following dual formulation of (P1′) that could be solved using SDP:

$${ℬ}_{\mathrm{D}\prime }(\mathit{h},N,L,R)≔\underset{\begin{array}{c}\hfill (\mathit{\lambda },\mathit{\tau },\eta ,\mathit{\beta })\in \mathrm{\Lambda }\prime ,\hfill \\ \hfill \gamma \in ℝ\hfill \end{array}}{\text{min}}\{\frac{1}{2}{L}^2{R}^2\gamma :(\begin{array}{cc}\hfill \mathit{S}\prime (\mathit{h},\mathit{\lambda },\mathit{\tau },\eta ,\mathit{\beta })\hfill & \hfill \frac{1}{2}{[{\mathit{\tau }}^{\top },0]}^{\top }\hfill \\ \hfill \frac{1}{2}[{\mathit{\tau }}^{\top },0]\hfill & \hfill \frac{1}{2}\gamma \hfill \end{array})⪰0\}$$ (D′)

where η ∈ ℝ₊, $\mathit{\beta }={[{\beta }_0,\cdots ,{\beta }_N]}^{\top }\in {ℝ}_{+}^{N+1}$, and

$$\mathrm{\Lambda }\prime =\left\{\begin{array}{cc}(\mathit{\lambda },\mathit{\tau },\eta ,\mathit{\beta })\in {ℝ}_{+}^{3N+1}:\hfill & \begin{array}{c}{\tau }_0={\lambda }_1,{\lambda }_{N-1}+{\tau }_{N-1}=\eta ,{\sum }_{i=0}^N{\beta }_i=1,\hfill \\ {\lambda }_i-{\lambda }_{i+1}+{\tau }_i=0,i=1,\dots ,N-2\hfill \end{array}\hfill \end{array}\right\},$$ (36)

$$\mathit{S}\prime (\mathit{h},\mathit{\lambda },\mathit{\tau },\eta ,\mathit{\beta })=\sum _{i=1}^{N-1}{\lambda }_i{\overline{\mathit{A}}}_{i-1,i}(\mathit{h})+\sum _{i=0}^{N-1}{\tau }_i{\overline{\mathit{D}}}_i(\mathit{h})+\frac{1}{2}\eta {\overline{\mathit{u}}}_N{\overline{\mathit{u}}}_N^{\top }-\sum _{i=0}^N{\beta }_i{\overline{\mathit{u}}}_i{\overline{\mathit{u}}}_i^{\top }.$$ (37)

The next section specifies a feasible point of interest that is in the class of GFPGM and analyzes the worst-case bound of the norm of the composite gradient mapping. Then we optimize the step coefficients of FSFOM with respect to the composite gradient mapping form of PEP leading to a new algorithm that differs from Nesterov’s acceleration for decreasing the cost function.

4.2. Worst-case analysis of the composite gradient mapping of GF-PGM

The following lemma provides feasible point of (D′) for the step coefficients (25) of GFPGM.

Lemma 9

For the step coefficients {h_i+1,k} in (25), the choice of variables

$${\lambda }_i=T_{i-1}{\tau }_0,i=1,\dots ,N-1,{\tau }_i=\{\begin{array}{cc}{\left(\frac{1}{2}\right({\sum }_{k=0}^{N-1}(T_k-t_k^2)+T_{N-1}\left)\right)}^{-1},\hfill & i=0,\hfill \\ t_i{\tau }_0,\hfill & i=1,\dots ,N-1,\hfill \end{array}$$ (38)

$$\eta =T_{N-1}{\tau }_0,{\beta }_i=\{\begin{array}{cc}\frac{1}{2}(T_i-t_i^2){\tau }_0,\hfill & i=0,\dots ,N-1,\hfill \\ \frac{1}{2}{T}_{N-1}{\tau }_0,\hfill & i=N,\hfill \end{array}\gamma ={\tau }_0.$$ (39)

is a feasible point of (D′) for any choice of t_i and T_i satisfying (27).

Proof

It is obvious that (λ, τ, η, β) in (38) and (39) with (27) is in Λ′ (36). Using (22) and (34), the (i, k)th entry of the symmetric matrix S′(h, λ, τ, η, β) in (37) can be written as

$$S_{i,k}^{\prime }(\mathit{h},\mathit{\lambda },\mathit{\tau },\eta ,\mathit{\beta })=\{\begin{array}{cc}\frac{1}{2}(({\lambda }_i+{\tau }_i)h_{i,k}+{\tau }_i{\sum }_{j=k+1}^{i-1}{h}_{j,k}),\hfill & i=2,\dots ,N-1,k=0,\dots ,i-2,\hfill \\ \frac{1}{2}(({\lambda }_i+{\tau }_i)h_{i,i-1}-{\lambda }_i),\hfill & i=1,\dots ,N-1,k=i-1,\hfill \\ \frac{1}{2}{\lambda }_{i+1}-{\beta }_i,\hfill & i=0,\dots ,N-2,k=i,\hfill \\ \frac{1}{2}\eta -{\beta }_i,\hfill & i=N-1,N,k=i,\hfill \\ 0,\hfill & i=N,k=0,\dots ,i-1,\hfill \end{array}$$

and inserting (25), (38), and (39) yields

$$S_{i,k}^{\prime }(\mathit{h},\mathit{\lambda },\mathit{\tau },\eta ,\mathit{\beta })=\{\begin{array}{cc}\frac{1}{2}\left(T_i{\tau }_0\frac{t_i}{T_i}\right(t_k-{\sum }_{j=k+1}^{i-1}{h}_{j,k})+t_i{\tau }_0{\sum }_{j=k+1}^{i-1}{h}_{j,k}),\hfill & i=2,\dots ,N-1,k=0,\dots ,i-2,\hfill \\ \frac{1}{2}\left(T_i{\tau }_0\right(1+\frac{(t_{i-1}-1)t_i}{T_i})-T_{i-1}{\tau }_0),\hfill & i=1,\dots ,N-1,k=i-1,\hfill \\ \frac{1}{2}{T}_i{\tau }_0-\frac{1}{2}(T_i-t_i^2){\tau }_0,\hfill & i=0,\dots ,N-1,k=i,\hfill \\ 0,\hfill & i=N,k=0,\dots ,i,\hfill \end{array}=\{\begin{array}{cc}\frac{1}{2}{t}_i{t}_k{\tau }_0,\hfill & i=0,\dots ,N-1,k=0,\dots ,i,\hfill \\ 0,\hfill & i=N,k=0,\dots ,i.\hfill \end{array}$$

Finally, by defining t̄ = (t₀, ⋯, t_N−1, 0, 1)^⊤ we have the feasibility condition of (D′):

$$\left(\begin{array}{cc}\hfill \mathit{S}\prime (\mathit{h},\mathit{\lambda },\mathit{\tau },\eta ,\mathit{\beta })\hfill & \hfill \frac{1}{2}{[{\mathit{\tau }}^{\top },0]}^{\top }\hfill \\ \hfill \frac{1}{2}[{\mathit{\tau }}^{\top },0]\hfill & \hfill \frac{1}{2}\gamma \hfill \end{array}\right)=\frac{1}{2}\overline{t}\overline{t}{\tau }_0⪰0.$$

Using Lemma 9, the following theorem bounds the (smallest) norm of the composite gradient mapping for the GFPGM iterates.

Theorem 10

Let f : ℝ^d → ℝ be in ℱ_L(ℝ^d) and let x₀, ⋯, x_N, y₀, ⋯, y_N−1 ∈ ℝ^d be generated by GFPGM. Then for N ≥ 1,

$$\underset{i\in \{0,\dots ,N\}}{\text{min}}\Vert {\stackrel{\sim }{\nabla }}_{L}F({\mathit{x}}_i)\Vert \le \underset{\mathit{x}\in {\mathrm{\Omega }}_N}{\text{min}}\Vert {\stackrel{\sim }{\nabla }}_{L}F(\mathit{x})\Vert \le \frac{\mathit{\text{LR}}}{\sqrt{{\sum }_{k=0}^{N-1}(T_k-t_k^2)+T_{N-1}}}.$$ (40)

Proof

Lemma 1 implies the first inequality of (40). Using (D′), Lemma 9 and Prop. 5, we have

$$\underset{\mathit{x}\in {\mathrm{\Omega }}_N}{\text{min}}{\Vert {\stackrel{\sim }{\nabla }}_{L}F(\mathit{x})\Vert }^2\le {ℬ}_{\mathrm{D}\prime }(\mathit{h},N,L,R)=\frac{1}{2}{L}^2{R}^2\gamma =\frac{L^2{R}^2}{{\sum }_{k=0}^{N-1}(T_k-t_k^2)+T_{N-1}},$$

which is equivalent to (40).

Although the bound (40) is not tight due to the relaxation on PEP, next two sections show that there exists choices of t_i that provide a rate $O(1/N^{\frac{3}{2}})$ for decreasing the composite gradient mapping, including the choice that optimizes the composite gradient mapping form of PEP.

FGM for smooth convex minimization was shown to achieve the rate $O(1/N^{\frac{3}{2}})$ for the decrease of the usual gradient in [16]. In contrast, Thm. 10 provides only a O(1/N) bound for FPGM (or GFPGM with t_i (11)) on the decrease of the composite gradient mapping since $T_i=t_i^2$ for all i and the value of T_N−1 is O(N²) for t_i (11). Sec. 5 below numerically studies a tight bound on the composite gradient mapping of FPGM and illustrates that it has a rate that is faster than the rate O(1/N) of Thm. 10, indicating there is a room for improvement in the composite gradient mapping form of the relaxed PEP

4.3. Optimizing step coefficients of FSFOM using the composite gradient mapping form of PEP

To optimize the step coefficients in the class FSFOM in terms of the composite gradient mapping form of the relaxed PEP (D′), we would like to solve the following problem:

$${\hat{\mathit{h}}}_{\mathrm{D}\prime }≔\underset{\mathit{h}\in {ℝ}^{N(N+1)/2}}{\text{arg min}}{ℬ}_{\mathrm{D}\prime }(\mathit{h},N,L,R).$$ (HD′)

Similar to (HD), we use a convex relaxation [11, Thm. 3] to make the bilinear problem (HD′) solvable using numerical methods. We then numerically optimized (HD′) for many choices of N using a SDP solver [7, 15] and found that the following choice of t_i:

$$t_i=\{\begin{array}{cc}1,\hfill & i=0,\hfill \\ \frac{1+\sqrt{1+4t_{i-1}^2}}{2},\hfill & i=1,\dots ,\lfloor \frac{N}{2}\rfloor -1,\hfill \\ \frac{N-i+1}{2},\hfill & i=\lfloor \frac{N}{2}\rfloor ,\dots ,N-1,\hfill \end{array}$$ (41)

makes the feasible point in Lemma 9 optimal empirically with respect to the relaxed bound (HD′). Interestingly, whereas the usual t_i factors (such as (11) and $t_i=\frac{i+a}{a}$ for any a ≥ 2) increase with i indefinitely, here, the factors begin decreasing after $i=\lfloor \frac{N}{2}\rfloor -1$.

We also noticed numerically that finding the t_i that minimizes the bound (40), i.e., solving the following constrained quadratic problem:

$$\underset{\{t_i\}}{\text{max}}\left\{\sum _{k=0}^{N-1}\right(\sum _{l=0}^k{t}_l-t_k^2)+\sum _{l=0}^{N-1}{t}_l\}\mathrm{s}.\mathrm{t}.t_i\text{ satisfies }(27)\text{ for all }i,$$ (42)

is equivalent to optimizing (HD′). This means that the solution of (42) numerically appears equivalent to (41), the (conjectured) solution of (HD′). Interestingly, the unconstrained maximizer of (42) without the constraint (27) is $t_i=\frac{N-i+1}{2}$, and this partially appears in the constrained maximizer (41) of the problem (42).

Based on this numerical evidence, we conjecture that the solution ĥ_D′ of problem (HD′) corresponds to (25) with (41). Using Prop. 5, the following GFPGM form with (41) is equivalent to FSFOM with the step coefficients (25) for (41) that are optimized step coefficients of FSFOM with respect to the norm of the composite gradient mapping, which we name FPGM-OCG (OCG for optimized over composite gradient mapping).

Algorithm FPGM-OCG (GFPGM with ti in (41))

Input: $f\in C_L^{1,1}({ℝ}^d)$ convex, x0 ∈ ℝd, y0 = x0, t0 = T0 = 1.

For i = 0, …, N − 1

$${\mathit{x}}_{i+1}={\mathbf{p}}_L({\mathit{y}}_i)$$

$$t_{i+1}=\{\begin{array}{cc}\frac{1+\sqrt{1+4t_i^2}}{2},\hfill & i=1,\dots ,\lfloor \frac{N}{2}\rfloor -2,\hfill \\ \frac{N-i}{2},\hfill & i=\lfloor \frac{N}{2}\rfloor -1,\dots ,N-2,\hfill \end{array}$$

$${\mathit{y}}_{i+1}={\mathit{x}}_{i+1}+\frac{(T_i-t_i)t_{i+1}}{t_i{T}_{i+1}}({\mathit{x}}_{i+1}-{\mathit{x}}_i)+\frac{(t_i^2-T_i)t_{i+1}}{t_i{T}_{i+1}}({\mathit{x}}_{i+1}-{\mathit{y}}_i),i<N=1$$

The following theorem bounds the cost function and the (smallest) norm of the composite gradient mapping for the FPGM-OCG iterates.

Theorem 11

Let F : ℝ^d → ℝ be in ℱ_L(ℝ^d) and let x₀, ⋯, x_N, y₀, ⋯, y_N−1 ∈ ℝ^d be generated by FPGM-OCG. Then for N ≥ 1,

$$F({\mathit{x}}_N)-F({\mathit{x}}_{\ast })\le \frac{4L{\Vert {\mathit{x}}_0-{\mathit{x}}_{\ast }\Vert }^2}{N(N+4)},$$ (43)

and for N ≥ 3,

$$\underset{i\in \{0,\dots ,N\}}{\text{min}}\Vert {\stackrel{\sim }{\nabla }}_{L}F({\mathit{x}}_i)\Vert \le \underset{\mathit{x}\in {\mathrm{\Omega }}_N}{\text{min}}\Vert {\stackrel{\sim }{\nabla }}_{L}F(\mathit{x})\Vert \le \frac{2\sqrt{6}\mathit{\text{LR}}}{N\sqrt{N-2}}.$$ (44)

Proof

FPGM-OCG is an instance of the GFPGM, and thus Thm. 6 implies (43) using

$$\begin{gathered} T_{N-1}=T_{m-1}+\sum _{k=m}^{N-1}{t}_k=t_{m-1}^2+\sum _{k=m}^{N-1}\frac{N-k+1}{2}=t_{m-1}^2 \\ +\sum _{k\prime =2}^{N-m+1}\frac{k\prime }{2}\ge \frac{{(m+1)}^2}{4}+\frac{(N-m+1)(N-m+2)}{4}-\frac{1}{2}\ge \frac{2N^2+8N+1}{16}, \end{gathered}$$

where $m=\lfloor \frac{N}{2}\rfloor \ge \frac{N-1}{2},N-m\ge \frac{N}{2}$, and $T_{m-1}=t_{m-1}^2\ge \frac{{(m+1)}^2}{4}$ (13).

In addition, Thm. 10 implies (44), using

$$\sum _{k=0}^{N-1}(T_k-t_k^2)+T_{N-1}\ge \frac{1}{24}(N-2)N^2,$$ (45)

which we prove in the Appendix E.

The composite gradient mapping bound (44) of FPGM-OCG is asymptotically $\frac{2\sqrt{2}}{3}$-times smaller than the bound (15) of $\text{FPGM}-(m=\lfloor \frac{2N}{3}\rfloor )$. In addition, the cost function bound (43) of FPGM-OCG satisfies the optimal rate O(1/N²), although the bound (43) is two-times larger than the analogous bound (12) of FPGM.

4.4. Decreasing the composite gradient mapping with a rate $O(1/N^{\frac{3}{2}})$ without selecting N in advance

FPGM-OCG and FPGM-m satisfy a fast rate $O(1/N^{\frac{3}{2}})$ for decreasing the norm of the composite gradient mapping but require one to select the total number of iterations N in advance, which could be undesirable in practice. One could use FPGM-σ in [21] that does not require selecting N in advance, but instead we suggest a new choice of t_i in GFPGM that satisfies a composite gradient mapping bound that is lower than the bound (17) of FPGM-σ.

Based on Thm. 10, the following corollary shows that GFPGM with $t_i=\frac{i+a}{a}$ (FPGM-a) for any a > 2 satisfies the rate $O(1/N^{\frac{3}{2}})$ of the norm of the composite gradient mapping without selecting N in advance. (Cor. 7 showed that FPGM-a for any a ≥ 2 satisfies the optimal rate O(1/N²) of the cost function.)

Corollary 12

Let f : ℝ^d → ℝ be in ℱ^L(ℝ^d) and let x₀, ⋯, x_N, y₀, ⋯, y_N−1 ∈ ℝ^d be generated by GFPGM with $t_i=\frac{i+a}{a}$ (FPGM-a) for any a ≥ 2. Then for N ≥ 1, we have the following bound on the (smallest) composite gradient mapping:

$$\underset{i\in \{0,\dots ,N\}}{\text{min}}\Vert {\stackrel{\sim }{\nabla }}_{L}F({\mathit{x}}_i)\Vert \le \underset{\mathit{x}\in {\mathrm{\Omega }}_N}{\text{min}}\Vert {\stackrel{\sim }{\nabla }}_{L}F(\mathit{x})\Vert \le \frac{a\sqrt{6}\mathit{\text{LR}}}{\sqrt{N((a-2)N^2+3(a^2-a+1)N+(3a^2+2a-1))}}.$$ (46)

Proof

With $T_i=\frac{(i+1)(i+2a)}{2a}$ and (31), Thm. 10 implies (46) using

$$\sum _{k=0}^{N-1}(T_k-t_k^2)+T_{N-1}=\sum _{k=0}^{N-1}(\frac{(k+1)(k+2a)}{2a}-\frac{{(k+a)}^2}{a^2})+\frac{N(N+2a-1)}{2a}=\sum _{k=0}^{N-1}\left(\frac{(a-2)k^2+a(2a-3)k}{2a^2}\right)+\frac{N(N+2a-1)}{2a}=\frac{N}{2a^2}(\frac{(a-2)(N-1)(2N-1)}{6}+\frac{a(2a-3)(N-1)}{2}+a(N+2a-1))=\frac{N((a-2)N^2+3(a^2-a+1)N+(3a^2+2a-1))}{6a^2}.$$

FPGM-a for any a > 2 has a composite gradient mapping bound (46) that is asymptotically $\frac{a}{2\sqrt{a-2}}$-times larger than the bound (44) of FPGM-OCG. This gap reduces to $\sqrt{2}$ at best when a = 4, which is clearly better than that of FPGM-σ. Therefore, this FPGM-a algorithm will be useful for minimizing the composite gradient mapping with a rate $O(1/N^{\frac{3}{2}})$ without selecting N in advance.

5. Discussion

5.1. Summary of analytical worst-case bounds on the cost function and the composite gradient mapping

Table 1 summarizes the asymptotic worst-case bounds of all algorithms discussed in this paper. (Note that the bounds are not guaranteed to be tight.) In Table 1, FPGM and FPGM-OCG provide the best known analytical worst-case bounds for decreasing the cost function and the composite gradient mapping respectively. When one does not want to select N in advance for decreasing the composite gradient mapping, FPGM-a will be a useful alternative to FPGM-OCG.

Table 1.

Asymptotic worst-case bounds on the cost function F(x_N) − F(x_*) and the norm of the composite gradient mapping min_x∈ΩN ‖∇̃_LF(x)‖ of PGM, FPGM, FPGM-σ, FPGM-m, FPGM-OCG, and FPGM-a. (The cost function bound for FPGM-m in the table corresponds to the bound for FPGM after m iterations because a tight bound for the final N th iteration is unknown. The bound on min_{i∈{0, …, N}} ‖∇̃_L/σ²F(y_i)‖ is used for FPGM-σ.)

Algorithm	Asymptotic worst-case bound		Require selecting N in advance
	Cost function (×LR2)	Proximal gradient (×LR)
PGM	$\frac{1}{2}{N}^{-1}$	2N−1	No
FPGM	2N−2	2N−1	No
FPGM-σ (0 < σ < 1)	$\frac{2}{{\sigma }^2}{N}^{-2}$	$\frac{2\sqrt{3}}{{\sigma }^2}\sqrt{\frac{1+\sigma }{1-\sigma }}{N}^{-\frac{3}{2}}$	No
FPGM-(σ = 0.78)	3.3N−2	$16.2N^{-\frac{3}{2}}$
$\text{FPGM}-(m=\lfloor \frac{2N}{3}\rfloor )$	4.5N−2	$5.2N^{-\frac{3}{2}}$	Yes
FPGM-OCG	4N−2	$\mathbf{4.9}{\mathit{N}}^{\mathbf{-}\frac{\mathbf{3}}{\mathbf{2}}}$	Yes
FPGM-a (a > 2)	aN −2	$\frac{a\sqrt{6}}{\sqrt{a-2}}{N}^{-\frac{3}{2}}$	No
FPGM-(a = 4)	4N−2	$6.9N^{-\frac{3}{2}}$

5.2. Tight worst-case bounds on the cost function and the smallest composite gradient mapping norm

Since none of the bounds presented in Table 1 are guaranteed to be tight, we modified the code⁹ (using SDP solvers [20, 28]) in Taylor et al. [29] to compare tight (numerical) bounds for the cost function and the composite gradient mapping in Tables 2 and 3 respectively for N = 1, 2, 4, 10, 20, 30, 40, 47, 50. This numerical bound is guaranteed to be tight when the large-scale condition is satisfied [29]. Taylor et al. [29, Fig. 1] already studied a tight worst-case bound on the cost function decrease of FPGM numerically, and found that the analytical bound (12) is asymptotically tight. Table 2 additionally provides numerical tight bounds on the cost function of all algorithms presented in this paper, also suggesting that our relaxation of the cost function form of the PEP from (P) to (D) is asymptotically tight (for some algorithms). In addition, the trend of the tight bounds of the composite gradient mapping in Table 3 follows that of the bounds in Table 1. However, there is gap between them that is not asymptotically tight, unlike the gap between the bounds of the cost function in Tables 1 and 2. In particular, the numerical tight bound for the composite gradient mapping of FPGM in Table 3 has a rate faster than the known rate O(1/N) in Thm. 10. We leave reducing this gap for the bounds on the norm of the composite gradient mapping as future work, possibly with a tighter relaxation of PEP. In addition, $\text{FPGM}-(m=\lfloor \frac{2N}{3}\rfloor )$ has a numerical tight bound in Table 3 that is even slightly better than that of FPGM-OCG, unlike our expectation from the analytical bounds in Table 1 and Sec. 4.3. This shows room for improvement in optimizing the step coefficients of FSFOM with respect to the composite gradient mapping, again possibly with a tighter relaxation of PEP.

Table 2.

Tight worst-case bounds on the cost function LR²/(F(x_N) − F(x_*)) of PGM, FPGM, FPGM-(σ = 0.78), $\text{FPGM}-(m=\lfloor \frac{2N}{3}\rfloor )$, FPGM-OCG, and FPGM-(a = 4). We computed empirical rates by assuming that the bounds follow the form bN^c with constants b and c, and then by estimating c from points N = 47, 50. Note that the corresponding empirical rates are underestimated due to the simplified exponential model.

N	PGM	FPGM	FPGM -(σ = 0.78)	$\text{FPGM}-(m=\lfloor \frac{2N}{3}\rfloor )$	FPGM -OCG	FPGM -(a = 4)
1	4.00	4.00	2.43	4.00	4.00	4.00
2	8.00	8.00	4.87	8.00	8.00	8.00
4	16.00	19.35	11.77	17.13	17.60	17.23
10	40.00	79.07	48.11	56.47	59.25	55.88
20	80.00	261.66	159.19	163.75	170.10	159.17
30	120.00	546.51	332.49	321.56	331.97	312.03
40	160.00	932.89	567.57	502.37	544.55	514.73
47	188.00	1263.58	768.76	675.68	723.06	686.33
50	200.00	1420.45	864.20	752.90	807.66	767.37
Empi. O(·)	N −1.00	N −1.89	N −1.89	N −1.75	N −1.79	N −1.80
Known O(·)	N −1	N −2	N −2	N −2	N −2	N −2

Table 3.

Tight worst-case bounds on the norm of the composite gradient mapping LR/(min_x∈ΩN ‖∇̃_LF(x)‖) of PGM, FPGM, FPGM-(σ = 0.78), $\text{FPGM}-(m=\lfloor \frac{2N}{3}\rfloor )$, FPGM-OCG, and FPGM-(a = 4). Empirical rates were computed as described in Table 2. (The bound for FPGM-σ uses min_x∈ΩN ‖∇̃_L/σ²F(x)‖.)

N	PGM	FPGM	FPGM -(σ = 0.78)	$\text{FPGM}-(m=\lfloor \frac{2N}{3}\rfloor )$	FPGM -OCG	FPGM -(a=4)
1	1.84	1.84	1.18	1.84	1.84	1.84
2	2.83	2.83	1.78	2.83	2.83	2.83
4	4.81	5.65	3.50	5.09	5.21	5.12
10	10.80	13.24	8.74	14.91	15.60	14.76
20	20.78	27.19	18.83	39.70	39.61	29.21
30	30.78	43.49	30.82	64.45	64.40	47.14
40	40.78	61.76	44.39	92.82	91.99	67.82
47	47.77	75.60	54.73	113.92	113.41	83.67
50	50.77	81.78	59.35	123.54	123.17	90.78
Empi. O(·)	N −0.98	N −1.27	N −1.31	N −1.31	N −1.33	N −1.32
Known O(·)	N −1	N −1	$N^{-\frac{3}{2}}$	$N^{-\frac{3}{2}}$	$N^{-\frac{3}{2}}$	$N^{-\frac{3}{2}}$

5.3. Tight worst-case bounds on the final compo site gradient mapping

This paper focused on analyzing the worst-case bound of the smallest composite gradient mapping among all iterates (min_x∈ΩN ‖∇̃_LF(x)‖) in addition to the cost function, whereas the composite gradient mapping at the final iterate (‖∇̃_LF(x_N)‖) could be also considered (see Appendix D). For example, the composite gradient mapping bounds (10) and (15) for PGM and FPGM-m also apply to the final composite gradient mapping, and using (6) we can easily derive a (loose) worst-case bound on the final composite gradient mapping for other algorithms, e.g., such a final composite gradient mapping bound for GFPGM is as follows:

$$\Vert {\stackrel{\sim }{\nabla }}_{L}F({\mathit{x}}_N)\Vert \stackrel{(6)}{\le }\sqrt{2L(F({\mathit{x}}_N)-F({\mathit{p}}_L({\mathit{x}}_N)))}\le \sqrt{2L(F({\mathit{x}}_N)-F({\mathit{x}}_{\ast }))}\stackrel{(28)}{\le }\frac{\mathit{\text{LR}}}{\sqrt{T_{N-1}}}.$$ (47)

Since the optimal rate for decreasing the cost function is O(1/N²), the composite gradient mapping worst-case bound (47) can provide only a rate O(1/N) at best. For completeness of the discussion, Table 4 reports tight numerical bounds for the final composite gradient mapping. Here, FPGM, FPGM-(σ = 0.78), and FPGM-(a = 4) have empirical rates of the worst-case bounds in Table 4 that are slower than those in Table 3, unlike the other three including FPGM-OCG.

Table 4.

Tight worst-case bounds on the norm of the final composite gradient mapping LR/‖∇̃_LF(x_N)‖ of PGM, FPGM, FPGM-(σ = 0.78), $\text{FPGM}-(m=\lfloor \frac{2N}{3}\rfloor )$, FPGM-OCG, and FPGM-(a = 4). Empirical rates were computed as described in Table 2. (The bound for FPGM-σ uses ‖∇̃_L/σ²F(x_N)‖.)

N	PGM	FPGM	FPGM -(σ = 0.78)	$\text{FPGM}-(m=\lfloor \frac{2N}{3}\rfloor )$	FPGM -OCG	FPGM -(a = 4)
1	1.84	1.84	1.18	1.84	1.84	1.84
2	2.83	2.83	1.78	2.83	2.83	2.83
4	4.81	5.65	3.50	5.09	5.21	5.12
10	10.80	12.68	8.41	14.91	15.60	14.76
20	20.78	22.02	14.26	39.65	39.10	25.96
30	30.78	31.26	20.12	64.40	63.40	34.21
40	40.78	40.46	25.97	92.78	90.16	42.39
47	47.77	46.89	30.06	113.92	110.12	48.13
50	50.77	49.65	31.81	123.53	118.99	50.59
Empi. O(·)	N −0.98	N −0.92	N −0.92	N −1.31	N −1.25	N −0.81
Known O(·)	N −1	N −1	N −1	$N^{-\frac{3}{2}}$	N −1	N −1

To best of our knowledge, FPGM-m (or algorithms that similarly perform accelerated algorithms in the beginning and run a proximal gradient method for the remaining iterations) is known only to have a rate $O(1/N^{\frac{3}{2}})$ in (15) for decreasing the final composite gradient mapping, while FPGM-OCG was also found to inherit such fast rate in Table 4. Therefore, searching for first-order methods that have a worst-case bound on the final composite gradient mapping that is lower than that of FPGM-m (and FPGM-OCG), and that possibly do not require knowing N in advance is an interesting open problem. Note that a regularization technique in [26] that provides a faster rate O(1/N²) (up to a logarithmic factor) for decreasing the final gradient norm for smooth convex minimization can be easily extended for rapidly minimizing the final composite gradient mapping with such rate for the composite problem (M); however, that approach requires knowing R in advance.

5.4. Tight worst-case bounds on the final subgradient

This paper has mainly focused on the norm of the composite gradient mapping based on (8), instead of the subgradient norm that is of primary interest in the dual problem (see e.g., [9, 22, 26]). Therefore to have a better sense of subgradient norm bounds, we computed tight numerical bounds on the final¹⁰ subgradient norm ‖F′(x_N)‖ in Table 5 and compared them with Table 4.

Table 5.

Tight worst-case bounds on the subgradient norm LR/‖F′(x_N)‖ of PGM, FPGM, FPGM-(σ = 0.78), $\text{FPGM}-(m=\lfloor \frac{2N}{3}\rfloor )$, FPGM-OCG, and FPGM-(a = 4), where F′(x) ∈ ∂F(x) is a subgradient. Empirical rates were computed as described in Table 2.

N	PGM	FPGM	FPGM -(σ = 0.78)	$\text{FPGM}-(m=\lfloor \frac{2N}{3}\rfloor )$	FPGM -OCG	FPGM -(a = 4)
1	1.00	1.00	0.61	1.00	1.00	1.00
2	2.00	2.00	1.22	2.00	2.00	2.00
4	4.00	4.83	2.94	4.28	4.40	4.31
10	10.00	7.60	4.67	14.12	14.81	12.10
20	20.00	12.58	7.67	38.29	36.65	16.85
30	30.00	17.63	10.74	62.71	60.40	21.61
40	40.00	22.67	13.80	91.00	86.62	26.47
47	47.00	26.20	15.94	112.01	106.21	29.91
50	50.00	27.71	16.86	121.53	114.93	31.39
Empi. O(·)	N −1.00	N −0.91	N −0.91	N −1.32	N −1.27	N −0.78
Known O(·)	N −1	N −1	N −1	$N^{-\frac{3}{2}}$	N −1	N −1

For all six algorithms, empirical rates in Table 5 are similar to those for the final composite gradient mapping in Table 4. In particular, the subgradient norm bounds for the three algorithms PGM, $\text{FPGM}-(m=\lfloor \frac{2N}{3}\rfloor )$, and FPGM-OCG are almost identical to those in Table 4 except for the first few iterations, eliminating the concern of using (8) for such cases. On the other hand, the other three algorithms FPGM, FPGM-(σ = 0.78), and FPGM-(a = 4) almost tightly satisfy the inequality (8) for most N, and thus have bounds on the final subgradient that are about twice larger than those on the final composite gradient mapping. Therefore, regardless of (8), Table 5 further supports the use of $\text{FPGM}-(m=\lfloor \frac{2N}{3}\rfloor )$ and FPGM-OCG over FPGM and other algorithms in dual problems.

6. Conclusion

This paper analyzed and developed fixed-step first-order methods (FSFOM) for nonsmooth composite convex cost functions. We showed an alternate proof of FPGM (FISTA) using PEP, and suggested that FPGM (FISTA) results from optimizing the step coefficients of FSFOM with respect to the cost function form of the (relaxed) PEP. We then described a new generalized version of FPGM and analyzed its worst-case bound using the (relaxed) PEP over both the cost function and the norm of the composite gradient mapping. Furthermore, we optimized the step coefficients of FSFOM with respect to the composite gradient mapping form of the (relaxed) PEP, yielding FPGM-OCG, which could be useful particularly when tackling dual problems.

Our relaxed PEP provided tractable analysis of the optimized step coefficients of FSFOM with respect to the cost function and the norm of the composite gradient mapping, but the relaxation is not guaranteed to be tight and the corresponding accelerations of PGM (FPGM and FPGM-OCG) are thus unlikely to be optimal. Therefore, finding optimal step coefficients of FSFOM over the cost function and the norm of the composite gradient mapping remain as future work. Nevertheless, the proposed FPGM-OCG that optimizes the composite gradient mapping form of the relaxed PEP and the FPGM-a (for any a > 2) may be useful in dual problems.

Software

Matlab codes for the SDP approaches in Sec. 3.4, Sec. 4.3 and Sec. 5 are available at https://gitlab.eecs.umich.edu/michigan-fast-optimization.

Appendix A

Derivation of the dual formulation (D) of (P1)

The derivation below is similar to [11, Lemma 2].

We replace max_G,δ LR²δ_N−1 of (P1) by min_G,δ{−δ_N−1} for convenience in this section. The corresponding dual function of such (P1) is then defined as

$$H(\mathit{\lambda },\mathit{\tau };\mathit{h})=\underset{\begin{array}{c}\hfill \mathit{G}\in {ℝ}^{N\times d},\hfill \\ \hfill \mathit{\delta }\in {ℝ}^N\hfill \end{array}}{\text{min}}\{ℒ(\mathit{G},\mathit{\delta },\mathit{\lambda },\mathit{\tau };\mathit{h})≔{ℒ}_1(\mathit{\delta },\mathit{\lambda },\mathit{\tau })+{ℒ}_2(\mathit{G},\mathit{\lambda },\mathit{\tau };\mathit{h})\}$$

for dual variables $\mathit{\lambda }={[{\lambda }_1,\cdots ,{\lambda }_{N-1}]}^{\top }\in {ℝ}_{+}^{N-1}$ and $\mathit{\tau }={[{\tau }_0,\cdots ,{\tau }_{N-1}]}^{\top }\in {ℝ}_{+}^N$, where ℒ(G, δ, λ, τ; h) is a Lagrangian function, and

$${ℒ}_1(\mathit{\delta },\mathit{\lambda },\mathit{\tau })≔-{\delta }_{N-1}+\sum _{i=1}^{N-1}{\lambda }_i({\delta }_i-{\delta }_{i-1})+\sum _{i=0}^{N1}{\tau }_i{\delta }_i,$$

$${ℒ}_2(\mathit{G},\mathit{\lambda },\mathit{\tau };\mathit{h})≔\sum _{i=1}^{N-1}{\lambda }_i\mathtt{\text{ Tr}}\{{\mathit{G}}^{\top }{\check{\mathit{A}}}_{i-1,i}(\mathit{h})\mathit{G}\}+\sum _{i=0}^{N-1}{\tau }_i\mathtt{\text{ Tr}}\{{\mathit{G}}^{\top }{\check{\mathit{D}}}_i(\mathit{h})\mathit{G}+\mathit{\nu }{\mathit{u}}_i^{\top }\mathit{G}\}.$$

Here, min_δ ℒ₁(δ, λ, τ) = 0 for any (λ, τ) ∈ Λ where Λ is defined in (23), and min_δ ℒ₁(δ, λ, τ) = −∞ otherwise.

For any given unit vector ν, [11, Lemma 1] implies

$$\underset{\mathit{G}\in {ℝ}^{N\times d}}{\text{min}}{ℒ}_2(\mathit{G},\mathit{\lambda },\mathit{\tau })=\underset{\mathit{w}\in {ℝ}^N}{\text{min}}{ℒ}_2(\mathit{w}{\mathit{\nu }}^{\top },\mathit{\lambda },\mathit{\tau }),$$

and thus for any (λ, τ) ∈ Λ, we can rewrite the dual function as

$$H(\mathit{\lambda },\mathit{\tau };\mathit{h})=\underset{\mathit{w}\in {ℝ}^N}{\text{min}}\{{\mathit{w}}^{\top }\mathit{S}(\mathit{h},\mathit{\lambda },\mathit{\tau })\mathit{w}+{\tau }^{\top }\mathit{w}\}=\underset{\gamma \in ℝ}{\text{max}}\{-\frac{1}{2}\gamma :{\mathit{w}}^{\top }\mathit{S}(\mathit{h},\mathit{\lambda },\mathit{\tau })\mathit{w}+{\mathit{\tau }}^{\top }\mathit{w}\ge -\frac{1}{2}\gamma ,\forall \mathit{w}\in {ℝ}^N\}=\underset{\gamma \in ℝ}{\text{max}}\{-\frac{1}{2}\gamma :(\begin{array}{cc}\hfill \mathit{S}(\mathit{h},\mathit{\lambda },\mathit{\tau })\hfill & \hfill \frac{1}{2}\mathit{\tau }\hfill \\ \hfill \frac{1}{2}{\mathit{\tau }}^{\top }\hfill & \hfill \frac{1}{2}\gamma \hfill \end{array})⪰0\},$$

where S(h, λ, τ) is defined in (24). Therefore the dual problem of (P1) becomes (D), recalling that we previously replaced max_G,δ LR²δ_N−1 of (P1) by min_G,δ{−δ_N−1}.

Appendix B

Proof of Prop. 5

The proof is similar to [17, Prop. 2, 3 and 4].

We first show that {h_i+1,k} in (25) is equivalent to

$$h_{i+1,k}=\{\begin{array}{cc}\frac{(T_i-t_i)t_{i+1}}{t_i{T}_{i+1}}{h}_{i,k}\hfill & i=0,\dots ,N-1,k=0,\dots ,i-2,\hfill \\ \frac{(T_i-t_i)t_{i+1}}{t_i{T}_{i+1}}(h_{i,i-1}-1),\hfill & i=0,\dots ,N-1,k=i-1,\hfill \\ 1+\frac{(t_i-1)t_{i+1}}{T_{i+1}},\hfill & i=0,\dots ,N-1,k=i,\hfill \end{array}$$ (48)

We use the notation $h_{i,k}^{\prime }$ for the coefficients (25) to distinguish from (48). It is obvious that $h_{i+1,i}^{\prime }=h_{i+1,i}$, i = 0, …, N − 1, and we clearly have

$$h_{i+1,i-1}^{\prime }=\frac{t_{i+1}}{T_{i+1}}(t_{i-1}-h_{i,i-1}^{\prime })=\frac{t_{i+1}}{T_{i+1}}(t_{i-1}-(1+\frac{(t_{i-1}-1)t_i}{T_i}\left)\right)=\frac{(t_{i-1}-1)(T_i-t_i)t_{i+1}}{T_i{T}_{i+1}}=\frac{(T_i-t_i)t_{i+1}}{t_i{T}_{i+1}}(h_{i,i-1}-1)=h_{i+1,i-1}$$

We next use induction by assuming $h_{i+1,k}^{\prime }=h_{i+1,k}$ for i = 0, …, n − 1, k = 0, …, i. We then have

$$h_{n+1,k}^{\prime }=\frac{t_{n+1}}{T_{n+1}}(t_k-\sum _{j=k+1}^n{h}_{j,k}^{\prime })=\frac{t_{n+1}}{T_{n+1}}(t_k-\sum _{j=k+1}^{n-1}{h}_{j,k}^{\prime }-h_{n,k}^{\prime })=\frac{t_{n+1}}{T_{n+1}}(\frac{T_n}{t_n}{h}_{n,k}^{\prime }-h_{n,k}^{\prime })=\frac{(T_n-t_n)t_{n+1}}{t_n{T}_{n+1}}{h}_{n,k}=h_{n+1,k}$$

Next, using (48), we show that FSFOM with (25) is equivalent to the GFPGM. We use induction, and for clarity, we use the notation ${\mathit{y}}_0^{\prime },\cdots ,{\mathit{y}}_N^{\prime }$ for FSFOM with (48). It is obvious that ${\mathit{y}}_0^{\prime }={\mathit{y}}_0$, and we have

$${\mathit{y}}_1^{\prime }={\mathit{y}}_0^{\prime }-\frac{1}{L}{h}_{1,0}{\stackrel{\sim }{\nabla }}_{L}F({\mathit{y}}_0^{\prime })={\mathit{y}}_0-\frac{1}{L}(1+\frac{(t_0-1)t_1}{T_1}){\stackrel{\sim }{\nabla }}_{L}F({\mathit{y}}_0)={\mathit{x}}_1+\frac{(T_0-t_0)t_1}{t_0{T}_1}({\mathit{x}}_1-{\mathit{x}}_0)+\frac{(t_0^2-T_0)t_1}{t_0{T}_1}({\mathit{x}}_1-{\mathit{y}}_0)={\mathit{y}}_1,$$

since T₀ = t₀. Assuming ${\mathit{y}}_i^{\prime }={\mathit{y}}_i$ for i = 0, …, n, we then have

$${\mathit{y}}_{n+1}^{\prime }={\mathit{y}}_n^{\prime }-\frac{1}{L}{h}_{n+1,n}{\stackrel{\sim }{\nabla }}_{L}F({\mathit{y}}_n^{\prime })-\frac{1}{L}{h}_{n+1,n-1}{\stackrel{\sim }{\nabla }}_{L}F({\mathit{y}}_{n-1}^{\prime })-\frac{1}{L}\sum _{k=0}^{n-2}{h}_{n+1,k}{\stackrel{\sim }{\nabla }}_{L}F({\mathit{y}}_k^{\prime })={\mathit{y}}_n-\frac{1}{L}(1+\frac{(t_n-1)t_{n+1}}{T_{n+1}}){\stackrel{\sim }{\nabla }}_{L}F({\mathit{y}}_n)-\frac{1}{L}\frac{(T_n-t_n)t_{n+1}}{t_n{T}_{n+1}}(h_{n,n-1}-1){\stackrel{\sim }{\nabla }}_{L}F({\mathit{y}}_{n-1})-\frac{1}{L}\sum _{k=0}^{n-2}\frac{(T_n-t_n)t_{n+1}}{t_n{T}_{n+1}}{h}_{n,k}{\stackrel{\sim }{\nabla }}_{L}F({\mathit{y}}_k)={\mathit{x}}_{n+1}-\frac{1}{L}\frac{(t_n^2-T_n)t_{n+1}}{t_n{T}_{n+1}}{\stackrel{\sim }{\nabla }}_{L}F({\mathit{y}}_n)-\frac{1}{L}\frac{(T_n-t_n)t_{n+1}}{t_n{T}_{n+1}}({\stackrel{\sim }{\nabla }}_{L}F({\mathit{y}}_n)-{\stackrel{\sim }{\nabla }}_{L}F({\mathit{y}}_{n-1})+\sum _{k=0}^{n-1}{h}_{n,k}{\stackrel{\sim }{\nabla }}_{L}F({\mathit{y}}_k))={\mathit{x}}_{n+1}+\frac{(t_n^2-T_n)t_{n+1}}{t_n{T}_{n+1}}({\mathit{x}}_{n+1}-{\mathit{y}}_n)+\frac{(T_n-t_n)t_{n+1}}{t_n{T}_{n+1}}(-\frac{1}{L}{\stackrel{\sim }{\nabla }}_{L}F({\mathit{y}}_n)+\frac{1}{L}{\stackrel{\sim }{\nabla }}_{L}F({\mathit{y}}_{n-1})+{\mathit{y}}_n-{\mathit{y}}_{n-1})={\mathit{x}}_{n+1}+\frac{(T_n-t_n)t_{n+1}}{t_n{T}_{n+1}}({\mathit{x}}_{n+1}-{\mathit{x}}_n)+\frac{(t_n^2-T_n)t_{n+1}}{t_n{T}_{n+1}}({\mathit{x}}_{n+1}-{\mathit{y}}_n)={\mathit{y}}_{n+1}.$$

Appendix C

Proof of Prop. 8

The proof is similar to [17, Prop. 1 and 5].

We use induction, and for clarity, we use the notation ${\mathit{y}}_0^{\prime },\cdots ,{\mathit{y}}_N^{\prime }$ for FSFOM with (25) that is equivalent to GFPGM by Prop. 5. It is obvious that ${\mathit{y}}_0^{\prime }={\mathit{y}}_0$, and we have

$${\mathit{y}}_1^{\prime }={\mathit{y}}_0^{\prime }-\frac{1}{L}{h}_{1,0}{\stackrel{\sim }{\nabla }}_{L}F({\mathit{y}}_0^{\prime })={\mathit{y}}_0-\frac{1}{L}(1+\frac{(t_0-1)t_1}{T_1}){\stackrel{\sim }{\nabla }}_{L}F({\mathit{y}}_0)=(1-\frac{t_1}{T_1})({\mathit{y}}_0-\frac{1}{L}{\stackrel{\sim }{\nabla }}_{L}F({\mathit{y}}_0))+\frac{t_1}{T_1}({\mathit{y}}_0-\frac{1}{L}{t}_0{\stackrel{\sim }{\nabla }}_{L}F({\mathit{y}}_0))=(1-\frac{t_1}{T_1}){\mathit{x}}_1+\frac{t_1}{T_1}{\mathit{z}}_1={\mathit{y}}_1.$$

Assuming ${\mathit{y}}_0^{\prime }={\mathit{y}}_i$ for i = 0, …, n, we then have

$${\mathit{y}}_{n+1}^{\prime }={\mathit{y}}_n^{\prime }-\frac{1}{L}{h}_{n+1,n}{\stackrel{\sim }{\nabla }}_{L}F({\mathit{y}}_n^{\prime })-\frac{1}{L}\sum _{k=0}^{n-1}{h}_{n+1,k}{\stackrel{\sim }{\nabla }}_{L}F({\mathit{y}}_k^{\prime })={\mathit{y}}_n-\frac{1}{L}(1+\frac{(t_n-1)t_{n+1}}{T_{n+1}}){\stackrel{\sim }{\nabla }}_{L}F({\mathit{y}}_n)-\frac{1}{L}\sum _{k=0}^{n-1}\frac{t_{n+1}}{T_{n+1}}(t_k-\sum _{j=k+1}^n{h}_{j,k}){\stackrel{\sim }{\nabla }}_{L}F({\mathit{y}}_k)=(1-\frac{t_{n+1}}{T_{n+1}})({\mathit{y}}_n-\frac{1}{L}{\stackrel{\sim }{\nabla }}_{L}F({\mathit{y}}_n))+\frac{t_{n+1}}{T_{n+1}}({\mathit{y}}_n-\frac{1}{L}\sum _{k=0}^n{t}_k{\stackrel{\sim }{\nabla }}_{L}F({\mathit{y}}_k)+\frac{1}{L}\sum _{k=0}^{n-1}\sum _{j=k+1}^n{h}_{j,k}{\stackrel{\sim }{\nabla }}_{L}F({\mathit{y}}_k))=(1-\frac{t_{n+1}}{T_{n+1}})({\mathit{y}}_n-\frac{1}{L}{\stackrel{\sim }{\nabla }}_{L}F({\mathit{y}}_n))+\frac{t_{n+1}}{T_{n+1}}({\mathit{y}}_0-\frac{1}{L}\sum _{k=0}^n{t}_k{\stackrel{\sim }{\nabla }}_{L}F({\mathit{y}}_k))=(1-\frac{t_{n+1}}{T_{n+1}}){\mathit{x}}_{n+1}+\frac{t_{n+1}}{T_{n+1}}{\mathit{z}}_{n+1}.$$

Appendix D

Discussion on the choice of Ω_N in Sec. 4.1

Our formulation (P′) examines the set Ω_N = {y₀, ⋯, y_N−1, x_N} and eventually leads to the best known analytical bound on the norm of the composite gradient mapping in Thm. 11 among fixed-step first-order methods.

An alternative formulation would be to use the set {y₀, ⋯, y_N−1} (i.e., excluding the point x_N). For this alternative, we could simply replace the inequality (33) with the condition 0 ≤ F(y_N−1) − F(x_*) to derive a slightly different relaxation. (One could use other conditions at the point y_N−1 as in [29] for a tight relaxation, but this is beyond the scope of this paper.) We found that the corresponding (loose) relaxation (P1′) using {y₀, ⋯, y_N−1} leads to a larger upper bound than (40) in Thm. 10 for the set Ω_N.

Another alternative would be to use the set {x₀, ⋯, x_N}, which we leave as future work. Nevertheless, the inequality in Lemma 1 provides a bound for that set {x₀, ⋯, x_N} as seen in Thm. 10 and 11.

We could also consider the final point x_N (or y_N) in (P′) instead of the minimum over a set of points. However, the corresponding (loose) relaxation (P1′) yielded only an O(1/N) bound at best (even for the corresponding optimized step coefficients of (HD′)) on the final composite gradient mapping norm. So we leave finding its tighter relaxation as future work. Note that Table 4 reports tight numerical bounds on the composite gradient mapping norm at the final point x_N of algorithms considered.

Appendix E

Proof of Equation (45) in Thm. 11

$$\sum _{k=0}^{N-1}(T_k-t_k^2)+T_{N-1}=\sum _{k=m}^{N-1}(t_{m-1}^2+\sum _{l=m}^k{t}_l-t_k^2)+t_{m-1}^2+\sum _{l=m}^{N-1}{t}_l=(N-m+1)t_{m-1}^2+\sum _{k=m}^{N-1}(\sum _{l=m}^k\frac{N-l+1}{2}-{\left(\frac{N-k+1}{2}\right)}^2)+\sum _{l=m}^{N-1}\frac{N-l+1}{2}=(N-m+1)t_{m-1}^2+\sum _{k\prime =0}^{N-m-1}(\sum _{l\prime =0}^{k\prime }\frac{N-l\prime -m+1}{2}-{\left(\frac{N-k\prime -m+1}{2}\right)}^2)+\sum _{l\prime =0}^{N-m-1}\frac{N-l\prime -m+1}{2}=(N-m+1)t_{m-1}^2+\sum _{k=0}^{N-m-1}(\frac{(N-m+1)(k+1)}{2}-\frac{k(k+1)}{4}-\frac{{(N-m+1)}^2-2(N-m+1)k+k^2}{4})+\frac{(N-m+1)(N-m)}{2}-\frac{(N-m-1)(N-m)}{4}=(N-m+1)t_{m-1}^2+\sum _{k=0}^{N-m-1}(-\frac{k^2}{2}+(N-m+3/4)k-\frac{(N-m-1)(N-m+1)}{4})+\frac{(N-m)(N-m+3)}{4}=(N-m+1)t_{m-1}^2-\frac{(N-m-1)(N-m-1/2)(N-m)}{6}+\frac{(N-m-1)(N-m)(N-m+3/4)}{2}-\frac{(N-m-1)(N-m)(N-m+1)}{4}+\frac{(N-m)(N-m+3)}{4}\ge \frac{(N-m+1){(m+1)}^2}{4}+\frac{(N-m-1){(N-m)}^2}{3}-\frac{{(N-m)}^2(N-m+1)}{4}\ge \frac{(N-m-1){(N-m)}^2}{3}\ge \frac{1}{24}(N-2)N^2,$$

where $m=\lfloor \frac{N}{2}\rfloor \ge \frac{N-1}{2},N-m\ge \frac{N}{2}$, and $t_{m-1}\ge \frac{m+1}{2}$ (13).