Minimizing Uniformly Convex Functions by Cubic Regularization of Newton Method

Abstract

In this paper, we study the iteration complexity of cubic regularization of Newton method for solving composite minimization problems with uniformly convex objective. We introduce the notion of second-order condition number of a certain degree and justify the linear rate of convergence in a nondegenerate case for the method with an adaptive estimate of the regularization parameter. The algorithm automatically achieves the best possible global complexity bound among different problem classes of uniformly convex objective functions with Hölder continuous Hessian of the smooth part of the objective. As a byproduct of our developments, we justify an intuitively plausible result that the global iteration complexity of the Newton method is always better than that of the gradient method on the class of strongly convex functions with uniformly bounded second derivative.

Introduction

A big step in a second-order optimization theory is related to the global complexity guarantees which were justified in [17] for the cubic regularization of the Newton method. The following results provide a good perspective for the development of this approach, discovering accelerated [14], adaptive [4, 5] and universal [10] schemes. The latter methods can automatically adjust to a smoothness properties of the particular objective function. In the same vein, the second-order algorithms for solving a system of nonlinear equations were discovered in [13], and randomized variants for solving large-scale optimization problems were proposed in  [7–9, 12, 18].

Despite to a number of nice properties, global complexity bounds of the cubically regularized Newton method for the cases of strongly convex and uniformly convex objective are not still fully investigated, as well as the notion of second-order non-degeneracy (see discussion in Sect. 5 in [14]). We are going to address this issue in the current paper.

The rest of the paper is organized as follows. Section 2 contains all necessary definitions and main properties of the classes of uniformly convex functions and twice-differentiable functions with Hölder continuous Hessian. We introduce the notion of the condition number ${\gamma }_f(\nu )$ of a certain degree $\nu \in [0,1]$ and present some basic examples.

In Sect. 3, we describe a general regularized Newton scheme and show the linear rate of convergence for this method on the class of uniformly convex functions with a known degree $\nu \in [0,1]$ of nondegeneracy. Then, we introduce the adaptive cubically regularized Newton method and collect useful inequalities and properties, which are related to this algorithm.

In Sect. 4, we study global iteration complexity of the cubically regularized Newton method on the classes of uniformly convex functions with Hölder continuous Hessian. We show that for nondegeneracy of any degree $\nu \in [0,1]$, which is formalized by the condition ${\gamma }_f(\nu )>0$, the algorithm automatically achieves the linear rate of convergence with the value ${\gamma }_f(\nu )$ being the main complexity factor.

Finally, in Sect. 5 we compare our complexity bounds with the known bounds for other methods and discuss the results. In particular, we justify an intuitively plausible (but quite a delayed) result that the global complexity of the cubically regularized Newton method is always better than that of the gradient method on the class of strongly convex functions with uniformly bounded second derivative.

Uniformly Convex Functions with Hölder Continuous Hessian

Let us start from some notation. In what follows, we denote by $\mathbb{E}$ a finite-dimensional real vector space and by ${\mathbb{E}}^{\ast }$ its dual space, which is a space of linear functions on $\mathbb{E}$. The value of function $s\in {\mathbb{E}}^{\ast }$ at point $x\in \mathbb{E}$ is denoted by $⟨s,x⟩$. Let us fix some linear self-adjoint positive-definite operator $B:\mathbb{E}\to {\mathbb{E}}^{\ast }$ and introduce the following Euclidean norms in the primal and dual spaces:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill \Vert x\Vert :=& {⟨Bx,x⟩}^{1/2},x\in \mathbb{E},{\Vert s\Vert }_{\ast }:={⟨s,B^{-1}s⟩}^{1/2},s\in {\mathbb{E}}^{\ast }.\end{array}\end{array}$$

For any linear operator $A:\mathbb{E}\to {\mathbb{E}}^{\ast }$, its norm is induced in a standard way:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill \Vert A\Vert :=& \underset{x\in \mathbb{E}}{max}{\{\Vert Ax\Vert }_{\ast }|\Vert x\Vert \le 1\}.\end{array}\end{array}$$

Our goal is to solve the convex optimization problem in the composite form:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill \underset{x\in \text{dom}F}{min}F(x):=& f(x)+h(x),\end{array}\end{array}$$ 1

where f is a twice differentiable on its open domain uniformly convex function, and h is a simple closed convex function with $\text{dom}h\subseteq \text{dom}f$. Simple means that all auxiliary subproblems with an explicit presence of h are easily solvable.

For a smooth function f, its gradient at point x is denoted by $\mathrm{\nabla }f(x)\in {\mathbb{E}}^{\ast }$, and its Hessian is denoted by ${\mathrm{\nabla }}^{2}f(x):\mathbb{E}\to {\mathbb{E}}^{\ast }$. For convex but not necessary differentiable function h, we denote by $\partial h(x)\subset {\mathbb{E}}^{\ast }$ its subdifferential at the point $x\in \text{dom}h$.

We say that differentiable function f is uniformly convex of degree $p\ge 2$ on a convex set $C\subseteq \text{dom}f$ if for some constant $\sigma >0$ it satisfies inequality

$$\begin{array}{c}\hfill \begin{array}{ccc}\hfill f(y)& \ge & f(x)+⟨\mathrm{\nabla }f(x),y-x⟩+\frac{\sigma }{p}{\Vert y-x\Vert }^p,x,y\in C.\hfill \end{array}\end{array}$$ 2

Uniformly convex functions of degree $p=2$ are known as strongly convex. If inequality (2) holds with $\sigma =0$, the function f is called just convex. The following convenient condition is sufficient for function f to be uniformly convex on a convex set $C\subseteq \text{dom}f$:

Lemma 2.1

Lemma 1 in [14]) Let for some $\sigma >0$ and $p\ge 2$ the following inequality holds:

$$\begin{array}{c}\hfill \begin{array}{ccc}\hfill ⟨\mathrm{\nabla }f(x)-\mathrm{\nabla }f(y),x-y⟩& \ge & {\sigma \Vert x-y\Vert }^p,x,y\in C.\hfill \end{array}\end{array}$$ 3

Then, function f is uniformly convex of degree p on set C with parameter $\sigma$.

From now on, we assume $C:=\text{dom}F\subseteq \text{dom}f.$ By the composite representation (1), we have for every $x\in \text{dom}F$ and for all $F^{\prime }(x)\in \partial F(x)$:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill F(y)\ge & F(x)+⟨F^{\prime }(x),y-x⟩+\frac{\sigma }{p}{\Vert x-y\Vert }^p,y\in \text{dom}F.\end{array}\end{array}$$ 4

Therefore, if $\sigma >0$, then we can have only one point $x^{\ast }\in \text{dom}F$ with $F(x^{\ast })=F^{\ast }$, which always exists for F being uniformly convex and closed. A useful consequence of uniform convexity is the following upper bound for the residual.

Lemma 2.2

Let f be uniformly convex of degree $p\ge 2$ with constant $\sigma >0$ on set $\text{dom}F$. Then, for every $x\in \text{dom}F$ and for all $F^{\prime }(x)\in \partial F(x)$ we have

$$\begin{array}{c}\hfill \begin{array}{ccc}\hfill F(x)-F^{\ast }& \le & \frac{p-1}{p}{\left(\frac{1}{\sigma }\right)}^{\frac{1}{p-1}}{\Vert F^{\prime }(x)\Vert }_{\ast }^{\frac{p}{p-1}}.\hfill \end{array}\end{array}$$ 5

Proof

In view of (4), bound (5) follows as in the proof of Lemma 3 in [14]. $\square$

It is reasonable to define the best possible constant $\sigma$ in inequality (3) for a certain degree p. This leads us to a system of constants:

$$\begin{array}{c}\hfill \begin{array}{ccc}\hfill {\sigma }_f(p)& :=& \underset{\begin{array}{c}x,y\in \text{dom}F\\ x\ne y\end{array}}{inf}\frac{⟨\mathrm{\nabla }f(x)-\mathrm{\nabla }f(y),x-y⟩}{{\Vert x-y\Vert }^p},p\ge 2.\hfill \end{array}\end{array}$$ 6

We prefer to use inequality (3) for the definition of ${\sigma }_f(p)$, instead of (2), because of its symmetry in x and y. Note that the value ${\sigma }_f(p)$ also depends on the domain of F. However, we omit this dependence in our notation since it is always clear from the context.

It is easy to see that the univariate function ${\sigma }_f(·)$ is log-concave. Thus, for all $p_2>p_1\ge 2$ we have:

$$\begin{array}{c}\hfill \begin{array}{ccc}\hfill {\sigma }_f(p)& \ge & ({\sigma }_f(p_1){)}^{\frac{p_2-p}{p_2-p_1}}·({\sigma }_f(p_2){)}^{\frac{p-p_1}{p_2-p_1}},p\in [p_1,p_2].\hfill \end{array}\end{array}$$ 7

For a twice-differentiable function f, we say that it has Hölder continuous Hessian of degree $\nu \in [0,1]$ on a convex set $C\subseteq \text{dom}f$, if for some constant $\mathcal{H}$, it holds:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill \Vert {\mathrm{\nabla }}^{2}f(x)-{\mathrm{\nabla }}^{2}f(y)\Vert \le & {\mathcal{H}\Vert x-y\Vert }^{\nu },x,y\in C.\end{array}\end{array}$$ 8

Two simple consequences of (8) are as follows:

$$\begin{array}{cc}& \Vert \mathrm{\nabla }f(y)-\mathrm{\nabla }f(x)-{\mathrm{\nabla }}^2{f(x)(y-x)\Vert }_{\ast }\le \frac{{\mathcal{H}\Vert x-y\Vert }^{1+\nu }}{1+\nu },\hfill \end{array}$$ 9

$$\begin{array}{cc}& |f(y)-Q(x;y)|\le \frac{{\mathcal{H}\Vert x-y\Vert }^{2+\nu }}{(1+\nu )(2+\nu )},\hfill \end{array}$$ 10

where Q(x; y) is the quadratic model of f at the point x:

$$\begin{array}{c}\hfill \begin{array}{ccc}\hfill Q(x;y)& :=& f(x)+⟨\mathrm{\nabla }f(x),y-x⟩+\frac{1}{2}⟨{\mathrm{\nabla }}^{2}f(x)(y-x),y-x⟩.\hfill \end{array}\end{array}$$

In order to characterize the level of smoothness of function f on the set $C:=\text{dom}F$, let us define the system of Hölder constants (see [10]):

$$\begin{array}{c}\hfill \begin{array}{ccc}\hfill {\mathcal{H}}_f(\nu )& :=& \underset{\begin{array}{c}x,y\in \text{dom}F\\ x\ne y\end{array}}{sup}\frac{\Vert {\mathrm{\nabla }}^{2}f(x)-{\mathrm{\nabla }}^{2}f(y)\Vert }{{\Vert x-y\Vert }^{\nu }},\nu \in [0,1].\hfill \end{array}\end{array}$$ 11

We allow ${\mathcal{H}}_f(\nu )$ to be equal to $+\infty$ for some $\nu$. Note that function ${\mathcal{H}}_f(·)$ is log-convex. Thus, any $0\le {\nu }_1<{\nu }_2\le 1$ such that ${\mathcal{H}}_f({\nu }_i)<+\infty ,i=1,2$, provide us with the following upper bounds for the whole interval:

$$\begin{array}{c}\hfill \begin{array}{ccc}\hfill {\mathcal{H}}_f(\nu )& \le & ({\mathcal{H}}_f({\nu }_1){)}^{\frac{{\nu }_2-\nu }{{\nu }_2-{\nu }_1}}·({\mathcal{H}}_f({\nu }_2){)}^{\frac{\nu -{\nu }_1}{{\nu }_2-{\nu }_1}},\nu \in [{\nu }_1,{\nu }_2].\hfill \end{array}\end{array}$$ 12

If for some specific $\nu \in [0,1]$ we have ${\mathcal{H}}_f(\nu )=0$, this implies that ${\mathrm{\nabla }}^{2}f(x)={\mathrm{\nabla }}^{2}f(y)$ for all $x,y\in \text{dom}F$. In this case restriction, ${\left(f\right|}_{\text{dom}F}$ is a quadratic function and we conclude that ${\mathcal{H}}_f(\nu )=0$ for all $\nu \in [0,1]$. At the same time, having two points $x,y\in \text{dom}F$ with $0<\Vert x-y\Vert \le 1$, we get a simple uniform lower bound for all constants ${\mathcal{H}}_f(\nu )$:

$$\begin{array}{c}\hfill \begin{array}{ccc}\hfill {\mathcal{H}}_f(\nu )& \ge & \Vert {\mathrm{\nabla }}^{2}f(x)-{\mathrm{\nabla }}^{2}f(y)\Vert ,\nu \in [0,1].\hfill \end{array}\end{array}$$

Let us give an example of function, which has Hölder continuous Hessian for all $\nu \in [0,1]$.

Example 2.1

For a given $a_i\in {\mathbb{E}}^{\ast }$, $1\le i\le m$, consider the following convex function:

$$\begin{array}{c}\hfill \begin{array}{ccc}\hfill f(x)& =& ln\left(\sum _{i=1}^m,e^{⟨a_i,x⟩}\right),x\in \mathbb{E}.\hfill \end{array}\end{array}$$

Let us fix Euclidean norm $\Vert x\Vert ={⟨Bx,x⟩}^{1/2},x\in \mathbb{E}$, with operator $B:={\sum }_{i=1}^m{a}_i{a}_i^{\ast }$. Without loss of generality, we assume that $B\succ 0$ (otherwise we can reduce dimension of the problem). Then,

$$\begin{array}{c}\hfill \begin{array}{ccc}\hfill {\mathcal{H}}_f(0)& \le & 1,{\mathcal{H}}_f(1)\le 2.\hfill \end{array}\end{array}$$

Therefore, by (12) we get, for any $\nu \in [0,1]$:

$$\begin{array}{c}\hfill \begin{array}{ccc}\hfill {\mathcal{H}}_f(\nu )& \le & 2^{\nu }.\hfill \end{array}\end{array}$$

Proof

Denote $\kappa (x)\equiv {\sum }_{i=1}^m{e}^{⟨a_i,x⟩}$. Let us fix arbitrary $x,y\in \mathbb{E}$ and direction $h\in \mathbb{E}$. Then, straightforward computation gives:

$$\begin{array}{cc}\hfill ⟨\mathrm{\nabla }f(x),h⟩=& \frac{1}{\kappa (x)}\sum _{i=1}^m{e}^{⟨a_i,x⟩}⟨a_i,h⟩,\hfill \\ \hfill ⟨{\mathrm{\nabla }}^{2}f(x)h,h⟩=& \frac{1}{\kappa (x)}\sum _{i=1}^m{e}^{⟨a_i,x⟩}{⟨a_i,h⟩}^2-(\frac{1}{\kappa (x)}\sum _{i=1}^m{e}^{⟨a_i,x⟩}⟨a_i,h⟩{)}^2\hfill \\ \hfill =& \frac{1}{\kappa (x)}\sum _{i=1}^m{e}^{⟨a_i,x⟩}{\left(⟨a_i,h⟩,-,⟨\mathrm{\nabla }f(x),h⟩\right)}^2\ge 0.\hfill \end{array}$$

Hence, we get

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill \Vert {\mathrm{\nabla }}^{2}f(x)\Vert =& \underset{\Vert h\Vert \le 1}{max}⟨{\mathrm{\nabla }}^{2}f(x)h,h⟩\le \underset{\Vert h\Vert \le 1}{max}{\sum }_{i=1}^m{⟨a_i,h⟩}^2=\underset{\Vert h\Vert \le 1}{max}{\Vert h\Vert }^2=1.\end{array}\end{array}$$

Since all Hessians of function f are positive definite, we conclude that ${\mathcal{H}}_f(0)\le 1$. Inequality ${\mathcal{H}}_f(1)\le 2$ can be easily obtained from the following representation of the third derivative:

$$\begin{array}{cc}\hfill f^{\prime \prime \prime }(x)[h,h,h]=& \frac{1}{\kappa (x)}\sum _{i=1}^m{e}^{⟨a_i,x⟩}{\left(⟨a_i,h⟩,-,⟨\mathrm{\nabla }f(x),h⟩\right)}^3\hfill \\ \hfill \le & ⟨{\mathrm{\nabla }}^{2}f(x)h,h⟩\underset{1\le i,j\le m}{max}⟨a_i-a_j,h⟩\le 2{\Vert h\Vert }^3.\hfill \end{array}$$

$\square$

Let us imagine now that we want to describe the iteration complexity of some method, which solves the composite optimization problem (1) up to an absolute accuracy $ϵ>0$ in the function value. We assume that the smooth part f of its objective is uniformly convex and has Hölder continuous Hessians. Which degrees p and $\nu$ should be used in our analysis? Suppose that, for the number of calls of the oracle, we are interested in obtaining a polynomial-time bound of the form:

$$\begin{array}{c}\hfill \begin{array}{c}O\left({({\mathcal{H}}_f(\nu ))}^{\alpha },·,{({\sigma }_f(p))}^{\beta },·,log,\frac{F(x_0)-F^{\ast }}{\epsilon }\right),\alpha ,\beta \ne 0.\end{array}\end{array}$$

Denote by $[x]$ the physical dimension of variable $x\in \mathbb{E}$, and by $[f]$ the physical dimension of the value f(x). Then, we have $[\mathrm{\nabla }f(x)]=[f]/[x]$ and $[{\mathrm{\nabla }}^{2}f(x)]=[f]/{[x]}^2$. This gives us

$$\begin{array}{c}\hfill \begin{array}{c}[{\mathcal{H}}_f(\nu )]=\frac{[f]}{{[x]}^{2+\nu }},[{\sigma }_f(p)]=\frac{[f]}{{[x]}^p},[{({\mathcal{H}}_f(\nu ))}^{\alpha }·{({\sigma }_f(p))}^{\beta }]=\frac{{[f]}^{\alpha +\beta }}{{[x]}^{\alpha (2+\nu )+\beta p}}.\end{array}\end{array}$$

While x and f(x) can be measured in arbitrary physical quantities, the value “number of iterations” cannot have physical dimension. This leads to the following relations:

$$\begin{array}{c}\hfill \alpha +\beta =0\text{and}\alpha (2+\nu )+\beta p=0.\end{array}$$

Therefore, despite to the fact that our function can belong to several problem classes simultaneously, from the physical point of view only one option is available:

$$\begin{array}{c}\hfill \boxed{p=2+\nu }\end{array}$$

Hence, for a twice-differentiable convex function f with ${inf}_{\nu \in [0,1]}{\mathcal{H}}_f(\nu )>0$, we can define only one meaningful condition number of degree $\nu \in [0,1]$:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill {\gamma }_f(\nu ):=\frac{{\sigma }_f(2+\nu )}{{\mathcal{H}}_f(\nu )}.\end{array}\end{array}$$ 13

If for some particular $\nu$ we have ${\mathcal{H}}_f(\nu )=+\infty$, then by our definition: ${\gamma }_f(\nu )=0$.

It will be shown that the condition number ${\gamma }_f(\nu )$ serves as a main factor in the global iteration complexity bounds for the regularized Newton method as applied to the problem (1). Let us prove that this number cannot be big.

Lemma 2.3

Let ${inf}_{\nu \in [0,1]}{\mathcal{H}}_f(\nu )>0$ and therefore the condition number ${\gamma }_f(·)$ be well defined. Then,

$$\begin{array}{c}\hfill \begin{array}{ccc}\hfill {\gamma }_f(\nu )& \le & \frac{1}{1+\nu }+\underset{x,y\in \text{dom}F}{inf}\frac{\Vert {\mathrm{\nabla }}^{2}f(x)\Vert }{\Vert {\mathrm{\nabla }}^{2}f(y)-{\mathrm{\nabla }}^{2}f(x)\Vert },\nu \in [0,1].\hfill \end{array}\end{array}$$ 14

In the case when $\text{dom}F$ is unbounded: ${sup}_{x\in \text{dom}F}\Vert x\Vert =+\infty$, then

$$\begin{array}{c}\hfill \begin{array}{ccc}\hfill {\gamma }_f(\nu )& \le & \frac{1}{1+\nu },\nu \in (0,1].\hfill \end{array}\end{array}$$ 15

Proof

Indeed, for any $x,y\in \text{dom}F$, $x\ne y$, we have:

$$\begin{array}{cc}& {\sigma }_f(2+\nu )\stackrel{(6)}{\le }\frac{⟨\mathrm{\nabla }f(y)-\mathrm{\nabla }f(x),y-x⟩}{{\Vert y-x\Vert }^{2+\nu }}\hfill \\ \hfill & =\frac{⟨\mathrm{\nabla }f(y)-\mathrm{\nabla }f(x)-{\mathrm{\nabla }}^{2}f(x)(y-x),y-x⟩}{{\Vert y-x\Vert }^{2+\nu }}+\frac{⟨{\mathrm{\nabla }}^{2}f(x)(y-x),y-x⟩}{{\Vert y-x\Vert }^{2+\nu }}\hfill \\ \hfill & \stackrel{(9)}{\le }\frac{{\mathcal{H}}_f(\nu )}{1+\nu }+\frac{\Vert {\mathrm{\nabla }}^{2}f(x)\Vert }{{\Vert y-x\Vert }^{\nu }}.\hfill \end{array}$$

Now, dividing both sides of this inequality by ${\mathcal{H}}_f(\nu )$, we get inequality (14) from the definition of ${\mathcal{H}}_f(\nu )$ (11). Inequality (15) can be obtained by taking the limit $\Vert y\Vert \to +\infty$. $\square$

From inequalities (7) and (12), we can get the following lower bound:

$$\begin{array}{c}\hfill \begin{array}{ccc}\hfill {\gamma }_f(\nu )& \ge & ({\gamma }_f({\nu }_1){)}^{\frac{{\nu }_2-\nu }{{\nu }_2-{\nu }_1}}·({\gamma }_f({\nu }_2){)}^{\frac{\nu -{\nu }_1}{{\nu }_2-{\nu }_1}},\nu \in [{\nu }_1,{\nu }_2],\hfill \end{array}\end{array}$$

where $0\le {\nu }_1<{\nu }_2\le 1$. However, it turns out that in unbounded case we can have a nonzero condition number ${\gamma }_f(\nu )$ only for a single degree.

Lemma 2.4

  Let $\text{dom}F$ be unbounded: ${sup}_{x\in \text{dom}F}\Vert x\Vert =+\infty$. Assume that for a fixed $\nu \in [0,1]$ we have ${\gamma }_f(\nu )>0$. Then,

$$\begin{array}{c}\hfill {\gamma }_f(\alpha )=0\text{for all}\alpha \in [0,1]\backslash \{\nu \}.\end{array}$$

Proof

Consider firstly the case: $\alpha >\nu$. From the condition ${\gamma }_f(\nu )>0$, we conclude that ${\mathcal{H}}_f(\nu )<+\infty$. Then, for any $x,y\in \text{dom}F$ we have:

$$\begin{array}{cc}& \frac{{\sigma }_f(2+\alpha ){\Vert y-x\Vert }^{2+\alpha }}{2+\alpha }\stackrel{(2)}{\le }f(y)-f(x)-⟨\mathrm{\nabla }f(x),y-x⟩\hfill \\ \hfill & \stackrel{(10)}{\le }\frac{1}{2}⟨{\mathrm{\nabla }}^{2}f(x)(y-x),(y-x)⟩+\frac{{\mathcal{H}}_f(\nu ){\Vert y-x\Vert }^{2+\nu }}{(1+\nu )(2+\nu )}.\hfill \end{array}$$

Dividing both sides of this inequality by ${\Vert y-x\Vert }^{2+\alpha }$ and letting $\Vert x\Vert \to +\infty$, we get ${\sigma }_f(2+\nu )=0$. Therefore, ${\gamma }_f(\alpha )=0$. For the second case, $\alpha <\nu$, we cannot have ${\gamma }_f(\alpha )>0$, since the previous reasoning results in ${\gamma }_f(\nu )=0$. $\square$

Let us look now at an important example of a uniformly convex function with Hölder continuous Hessian. It is convenient to start with some properties of powers of Euclidean norm.

Lemma 2.5

For fixed real $p\ge 1$, consider the following function:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill f_p(x)=\frac{1}{p}{\Vert x\Vert }^p,x\in \mathbb{E}.\end{array}\end{array}$$

1. For $p\ge 2$, function $f_p(·)$ is uniformly convex of degree p:1${}^{)}$

$$\begin{array}{c}\hfill ⟨\mathrm{\nabla }{f}_p(x)-\mathrm{\nabla }{f}_p(y),x-y⟩\ge 2^{2-p}{\Vert x-y\Vert }^p,x,y\in \mathbb{E}.\end{array}$$ 16

2. If $1\le p\le 2$, then function $f_p(·)$ has $\nu$-Hölder continuous gradient with $\nu =p-1$:

$$\begin{array}{c}\hfill \Vert \mathrm{\nabla }{f}_p(x)-\mathrm{\nabla }{f}_p{(y)\Vert }_{\ast }\le 2^{1-\nu }{\Vert x-y\Vert }^{\nu },x,y\in \mathbb{E}.\end{array}$$ 17

Proof

Firstly, recall two useful inequalities, which are valid for all $a,b\ge 0$:

$$\begin{array}{cc}& |a^{\alpha }-b^{\alpha }{|\le |a-b|}^{\alpha },\text{when}0\le \alpha \le 1,\hfill \end{array}$$ 18

$$\begin{array}{cc}& |a^{\alpha }-b^{\alpha }{|\ge |a-b|}^{\alpha },\text{when}\alpha \ge 1.\hfill \end{array}$$ 19

Let us fix arbitrary $x,y\in \mathbb{E}$. The left-hand side of inequality (16) equals

$$\begin{array}{c}\hfill {⟨\Vert x\Vert }^{p-2}Bx-{\Vert y\Vert }^{p-2}{By,x-y⟩=\Vert x\Vert }^p+{\Vert y\Vert }^p-{⟨Bx,y⟩(\Vert x\Vert }^{p-2}+{\Vert y\Vert }^{p-2}),\end{array}$$

and we need to verify that it is bigger than $2^{2-p}{[\Vert x\Vert }^2+{\Vert y\Vert }^2-2⟨Bx,y⟩{]}^{\frac{p}{2}}.$ The case $x=0$ or $y=0$ is trivial. Therefore, assume $x\ne 0$ and $y\ne 0$. Denoting $\tau :=\frac{\Vert y\Vert }{\Vert x\Vert }$, $r:=\frac{⟨Bx,y⟩}{\Vert x\Vert ·\Vert y\Vert }$, we have the following statement to prove:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill 1+{\tau }^p\ge & r\tau (1+{\tau }^{p-2})+2^{2-p}[1+{\tau }^2-2r\tau {]}^{\frac{p}{2}},\tau >0,|r|\le 1.\end{array}\end{array}$$

Since the function in the right-hand side is convex in r, we need to check only two marginal cases:

1. $r=1:$ $1+{\tau }^p\ge \tau (1+{\tau }^{p-2})+2^{2-p}{|1-\tau |}^p$, which is equivalent to $(1-\tau )(1-{\tau }^{p-1})\ge 2^{2-p}{|1-\tau |}^p$. This is true by (19).

2. $r=-1:$ $1+{\tau }^p\ge -\tau (1+{\tau }^{p-2})+2^{2-p}{(1+\tau )}^p$, which is equivalent to $(1+{\tau }^{p-1})\ge 2^{2-p}{(1+\tau )}^{p-1}$. This is true in view of convexity of function ${\tau }^{p-1}$ for $\tau \ge 0$.

Thus, we have proved (16). Let us prove the second statement. Consider the function ${\widehat{f}}_q(s)=\frac{1}{q}{\Vert s\Vert }_{\ast }^q$, $s\in {\mathbb{E}}^{\ast }$, with $q=\frac{p}{p-1}\ge 2$. In view of our first statement, we have:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill ⟨s_1-s_2,\mathrm{\nabla }{\widehat{f}}_q(s_1)-\mathrm{\nabla }{\widehat{f}}_q(s_2)⟩\ge & {\left(\frac{1}{2}\right)}^{q-2}{\Vert s_1-s_2\Vert }_{\ast }^q,s_1,s_2\in {\mathbb{E}}^{\ast }.\end{array}\end{array}$$ 20

For arbitrary $x_1,x_2\in \mathbb{E}$, define $s_i=\mathrm{\nabla }{f}_p(x_i)=\frac{B{x}_i}{\Vert x_i{\Vert }^{2-p}}$, $i=1,2$. Then $\Vert s_i{\Vert }_{\ast }={\Vert x_i\Vert }^{p-1}$, and consequently,

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill x_i=& \Vert x_i{\Vert }^{2-p}{B}^{-1}{s}_i={\Vert s_i\Vert }_{\ast }^{\frac{2-p}{p-1}}{B}^{-1}{s}_i=\mathrm{\nabla }{\widehat{f}}_q(s_i).\end{array}\end{array}$$

Therefore, substituting these vectors in (20), we get

$$\begin{array}{c}\hfill {\left(\frac{1}{2}\right)}^{q-2}{\Vert \mathrm{\nabla }{f}_p(x_1)-\mathrm{\nabla }{f}_p(x_2)\Vert }_{\ast }^q\le ⟨\mathrm{\nabla }{f}_p(x_1)-\mathrm{\nabla }{f}_p(x_2),x_1-x_2⟩.\end{array}$$

Thus, $\Vert \mathrm{\nabla }{f}_p(x_1)-\mathrm{\nabla }{f}_p(x_2){\Vert }_{\ast }\le 2^{\frac{q-2}{q-1}}{\Vert x_1-x_2\Vert }^{\frac{1}{q-1}}$. It remains to note that $\frac{1}{q-1}=p-1=\nu$. $\square$

Example 2.2

For real $p\ge 2$ and arbitrary $x_0\in \mathbb{E}$, consider the following function:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill f(x)=\frac{1}{p}{\Vert x-x_0\Vert }^p=f_p(x-x_0),x\in \mathbb{E}.\end{array}\end{array}$$

Then, ${\sigma }_f(p)={\left(\frac{1}{2}\right)}^{p-2}$. Moreover, if $p=2+\nu$ for some $\nu \in (0,1]$, then it holds

$$\begin{array}{c}\hfill \begin{array}{c}\hfill {\mathcal{H}}_f(\nu )\le (1+\nu )2^{1-\nu },\end{array}\end{array}$$

and ${\mathcal{H}}_f(\alpha )=+\infty$, for all $\alpha \in [0,1]\backslash \{\nu \}$. Therefore, in this case we have ${\gamma }_f(\nu )\ge \frac{1}{2(1+\nu )},$ and ${\gamma }_f(\alpha )=0$ for all $\alpha \in [0,1]\backslash \{\nu \}$.

Proof

Let us take an arbitrary $x\ne 0$ and set $y:=-x$. Then,

$$\begin{array}{c}\hfill ⟨\mathrm{\nabla }f(x)-\mathrm{\nabla }f(y),y-x⟩={⟨\Vert x\Vert }^{p-2}Bx+{\Vert x\Vert }^{p-2}{Bx,2x⟩=4\Vert x\Vert }^p.\end{array}$$

On the other hand, ${\Vert y-x\Vert }^p=2^p{\Vert x\Vert }^p$. Therefore, ${\sigma }_f(p)\stackrel{(6)}{\le }{2}^{2-p}$, and (16) tells us that this inequality is satisfied as equality.

Let us prove now that ${\mathcal{H}}_f(\nu )\le (1+\nu )2^{1-\nu }$ for $p=2+\nu$ with some $\nu \in (0,1]$. This is

$$\begin{array}{c}\hfill \Vert {\mathrm{\nabla }}^{2}f(x)-{\mathrm{\nabla }}^{2}f(y)\Vert \le (1+\nu )2^{1-\nu }{\Vert x-y\Vert }^{\nu },x,y\in \mathbb{E}.\end{array}$$ 21

The corresponding Hessians can be represented as follows:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill {\mathrm{\nabla }}^{2}f(x)=& {\Vert x\Vert }^{\nu }B+\frac{\nu Bx{x}^{\ast }B}{{\Vert x\Vert }^{2-\nu }},x\in \mathbb{E}\backslash \{0\},{\mathrm{\nabla }}^{2}f(0)=0.\end{array}\end{array}$$

For the case $x=y=0$, inequality (21) is trivial. Assume now that $x\ne 0$. If $0\in [x,y]$, then $y=-\beta x$ for some $\beta \ge 0$ and we have:

$$\begin{array}{cc}\hfill \Vert {\mathrm{\nabla }}^{2}f(x)-{\mathrm{\nabla }}^{2}f(-\beta x)\Vert \le & |1-{\beta }^{\nu }{|(1+\nu )\Vert x\Vert }^{\nu }\le {(1+\beta )}^{\nu }(1+\nu )2^{1-\nu }{\Vert x\Vert }^{\nu }\hfill \\ \hfill =& (1+\nu )2^{1-\nu }{\Vert x-y\Vert }^{\nu },\hfill \end{array}$$

which is (21). Let $0\notin [x,y]$. For an arbitrary fixed direction $h\in \mathbb{E}$, we get:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill |\langle ({\mathrm{\nabla }}^{2}f(x)-{\mathrm{\nabla }}^{2}f(y))h,h\rangle |=& |\left({\Vert x\Vert }^{\nu },-,{\Vert y\Vert }^{\nu }\right)·{\Vert h\Vert }^2+\nu ·\left(\frac{{⟨Bx,h⟩}^2}{{\Vert x\Vert }^{2-\nu }},-,\frac{{⟨By,h⟩}^2}{{\Vert y\Vert }^{2-\nu }}\right)|.\end{array}\end{array}$$

Consider the points $u=\frac{\mathit{Bx}}{{\Vert x\Vert }^{1-\nu }}=\mathrm{\nabla }{f}_q(x)$ and $v=\frac{\mathit{By}}{{\Vert y\Vert }^{1-\nu }}=\mathrm{\nabla }{f}_q(y)$ with $q=1+\nu$. Then,

$$\begin{array}{c}\hfill \begin{array}{c}{\Vert x\Vert }^{\nu }={\Vert u\Vert }_{\ast },\frac{{⟨Bx,h⟩}^2}{{\Vert x\Vert }^{2-\nu }}=\frac{{⟨u,h⟩}^2}{{\Vert u\Vert }_{\ast }}{\text{and}\Vert y\Vert }^{\nu }={\Vert v\Vert }_{\ast },\frac{{⟨By,h⟩}^2}{{\Vert y\Vert }^{2-\nu }}=\frac{{⟨v,h⟩}^2}{{\Vert v\Vert }_{\ast }}.\end{array}\end{array}$$

Therefore,

$$\begin{array}{cc}& |\langle ({\mathrm{\nabla }}^{2}f(x)-{\mathrm{\nabla }}^{2}f(y))h,h\rangle |\hfill \\ \hfill & =|\left({\Vert u\Vert }_{\ast },-,{\Vert v\Vert }_{\ast }\right)·{\Vert h\Vert }^2+\nu ·\left(\frac{{⟨u,h⟩}^2}{{\Vert u\Vert }_{\ast }},-,\frac{{⟨v,h⟩}^2}{{\Vert v\Vert }_{\ast }}\right)|.\hfill \end{array}$$ 22

Let us estimate the right-hand side of (22) from above. Consider a continuously differentiable univariate function:

$$\begin{array}{cc}\hfill \varphi (\tau ):=& {\Vert u(\tau )\Vert }_{\ast }·{\Vert h\Vert }^2+\nu ·\frac{{⟨u(\tau ),h⟩}^2}{{\Vert u(\tau )\Vert }_{\ast }},\hfill \\ \hfill u(\tau ):=& u+\tau (v-u),\tau \in [0,1].\hfill \end{array}$$

Note that

$$\begin{array}{cc}\hfill {\varphi }^{\prime }(\tau )=& \frac{⟨u(\tau ),B^{-1}(v-u)⟩}{{\Vert u(\tau )\Vert }_{\ast }}·{\Vert h\Vert }^2+\frac{2\nu ⟨u(\tau ),h⟩⟨v-u,h⟩}{{\Vert u(\tau )\Vert }_{\ast }}\hfill \\ \hfill & -\frac{\nu {⟨u(\tau ),h⟩}^2⟨u(\tau ),B^{-1}(v-u)⟩}{{\Vert u(\tau )\Vert }_{\ast }^3}\hfill \\ \hfill =& \frac{⟨u(\tau ),B^{-1}(v-u)⟩}{{\Vert u(\tau )\Vert }_{\ast }}·\underset{\ge 0}{\underbrace{\left({\Vert h\Vert }^2,-,\frac{\nu {⟨u(\tau ),h⟩}^2}{{\Vert u(\tau )\Vert }_{\ast }^2}\right)}}+\frac{2\nu ⟨u(\tau ),h⟩⟨v-u,h⟩}{{\Vert u(\tau )\Vert }_{\ast }}.\hfill \end{array}$$

Denote $\gamma :=\frac{⟨u(\tau ),h⟩}{{\Vert u(\tau )\Vert }_{\ast }·\Vert h\Vert }\in [-1,1]$. Then,

$$\begin{array}{c}\hfill |{\varphi }^{\prime }(\tau )|\le {\Vert v-u\Vert }_{\ast }·{\Vert h\Vert }^2·(1-\nu {\gamma }^2+2\nu |\gamma |)\le {(1+\nu )·\Vert v-u\Vert }_{\ast }·{\Vert h\Vert }^2.\end{array}$$

Thus, we have:

$$\begin{array}{c}\hfill |\langle ({\mathrm{\nabla }}^{2}f(x)-{\mathrm{\nabla }}^{2}f(y))h,h\rangle |=|\varphi (1)-\varphi (0)|\le {(1+\nu )·\Vert v-u\Vert }_{\ast }·{\Vert h\Vert }^2.\end{array}$$ 23

It remains to use the definition of u and v and apply inequality (17) with $p=q$. Thus, we have proved, that for $p=2+\nu$ the Hessian of f is Hölder continuous of degree $\nu$. At the same time, taking $y=0$, we get $\Vert {\mathrm{\nabla }}^{2}f(x)-{\mathrm{\nabla }}^{2}f(y)\Vert =\Vert {\mathrm{\nabla }}^2{f(x)\Vert =(1+\nu )\Vert x\Vert }^{\nu }$. These values cannot be uniformly bounded in $x\in \mathbb{E}$ by any multiple of ${\Vert x\Vert }^{\alpha }$ with $\alpha \ne \nu$. So, the Hessian of f is not Hölder continuous for any degree different from $2+\nu$. $\square$

Remark 2.1

Inequalities (16) and (17) have the following symmetric consequences:

$$\begin{array}{cc}\hfill p\ge 2\Rightarrow & \Vert \mathrm{\nabla }{f}_p(x)-\mathrm{\nabla }{f}_p{(y)\Vert }_{\ast }\ge 2^{2-p}{\Vert x-y\Vert }^{p-1},\hfill \\ \hfill p\le 2\Rightarrow & \Vert \mathrm{\nabla }{f}_p(x)-\mathrm{\nabla }{f}_p{(y)\Vert }_{\ast }\le 2^{2-p}{\Vert x-y\Vert }^{p-1},\hfill \end{array}$$

which are valid for all $x,y\in \mathbb{E}$.

Regularized Newton Method

Let us start from the case when we know that for a specific $\nu \in [0,1]$ function f has Hölder continuous Hessian: ${\mathcal{H}}_f(\nu )<+\infty$. Then, from (10), we have the global upper bound for the objective function:

$$\begin{array}{c}\hfill \begin{array}{ccc}\hfill F(y)& \le & M_{\nu ,H}(x;y):=Q(x;y)+\frac{{H\Vert x-y\Vert }^{2+\nu }}{(1+\nu )(2+\nu )}+h(y),x,y\in \text{dom}F,\hfill \end{array}\end{array}$$

where $H>0$ is large enough: $H\ge {\mathcal{H}}_f(\nu )$. Thus, it is natural to employ the minimum of a regularized quadratic model:

$$\begin{array}{c}\hfill \begin{array}{c}{T}_{\nu ,H}(x):=\underset{y\in \text{dom}F}{\mathrm{argmin}}{M}_{\nu ,H}(x;y),M_{\nu ,H}^{\ast }(x):=\underset{y\in \text{dom}F}{min}{M}_{\nu ,H}(x;y),\end{array}\end{array}$$

and define the following general iteration process [10]:

$$\begin{array}{c}\hfill \boxed{x_{k+1}:=T_{\nu ,H_k}(x_k),k\ge 0}\end{array}$$ 24

where the value $H_k$ is chosen either to be a constant from the interval $[0,2{\mathcal{H}}_f(\nu )]$ or by some adaptive procedure.

For the class of uniformly convex functions of degree $p=2+\nu$, we can justify the following global convergence result for this process.

Theorem 3.1

Assume that for some $\nu \in [0,1]$ we have $0<{\mathcal{H}}_f(\nu )<+\infty$ and ${\sigma }_f(2+\nu )>0$. Let the coefficients ${\{H_k\}}_{k\ge 0}$ in the process (24) satisfy the following conditions:

$$\begin{array}{c}\hfill 0\le H_k\le \beta {\mathcal{H}}_f(\nu ),F(x_{k+1})\le M_{\nu ,H_k}^{\ast }(x_k),k\ge 0,\end{array}$$ 25

with some constant $\beta \ge 0$. Then, for the sequence ${\{x_k\}}_{k\ge 0}$ generated by the process we have:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill F(x_{k+1})-F^{\ast }\le & (1-\frac{1+\nu }{2+\nu }·min\{\frac{{\gamma }_f(\nu )(1+\nu )}{(1+\beta )(2+\nu )},1{\}}^{\frac{1}{1+\nu }})\left(F,(x_k),-,F^{\ast }\right).\end{array}\end{array}$$ 26

Thus, the rate of convergence is linear and for reaching the gap $F(x_K)-F^{\ast }\le \epsilon$ it is enough to perform $K=\lceil \frac{2+\nu }{1+\nu }·max\{\frac{(1+\beta )(2+\nu )}{{\gamma }_f(\nu )(1+\nu )},1{\}}^{\frac{1}{1+\nu }}log\frac{F(x_0)-F^{\ast }}{\epsilon }\rceil$ iterations.

Proof

As in the proof of Theorem 3.1 in [10], from (25) one can see that

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill F(x_{k+1})\le & F(x_k)-\alpha \left(F,(x_k),-,F^{\ast }\right)+{\alpha }^{2+\nu }\frac{(1+\beta ){\mathcal{H}}_f(\nu ){\Vert x_k-x^{\ast }\Vert }^{2+\nu }}{(1+\nu )(2+\nu )},\end{array}\end{array}$$

for any $\alpha \in [0,1]$. Then, taking into account the uniform convexity (4), we get

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill F(x_{k+1})\le & F(x_k)-\left(\alpha ,-,{\alpha }^{2+\nu },\frac{(1+\beta ){\mathcal{H}}_f(\nu )}{(1+\nu ){\sigma }_f(2+\nu )}\right)\left(F,(x_k),-,F^{\ast }\right).\end{array}\end{array}$$

The minimum of the right-hand side is attained at ${\alpha }^{\ast }=min\{\frac{{\gamma }_f(\nu )(1+\nu )}{(2+\nu )(1+\beta )},1{\}}^{\frac{1}{1+\nu }}$. Plugging this value into the bound above, we get inequality (26). $\square$

Unfortunately, in practice it is difficult to decide on an appropriate value of $\nu \in [0,1]$ with ${\mathcal{H}}_f(\nu )<+\infty$. Therefore, it is interesting to develop the universal methods which are not based on some particular parameters. Recently, it was shown [10] that one good choice for such universal scheme is the cubic regularization of the Newton Method [17]. This is actually the process (24) with the fixed parameter $\nu =1$. For this choice, in the rest part of the paper we omit the corresponding index in the definitions of all necessary objects: $M_H(x;y):=M_{1,H}(x;y)$, $T_H(x):=T_{1,H}(x)$, and $M_H^{\ast }(x):=M_{1,H}^{\ast }(x)=M_H(x;T_H(x))$. The adaptive scheme of our method with dynamic estimation of the constant H is as follows.

Algorithm 1: Adaptive Cubic Regularization of Newton Method
Initialization. Choose $x_0\in \text{dom}F$, $H_0>0$.
Iteration $k\ge 0$.
1: Find the minimal integer $i_k\ge 0$ such that $F(T_{H_k{2}^{i_k}}(x_k))\le M_{H_k{2}^{i_k}}^{\ast }(x_k)$.
2: Perform the Cubic Step: $x_{k+1}=T_{H_k{2}^{i_k}}(x_k)$.
3: Set $H_{k+1}:=2^{i_k-1}{H}_k$.

Let us present the main properties of the composite Cubic Newton step $x\mapsto T_H(x)$. Denote

$$\begin{array}{c}\hfill r_H(x):=\Vert T_H(x)-x\Vert .\end{array}$$

Since point $T_H(x)$ is a minimum of strictly convex function $M_H(x;·)$, it satisfies the following first-order optimality condition:

$$\begin{array}{cc}& \langle \mathrm{\nabla }f(x)+{\mathrm{\nabla }}^{2}f(x)(T_H(x)-x)+\frac{H{r}_H(x)}{2}B(T_H(x)-x),y-T_H(x)\rangle \hfill \\ \hfill & +h(y)\ge h(T_H(x)),y\in \text{dom}F.\hfill \end{array}$$ 27

In other words, the vector

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill h^{\prime }(T_H(x)):=& -\mathrm{\nabla }f(x)-{\mathrm{\nabla }}^{2}f(x)(T_H(x)-x)-\frac{H{r}_H(x)}{2}B(T_H(x)-x)\end{array}\end{array}$$

belongs to the subdifferential of h:

$$\begin{array}{c}\hfill \begin{array}{ccc}\hfill h^{\prime }(T_H(x))& \in & \partial h(T_H(x)).\hfill \end{array}\end{array}$$ 28

Computation of a point $T=T_H(x)$, satisfying condition (28), requires some standard techniques of Convex Optimization and Linear Algebra (see [1, 3, 16, 17]). Arithmetical complexity of such a procedure is usually similar to that of the standard Newton step.

Plugging into (27) $y:=x\in \text{dom}F$, we get:

$$\begin{array}{cc}& ⟨\mathrm{\nabla }f(x),x-T_H(x)⟩\hfill \\ \hfill & \ge ⟨{\mathrm{\nabla }}^{2}f(x)(T_H(x)-x),T_H(x)-x⟩+\frac{H{r}_H^3(x)}{2}+h(T_H(x))-h(x).\hfill \end{array}$$ 29

Thus, we obtain the following bound for the minimal value $M_H^{\ast }(x)$ of the cubic model:

$$\begin{array}{cc}& M_H^{\ast }(x)\stackrel{(29)}{\le }f(x)-\frac{1}{2}⟨{\mathrm{\nabla }}^{2}f(x)(T_H(x)-x),T_H(x)-x⟩-\frac{H{r}_H^3(x)}{3}+h(x)\hfill \\ \hfill & =F(x)-\frac{1}{2}⟨{\mathrm{\nabla }}^{2}f(x)(T_H(x)-x),T_H(x)-x⟩-\frac{H{r}_H^3(x)}{3}.\hfill \end{array}$$ 30

If for some value $\nu \in [0,1]$ the Hessian is Hölder continuous: ${\mathcal{H}}_f(\nu )<+\infty$, then by (9) and (28) we get the following bound for the subgradient:

$$\begin{array}{c}\hfill F^{\prime }(T_H(x)):=\mathrm{\nabla }f(T_H(x))+h^{\prime }(T_H(x))\end{array}$$

at the new point:

$$\begin{array}{cc}& \Vert F^{\prime }(T_H(x)){\Vert }_{\ast }\hfill \\ \hfill & \le \Vert \mathrm{\nabla }f(T_H(x))-\mathrm{\nabla }f(x)-{\mathrm{\nabla }}^{2}f(x)(T_H(x)-x){\Vert }_{\ast }+\frac{H{r}_H^2(x)}{2}\hfill \\ \hfill & \stackrel{(9)}{\le }\frac{{\mathcal{H}}_f(\nu )r_H^{1+\nu }(x)}{1+\nu }+\frac{H{r}_H^2(x)}{2}=r_H^{1+\nu }(x)·(\frac{{\mathcal{H}}_f(\nu )}{1+\nu }+\frac{H{r}_H^{1-\nu }(x)}{2}).\hfill \end{array}$$ 31

One of the main strong point of the classical Newton’s is its local quadratic convergence for the class of strongly convex functions with Lipschitz continuous Hessian: ${\sigma }_f(2)>0$ and $0<{\mathcal{H}}_f(1)<+\infty$ (see, for example, [15]). This property holds for the cubically regularized Newton as well [14, 17]. Indeed, ensuring $F(T_H(x))\le M_H^{\ast }(x)$ as in Algorithm 1, and having $H\le \beta {\mathcal{H}}_f(1)$ with some $\beta \ge 0$, we get:

$$\begin{array}{cc}& F(T_H(x))-F^{\ast }\stackrel{(5)}{\le }\frac{1}{2{\sigma }_f(2)}{\Vert F^{\prime }(T_H(x))\Vert }_{\ast }^2\stackrel{(31)}{\le }\frac{{(1+\beta )}^2{\mathcal{H}}_f^2(1)}{8{\sigma }_f(2)}{r}_H^4(x)\hfill \\ \hfill & \le \frac{{(1+\beta )}^2{\mathcal{H}}_f^2(1)}{8{\sigma }_f^3(2)}{⟨{\mathrm{\nabla }}^{2}f(x)(T_H(x)-x),T_H(x)-x⟩}^2\hfill \\ \hfill & \stackrel{(30)}{\le }\frac{{(1+\beta )}^2{\mathcal{H}}_f^2(1)}{2{\sigma }_f^3(2)}{\left(F,(x),-,F^{\ast }\right)}^2.\hfill \end{array}$$

And the region of quadratic convergence is as follows:

$$\begin{array}{c}\hfill \mathcal{Q}=\{x\in \text{dom}F:F(x)-F^{\ast }\le \frac{2{\sigma }_f^3(2)}{{(1+\beta )}^2{\mathcal{H}}_f^2(1)}\}.\end{array}$$

After reaching it, the method starts to double the right digits of the answer at every step, and this cannot last for a long time. Therefore, from now on we are mainly interested in the global complexity bounds of Algorithm 1, which work for an arbitrary starting point $x_0$.

For noncomposite case, as it was shown in [10], if for some $\nu \in [0,1]$ we have $0<{\mathcal{H}}_f(\nu )<+\infty$ and the objective is just convex, then Algorithm 1 with small initial parameter $H_0$ generates a solution $\widehat{x}$ with $f(\widehat{x})-f^{\ast }\le \epsilon$ in $O((\frac{{\mathcal{H}}_f(\nu )D_0^{2+\nu }}{\epsilon }{)}^{\frac{1}{1+\nu }})$ iterations, where $D_0:=\underset{x}{max}\left\{\Vert x-,x^{\ast },\Vert :f(x)\le f,(x_0)\right\}$. Thus, the method in [10] has a sublinear rate of convergence on the class of convex functions with Hölder continuous Hessian. It can automatically adapt to the actual level of smoothness. In what follows we show that the same algorithm achieves linear rate of convergence for the class of uniformly convex functions of degree $p=2+\nu$, namely for functions with strictly positive condition number: ${sup}_{\nu \in [0,1]}{\gamma }_f(\nu )>0.$

In the remaining part of the paper, we usually assume that the smooth part of our objective is not purely quadratic. This is equivalent to the condition ${inf}_{\nu \in [0,1]}{\mathcal{H}}_f(\nu )>0$. However, to conclude this section, let us briefly discuss the case ${min}_{\nu \in [0,1]}{\mathcal{H}}_f(\nu )=0$. If we would know in advance that f is a convex quadratic function, then no regularization is needed since a single step $x\mapsto T_H(x)$ with $H:=0$ solves the problem. However, if our function is given by a black-box oracle and we do not know a priori that its smooth part is quadratic, then we can still use Algorithm 1. For this case, we prove the following simple result.

Proposition 3.1

Let $A:\mathbb{E}\to {\mathbb{E}}^{\ast }$ be a self-adjoint positive semidefinite linear operator and $b\in {\mathbb{E}}^{\ast }$. Assume that $f(x):=\frac{1}{2}⟨Ax,x⟩-⟨b,x⟩,$ and the minimum $x^{\ast }\in \underset{x\in \text{dom}F}{\mathrm{Argmin}}\{F(x):=f(x)+h(x)\}$ does exist. Then, in order to get $F(x_K)-F^{\ast }\le \epsilon$ with arbitrary $\epsilon >0$, it is enough to perform

$$\begin{array}{c}\hfill \begin{array}{c}\hfill K=\lceil {log}_2\frac{H_0{\Vert x_0-x^{\ast }\Vert }^3}{6\epsilon }+1\rceil \end{array}\end{array}$$ 32

iterations of Algorithm 1.

Proof

In our case, the quadratic model coincides with the smooth part of the objective: $Q(x;y)\equiv f(y),x,y\in \mathbb{E}.$ Therefore, at every iteration $k\ge 0$ of Algorithm 1 we have $i_k=0$ and $H_k=2^{-k}{H}_0$. Note that $x_{k+1}=T_{2^{-k}{H}_0}(x_k)=\underset{y\in \text{dom}F}{\mathrm{argmin}}\{F(y)+\frac{2^{-k}{H}_0}{6}{\Vert y-x_k\Vert }^3\}$, and

$$\begin{array}{c}\hfill \begin{array}{ccc}\hfill F(x_{k+1})& \le & F(y)+\frac{2^{-k}{H}_0}{6}{\Vert y-x_k\Vert }^3,y\in \text{dom}F.\hfill \end{array}\end{array}$$ 33

Let us prove that $\Vert x_{k+1}-x^{\ast }\Vert \le \Vert x_k-x^{\ast }\Vert$ for all $k\ge 0$. If this is true, then plugging $y\equiv x^{\ast }$ into (33), we get: $F(x_{k+1})-F^{\ast }\le 2^{-k}\frac{H_0}{6}{\Vert x_0-x^{\ast }\Vert }^3$ which results in the estimate (32). Indeed,

$$\begin{array}{cc}\hfill \Vert x_k-x^{\ast }{\Vert }^2=& \Vert (x_k-x_{k+1})+(x_{k+1}-x^{\ast }){\Vert }^2\hfill \\ \hfill =& \Vert x_{k+1}-x^{\ast }{\Vert }^2+{\Vert x_k-x_{k+1}\Vert }^2+2⟨B(x_k-x_{k+1}),x_{k+1}-x^{\ast }⟩,\hfill \end{array}$$

and it is enough to show that $⟨B(x_k-x_{k+1}),x^{\ast }-x_{k+1}⟩\le 0$. Since $x_{k+1}$ satisfies the first-order optimality condition:

$$\begin{array}{c}\hfill \begin{array}{ccc}\hfill -2^{-(k+1)}{H}_0\Vert x_{k+1}-x_k\Vert B(x_{k+1}-x_k)& :=& F^{\prime }(x_{k+1})\in \partial F(x_{k+1}),\hfill \end{array}\end{array}$$ 34

we have:

$$\begin{array}{cc}& ⟨B(x_k-x_{k+1}),x^{\ast }-x_{k+1}⟩\hfill \\ \hfill & \stackrel{(34)}{=}\frac{2^{k+1}}{H_0\Vert x_k-x_{k+1}\Vert }⟨F^{\prime }(x_{k+1}),x^{\ast }-x_{k+1}⟩\le 0,\hfill \end{array}$$

where the last inequality follows from the convexity of the objective. $\square$

Complexity Results for Uniformly Convex Functions

In this section, we are going to justify the global linear rate of convergence of Algorithm 1 for a class of twice differentiable uniformly convex functions with Hölder continuous Hessian. Universality of this method is ensured by the adaptive estimation of the parameter H over the whole sequence of iterations. It is important to distinguish two cases: $H_{k+1}<H_k$ and $H_{k+1}\ge H_k$.

First, we need to estimate the progress in the objective function after minimizing the cubic model. There are two different situations here:

$$\begin{array}{c}\hfill \text{either}H{r}_H^{1-\nu }(x)\le \frac{2{\mathcal{H}}_f(\nu )}{1+\nu },\text{or}H{r}_H^{1-\nu }(x)>\frac{2{\mathcal{H}}_f(\nu )}{1+\nu }.\end{array}$$

Lemma 4.1

Let $0<{\mathcal{H}}_f(\nu )<+\infty$ and ${\sigma }_f(2+\nu )>0$ for some $\nu \in [0,1]$. Then, for arbitrary $x\in \text{dom}F$ and $H>0$ we have:

$$\begin{array}{cc}& F(x)-M_H^{\ast }(x)\hfill \\ \hfill & \ge min[\left(F,(x),-,F^{\ast }\right)·\frac{(1+\nu )}{(2+\nu )}·min\{(\frac{(1+\nu ){\gamma }_f(\nu )}{2(2+\nu )}{)}^{\frac{1}{1+\nu }},1\},\hfill \\ \hfill & {\left(F,(T_H(x)),-,F^{\ast }\right)}^{\frac{3(1+\nu )}{2(2+\nu )}}·(\frac{2+\nu }{1+\nu }{)}^{\frac{3(1+\nu )}{2(2+\nu )}}·\frac{{({\sigma }_f(2+\nu ))}^{\frac{3}{2(2+\nu )}}}{3\sqrt{H}}].\hfill \end{array}$$ 35

Proof

Let us consider two cases. 1) $H{r}_H^{1-\nu }(x)\le \frac{2{\mathcal{H}}_f(\nu )}{1+\nu }$. Then, for arbitrary $y\in \text{dom}F$, we have:

$$\begin{array}{cc}& M_H^{\ast }(x):=Q(x;T_H(x))+\frac{H}{6}{\Vert T_H(x)-x\Vert }^3+h(T_H(x))\hfill \\ \hfill & \le Q(x;y)+\frac{H{r}_H^{1-\nu }(x){\Vert y-x\Vert }^{2+\nu }}{2(2+\nu )}+h(y)\hfill \\ \hfill & \stackrel{(10)}{\le }F(y)+\frac{{\mathcal{H}}_f(\nu ){\Vert y-x\Vert }^{2+\nu }}{(1+\nu )(2+\nu )}+\frac{H{r}_H^{1-\nu }(x){\Vert y-x\Vert }^{2+\nu }}{2(2+\nu )}\hfill \\ \hfill & \le F(y)+\frac{2{\mathcal{H}}_f(\nu ){\Vert y-x\Vert }^{2+\nu }}{(1+\nu )(2+\nu )},\hfill \end{array}$$

where the first inequality follows from the fact, that

$$\begin{array}{cc}\hfill T_H(x)=& \underset{y\in \text{dom}F}{\mathrm{argmin}}\{Q(x;y)+\frac{H{r}_H^{1-\nu }(x){\Vert y-x\Vert }^{2+\nu }}{2(2+\nu )}+h(y)\}.\hfill \end{array}$$

Let us restrict y to the segment: $y=\alpha x^{\ast }+(1-\alpha )x,$ with $\alpha \in [0,1]$. Taking into account the uniform convexity, we get:

$$\begin{array}{cc}& M_H^{\ast }(x)\le F(x)-\alpha \left(F,(x),-,F^{\ast }\right)+{\alpha }^{2+\nu }\frac{2{\mathcal{H}}_f(\nu ){\Vert x^{\ast }-x\Vert }^{2+\nu }}{(1+\nu )(2+\nu )}\hfill \\ \hfill & \stackrel{(4)}{\le }F(x)-(\alpha -{\alpha }^{2+\nu }\frac{2{\mathcal{H}}_f(\nu )}{(1+\nu ){\sigma }_f(2+\nu )})\left(F,(x),-,F^{\ast }\right).\hfill \end{array}$$

The minimum of the right-hand side is attained at ${\alpha }^{\ast }=min\{\frac{(1+\nu ){\gamma }_f(\nu )}{2(2+\nu )},1{\}}^{\frac{1}{1+\nu }}.$ Plugging this value into the bound, we have:

$$\begin{array}{cc}\hfill M_H^{\ast }(x)\le & F(x)-min\{(\frac{(1+\nu ){\gamma }_f(\nu )}{2(2+\nu )}{)}^{1/(1+\nu )},1\}\hfill \\ \hfill & ·\frac{(1+\nu )}{(2+\nu )}·\left(F,(x),-,F^{\ast }\right),\hfill \end{array}$$

and this is the first argument of the minimum in (35).

2) $H{r}_H^{1-\nu }(x)>\frac{2{\mathcal{H}}_f(\nu )}{1+\nu }.$ By (31), we have the bound:

$$\begin{array}{c}\hfill \Vert F^{\prime }(T_H(x)){\Vert }_{\ast }<H{r}_H^2(x).\end{array}$$ 36

Using the fact that ${\mathrm{\nabla }}^{2}f(x)⪰0$, we get the second argument of the minimum:

$$\begin{array}{cc}& F(x)-M_H^{\ast }(x)\stackrel{(30)}{\ge }\frac{H{r}_H^3(x)}{3}\stackrel{(36)}{\ge }\frac{\Vert F^{\prime }(T_H(x)){\Vert }_{\ast }^{\frac{3}{2}}}{3\sqrt{H}}\hfill \\ \hfill & \stackrel{(5)}{\ge }{\left(\frac{2+\nu }{1+\nu }\right)}^{\frac{3(1+\nu )}{2(2+\nu )}}·\frac{{({\sigma }_f(2+\nu ))}^{\frac{3}{2(2+\nu )}}}{3\sqrt{H}}·{\left(F,(T_H(x)),-,F^{\ast }\right)}^{\frac{3(1+\nu )}{2(2+\nu )}}.\hfill \end{array}$$

$\square$

Denote by ${\kappa }_f(\nu )$ the following auxiliary value:

$$\begin{array}{cc}\hfill {\kappa }_f(\nu ):=& \frac{{\mathcal{H}}_f{(\nu )}^{\frac{2}{1+\nu }}}{{({\sigma }_f(2+\nu ))}^{\frac{1-\nu }{(1+\nu )(2+\nu )}}}\hfill \\ \hfill & ·\frac{6·{(8+\nu )}^{\frac{1-\nu }{1+\nu }}}{{\left((,1,+,\nu ,),(,2,+,\nu ,)\right)}^{\frac{2}{1+\nu }}}·(\frac{1+\nu }{2+\nu }{)}^{\frac{1-\nu }{2+\nu }},\nu \in [0,1].\hfill \end{array}$$ 37

The next lemma shows what happens when parameter H is increasing during the iterations.

Lemma 4.2

Assume that for a fixed $x\in \text{dom}F$ the parameter $H>0$ is such that:

$$\begin{array}{c}\hfill \begin{array}{ccc}\hfill F(T_H(x))& >& M_H^{\ast }(x).\hfill \end{array}\end{array}$$ 38

If for some $\nu \in [0,1]$, we have ${\sigma }_f(2+\nu )>0$, then it holds:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill H{\left(F,(T_{2H}(x)),-,F^{\ast }\right)}^{\frac{1-\nu }{2+\nu }}<{\kappa }_f(\nu ).\end{array}\end{array}$$ 39

Proof

Firstly, let us prove that from (38) we have:

$$\begin{array}{c}\hfill \begin{array}{ccc}\hfill H{r}_H^{1-\nu }(x)& <& \frac{6{\mathcal{H}}_f(\nu )}{(1+\nu )(2+\nu )}.\hfill \end{array}\end{array}$$ 40

Assuming by contradiction, $H{r}_H^{1-\nu }(x)\ge \frac{6{\mathcal{H}}_f(\nu )}{(1+\nu )(2+\nu )}$, we get:

$$\begin{array}{cc}& M_H^{\ast }(x):=\frac{H\Vert T_H{(x)-x\Vert }^3}{6}+Q(x;T_H(x))+h(T_H(x))\hfill \\ \hfill & \ge \frac{{\mathcal{H}}_f(\nu ){\Vert T_H(x)-x\Vert }^{2+\nu }}{(1+\nu )(2+\nu )}+Q(x;T_H(x))+h(T_H(x))\hfill \\ \hfill & \stackrel{(10)}{\ge }F(T_H(x)),\hfill \end{array}$$

which contradicts (38). Secondly, by its definition, $M_H^{\ast }(x)$ is a concave function of H. Therefore, its derivative $\frac{d}{\mathit{dH}}{M}_H^{\ast }(x)=\frac{1}{6}{r}_H^3(x)$ is non-increasing. Hence, it holds:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill r_{2H}(x)\le & r_H(x)\stackrel{(40)}{<}(\frac{6{\mathcal{H}}_f(\nu )}{(1+\nu )(2+\nu )H}{)}^{\frac{1}{1-\nu }}.\end{array}\end{array}$$ 41

Finally, by the smoothness and the uniform convexity, we obtain:

$$\begin{array}{cc}& H{\left(F,(T_{2H}(x)),-,F^{\ast }\right)}^{\frac{1-\nu }{2+\nu }}\stackrel{(5)}{\le }H{\left(\frac{1+\nu }{2+\nu },(,\frac{1}{{\sigma }_f(2+\nu )},{)}^{\frac{1}{1+\nu }}\right)}^{\frac{1-\nu }{2+\nu }}{\Vert F^{\prime }(T_{2H}(x))\Vert }_{\ast }^{\frac{1-\nu }{1+\nu }}\hfill \\ \hfill & \stackrel{(31)}{\le }H{\left(\frac{1+\nu }{2+\nu },(,\frac{1}{{\sigma }_f(2+\nu )},{)}^{\frac{1}{1+\nu }}\right)}^{\frac{1-\nu }{2+\nu }}(r_{2H}^{1+\nu }(x)·\left(\frac{{\mathcal{H}}_f(\nu )}{1+\nu },+,H,r_{2H}^{1-\nu },(x)\right){)}^{\frac{1-\nu }{1+\nu }}\hfill \\ \hfill & \stackrel{(41)}{<}H{\left(\frac{1+\nu }{2+\nu },(,\frac{1}{{\sigma }_f(2+\nu )},{)}^{\frac{1}{1+\nu }}\right)}^{\frac{1-\nu }{2+\nu }}(r_{2H}^{1+\nu }(x)·\frac{(8+\nu ){\mathcal{H}}_f(\nu )}{(1+\nu )(2+\nu )}{)}^{\frac{1-\nu }{1+\nu }}\hfill \\ \hfill & \stackrel{(41)}{<}{\left(\frac{1+\nu }{2+\nu },(,\frac{1}{{\sigma }_f(2+\nu )},{)}^{\frac{1}{1+\nu }}\right)}^{\frac{1-\nu }{2+\nu }}{\left(\frac{{\mathcal{H}}_f(\nu )}{(1+\nu )(2+\nu )}\right)}^{\frac{2}{1+\nu }}6{(8+\nu )}^{\frac{1-\nu }{1+\nu }}=:{\kappa }_f(\nu ).\hfill \end{array}$$

$\square$

We are ready to prove the main result of this paper.

Theorem 4.1

Assume that for a fixed $\nu \in [0,1]$ we have $0<{\mathcal{H}}_f(\nu )<+\infty$ and ${\sigma }_f(2+\nu )>0$. Let parameter $H_0$ in Algorithm 1 be small enough:

$$\begin{array}{c}\hfill \begin{array}{ccc}\hfill H_0& \le & \frac{{\kappa }_f(\nu )}{{\left(F,(x_0),-,F^{\ast }\right)}^{(1-\nu )/(2+\nu )}},\hfill \end{array}\end{array}$$ 42

where ${\kappa }_f(\nu )$ is defined by (37). Let the sequence ${\{x_k\}}_{k=0}^K$ generated by the method satisfy condition:

$$\begin{array}{c}\hfill F(T_{H_k{2}^j}(x_k))-F^{\ast }\ge \epsilon >0,0\le j\le i_k,0\le k\le K-1.\end{array}$$ 43

Then, for every $0\le k\le K-1$, we have:

$$\begin{array}{cc}& F(x_{k+1})-F^{\ast }\hfill \\ \hfill & \le (1-min\{\frac{(2+\nu ){\left((,1,+,\nu ,),(,2,+,\nu ,)\right)}^{1/(1+\nu )}{\left({\gamma }_f,(\nu )\right)}^{\frac{1}{1+\nu }}}{(1+\nu )6^{3/2}·2^{1/2}·{(8+\nu )}^{(1-\nu )/(2+2\nu )}},\frac{1}{2}\})·\left(F,(x_k),-,F^{\ast }\right).\hfill \\ \hfill \end{array}$$ 44

Therefore, the rate of convergence is linear, and

$$\begin{array}{c}\hfill \begin{array}{ccc}\hfill K& \le & max\{{\left({\gamma }_f,(\nu )\right)}^{\frac{-1}{1+\nu }}·\frac{1+\nu }{2+\nu }·\frac{6^{3/2}·2^{1/2}·{(8+\nu )}^{(1-\nu )/(2+2\nu )}}{{\left((,1,+,\nu ,),(,2,+,\nu ,)\right)}^{1/(1+\nu )}},1\}·log\frac{F(x_0)-F^{\ast }}{\epsilon }.\hfill \end{array}\end{array}$$

Moreover, we have the following bound for the total number of oracle calls $N_K$ during the first K iterations:

$$\begin{array}{c}\hfill \begin{array}{ccc}\hfill N_K& \le & 2K+{log}_2\frac{{\kappa }_f(\nu )}{{\epsilon }^{(1-\nu )/(2+\nu )}}-{log}_2{H}_0.\hfill \end{array}\end{array}$$ 45

Proof

The proof is based on Lemmas 4.1 and 4.2, and monotonicity of the sequence $\{F(x_k){\}}_{k\ge 0}$. Firstly, we need to show that every iteration of the method is well-defined. Namely, we are going to verify that for a fixed $0\le k\le K-1$, there exists a finite integer $\ell \ge 0$ such that either $F(T_{H_k{2}^{\ell }}(x_k))\le M_{H_k{2}^{\ell }}^{\ast }(x_k)$ or $F(T_{H_k{2}^{\ell +1}}(x_k))-F^{\ast }<\epsilon$. Indeed, let us set

$$\begin{array}{c}\hfill \begin{array}{ccc}\hfill \ell :=max\left\{0,,,{log}_2,⌈\frac{{\kappa }_f(\nu )}{H_k{\epsilon }^{(1-\nu )/(2+\nu )}}⌉\right\},& \text{and}& H:=H_k{2}^{\ell }\ge \frac{{\kappa }_f(\nu )}{{\epsilon }^{(1-\nu )/(2+\nu )}}.\hfill \end{array}\end{array}$$ 46

Then, if we have both $F(T_H(x_k))>M_H^{\ast }(x_k)$ and $F(T_{2H}(x_k))-F^{\ast }\ge \epsilon$, we get by Lemma 4.2:

$$\begin{array}{c}\hfill \begin{array}{ccc}\hfill H& \stackrel{(39)}{<}& \frac{{\kappa }_f(\nu )}{{\left(F,(T_{2H}(x_k)),-,F^{\ast }\right)}^{(1-\nu )/(2+\nu )}}\le \frac{{\kappa }_f(\nu )}{{\epsilon }^{(1-\nu )/(2+\nu )}},\hfill \end{array}\end{array}$$

which contradicts (46). Therefore, if we are unable to find the value $0\le i_k\le \ell$ (see line 1 of Algorithm) in a finite number of steps, that only means we have already solved the problem up to accuracy $\epsilon$.

Now, let us show that for every $0\le k\le K$ it holds:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill H_k{\left(F,(x_k),-,F^{\ast }\right)}^{\frac{1-\nu }{2+\nu }}\le & max\left\{{\kappa }_f,(\nu ),,,,H_0,{\left(F,(x_0),-,F^{\ast }\right)}^{\frac{1-\nu }{2+\nu }}\right\}.\end{array}\end{array}$$ 47

This inequality is obviously valid for $k=0$. Assume it is also valid for some $k\ge 0$. Then, by definition of $H_{k+1}$ (see line 3 of Algorithm), we have $H_{k+1}=H_k{2}^{i_k-1}$. There are two cases. 1) $i_k=0$. Then, $H_{k+1}<H_k$. By monotonicity of $\{F(x_k){\}}_{k\ge 0}$ and by induction, we get:

$$\begin{array}{c}\hfill \begin{array}{ccc}\hfill H_{k+1}{\left(F,(x_{k+1}),-,F^{\ast }\right)}^{\frac{1-\nu }{2+\nu }}& <& H_k{\left(F,(x_k),-,F^{\ast }\right)}^{\frac{1-\nu }{2+\nu }}\hfill \\ \hfill & \le & max\left\{{\kappa }_f,(\nu ),,,,H_0,{\left(F,(x_0),-,F^{\ast }\right)}^{\frac{1-\nu }{2+\nu }}\right\}.\hfill \end{array}\end{array}$$

2) $i_k>0$. Then, applying Lemma 4.2 with $H:=H_k{2}^{i_k-1}=H_{k+1}$ and $x:=x_k$, we have:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill H_{k+1}{\left(F,(x_{k+1}),-,F^{\ast }\right)}^{\frac{1-\nu }{2+\nu }}=H{\left(F,(T_{2H}(x)),-,F^{\ast }\right)}^{\frac{1-\nu }{2+\nu }}\stackrel{(39)}{\le }{\kappa }_f(\nu ).\end{array}\end{array}$$

Thus, (47) is true by induction. Choosing $H_0$ small enough (42), we have:

$$\begin{array}{c}\hfill \begin{array}{ccc}\hfill 2H_k{\left(F,(x_k),-,F^{\ast }\right)}^{\frac{1-\nu }{2+\nu }}& \le & 2{\kappa }_f(\nu ),0\le k\le K.\hfill \end{array}\end{array}$$ 48

From Lemma 4.1 we know, that one of the two following estimates is true (denote ${\delta }_k:=F(x_k)-F^{\ast }$):

1. $F(x_k)-F(x_{k+1})\ge \alpha ·{\delta }_k\iff {\delta }_{k+1}\le (1-\alpha )·{\delta }_k,$ or

2. $F(x_k)-F(x_{k+1})\ge \beta ·{\delta }_{k+1}\iff {\delta }_{k+1}\le {(1+\beta )}^{-1}{\delta }_k\le (1-min\{\beta ,1\}/2)·{\delta }_k$,

where $\alpha :=\frac{1+\nu }{2+\nu }·min\{(\frac{(1+\nu ){\gamma }_f(\nu )}{2(2+\nu )}{)}^{\frac{1}{1+\nu }},1\},$ and

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill \beta :=& (\frac{2+\nu }{1+\nu }{)}^{\frac{3(1+\nu )}{2(2+\nu )}}·\frac{{({\sigma }_f(2+\nu ))}^{\frac{3}{2(2+\nu )}}}{3{(2{\kappa }_f(\nu ))}^{1/2}}\stackrel{(37)}{=}\frac{2+\nu }{1+\nu }·\frac{2^{1/2}·{\left((,1,+,\nu ,),(,2,+,\nu ,)\right)}^{\frac{1}{1+\nu }}}{6^{3/2}·{(8+\nu )}^{(1-\nu )/(2+2\nu )}}·{\gamma }_f{(\nu )}^{\frac{1}{1+\nu }}.\end{array}\end{array}$$

It remains to notice that $\alpha \ge min\{\beta ,1\}/2$. Thus, we obtain (44).

Finally, let us estimate the total number of the oracle calls $N_K$ during the first K iterations. At each iteration, the oracle is called $i_k+1$ times, and we have $H_{k+1}=H_k{2}^{i_k-1}$. Therefore,

$$\begin{array}{cc}\hfill N_K=& \sum _{k=0}^{K-1}(i_k+1)=\sum _{k=0}^{K-1}\left({log}_2,\frac{H_{k+1}}{H_k},+,2\right)\hfill \\ \hfill =& 2K+{log}_2{H}_K-{log}_2{H}_0\stackrel{(48),(43)}{\le }2K+{log}_2\frac{{\kappa }_f(\nu )}{{\epsilon }^{(1-\nu )/(2+\nu )}}-{log}_2{H}_0.\hfill \end{array}$$

$\square$

Note that condition (42) for the initial choice of $H_0$ can be seen as a definition of the moment, after which we can guarantee the linear rate of convergence (44). In practice, we can launch Algorithm 1 with arbitrary $H_0>0$. There are two possible options: either the method halves $H_k$ at every step in the beginning, so $H_k$ becomes small very quickly, or this value is increased at least once, and the required bound is guaranteed by Lemma 4.2. It can be easily proved, that this initial phase requires no more than $K_0=\lceil {log}_2\frac{H_0{\epsilon }^{(1-\nu )/(1+\nu )}}{{\kappa }_f(\nu )}\rceil$ oracle calls.

Discussion

Let us discuss the global complexity results, provided by Theorem 4.1 for the Cubic Regularization of the Newton Method with the adaptive adjustment of the regularization parameter.

For the class of twice continuously differentiable strongly convex functions with Lipschitz continuous gradients $f\in {\mathcal{S}}_{\mu ,L}^{2,1}(\text{dom}F)$, it is well known that the classical gradient descent method needs

$$\begin{array}{c}\hfill \begin{array}{c}O(\frac{L}{\mu }log\frac{F(x_0)-F^{\ast }}{\epsilon })\end{array}\end{array}$$ 49

iterations for computing $\epsilon$-solution of the problem (e.g., [15]). As it was shown in [6], this result is shared by a variant of Cubic Regularization of the Newton method. This is much better than the bound $O((\frac{L}{\mu }{)}^{2}log\frac{F(x_0)-F^{\ast }}{\epsilon }),$ known for the damped Newton method (e.g., [2]).

For the class of uniformly convex functions of degree $p=2+\nu$ having Hölder continuous Hessian of degree $\nu \in [0,1]$, we have proved the following parametric estimates: $O(max\{({\gamma }_f(\nu ){)}^{\frac{-1}{1+\nu }},1\}·log\frac{F(x_0)-F^{\ast }}{\epsilon }),$ where ${\gamma }_f(\nu ):=\frac{{\sigma }_f(2+\nu )}{{\mathcal{H}}_f(\nu )}$ is the condition number of degree $\nu$. However, in practice we may not know exactly an appropriate value of the parameter $\nu$. It is important that our algorithm automatically adjusts to the best possible complexity bound:

$$\begin{array}{c}\hfill \begin{array}{c}O(max\{{inf}_{\nu \in [0,1]}({\gamma }_f(\nu ){)}^{\frac{-1}{1+\nu }},1\}·log\frac{F(x_0)-F^{\ast }}{\epsilon }).\end{array}\end{array}$$ 50

Note that for $f\in S_{\mu ,L}^{2,1}(\text{dom}F)$ we have:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill \Vert {\mathrm{\nabla }}^{2}f(x)-{\mathrm{\nabla }}^{2}f(y)\Vert \le L-\mu ,x,y\in \text{dom}F.\end{array}\end{array}$$

Thus, ${\mathcal{H}}_f(0)\le L-\mu$ and ${\gamma }_f(0)\ge \frac{\mu }{L-\mu }$. So we can conclude that the estimate (50) is better than (49). Moreover, addition to our objective arbitrary convex quadratic function does not change any of ${\mathcal{H}}_f(\nu ),\nu \in [0,1]$. Thus, it can only improve the condition number ${\gamma }_f(\nu )$, while the ratio $L/\mu$ may become arbitrarily bad. It confirms an intuition that a natural Newton-type minimization scheme should not be affected by any quadratic parts of the objective, and the notion of well-conditioned and ill-conditioned problems for second-order methods should be different from that of for first-order ones.

Note that in the recent paper [11], a linear rate of convergence was also proven for the accelerated second-order scheme, with the complexity bound:

$$\begin{array}{c}\hfill \begin{array}{c}O(max\{{({\gamma }_f(\nu ))}^{\frac{-1}{2+\nu }},1\}·log\frac{{\mathcal{H}}_f(\nu )D_0^{2+\nu }}{\epsilon }).\end{array}\end{array}$$ 51

This is the better rate than (50). However, the method requires to know the parameter $\nu$, and the constant of uniform convexity. Thus, one theoretical question remains open: is it possible to construct universal second-order scheme, matching (51) in the uniformly convex case.

Looking at the definitions of ${\mathcal{H}}_f(\nu )$ and ${\sigma }_f(2+\nu )$, we can see that, for all $x,y\in \text{dom}F,x\ne y$,

$$\begin{array}{c}\hfill \begin{array}{c}\hfill {\sigma }_f(2+\nu )\le \frac{⟨\mathrm{\nabla }f(x)-\mathrm{\nabla }f(y),x-y⟩}{{\Vert x-y\Vert }^{2+\nu }},\frac{1}{{\mathcal{H}}_f(\nu )}\le \frac{{\Vert x-y\Vert }^{\nu }}{\Vert {\mathrm{\nabla }}^{2}f(x)-{\mathrm{\nabla }}^{2}f(y)\Vert },\end{array}\end{array}$$

and

$$\begin{array}{c}\hfill \begin{array}{c}\hfill {\gamma }_f(\nu ):=\frac{{\sigma }_f(2+\nu )}{{\mathcal{H}}_f(\nu )}\le \frac{⟨\mathrm{\nabla }f(x)-\mathrm{\nabla }f(y),x-y⟩}{\Vert {\mathrm{\nabla }}^{2}f(x)-{\mathrm{\nabla }}^2{f(y)\Vert ·\Vert x-y\Vert }^2}.\end{array}\end{array}$$

The last fraction does not depend on any particular $\nu$. So, for any twice-differentiable convex function, we can define the following number:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill {\gamma }_f:=\underset{\begin{array}{c}x,y\in \text{dom}F\\ x\ne y\end{array}}{inf}\frac{⟨\mathrm{\nabla }f(x)-\mathrm{\nabla }f(y),x-y⟩}{\Vert {\mathrm{\nabla }}^{2}f(x)-{\mathrm{\nabla }}^2{f(y)\Vert ·\Vert x-y\Vert }^2}.\end{array}\end{array}$$

If it is positive, then it could serve as an indicator of the second-order non-degeneracy, for which we have a lower bound: ${\gamma }_f\ge {\gamma }_f(\nu ),\nu \in [0,1].$

Conclusions

In this work, we have introduced the second-order condition number of a certain degree, which plays as the main complexity factor for solving uniformly convex minimization problems with Hölder-continuous Hessian of the objective by second-order optimization schemes.

We have proved that cubically regularized Newton method with an adaptive estimation of the regularization parameter achieves global linear rate of convergence on this class of functions. The algorithm does not require to know any parameters of the problem class and automatically fits to the best possible degree of nondegeneracy.

Using this technique, we have justified that global iteration complexity of cubic Newton is always better than corresponding one of gradient method for the standard class of strongly convex functions with uniformly bounded second derivative.

Data sharing not applicable to this article as no datasets were generated or analyzed during the current study.