A family of conjugate gradient methods for large-scale nonlinear equations

Abstract

In this paper, we present a family of conjugate gradient projection methods for solving large-scale nonlinear equations. At each iteration, it needs low storage and the subproblem can be easily solved. Compared with the existing solution methods for solving the problem, its global convergence is established without the restriction of the Lipschitz continuity on the underlying mapping. Preliminary numerical results are reported to show the efficiency of the proposed method.

Introduction

Consider the following nonlinear equations problem of finding $x\in C$ such that

$$F(x)=0,$$ 1.1

where $F:R^n\to R^n$ is a continuous nonlinear mapping and C is a nonempty closed convex set of $R^n$. The problem finds wide applications in areas such as ballistic trajectory computation and vibration systems [1, 2], the power flow equations [3–5], economic equilibrium problem [6–8], etc.

Generally, there are two categories of solution methods for solving the problem. The first one is first-order methods including the trust region method, the Levenberg-Marquardt method and the projection method. The second one is second-order methods including the Newton method and quasi-Newton method. For the first method, Zhang et al. [9] proposed a spectral gradient method for problem (1.1) with $C=R^n$, and Wang et al. [3] proposed a projection method for problem (1.1). Later, Yu et al. [10] proposed a spectral gradient projection method for constrained nonlinear equations. Compared with the projection method in [3], the methods [9, 10] need the Lipschitz continuity of the underlying mapping $F(\cdot )$, but the former needs to solve a linear equation at each iteration, and its variants [11, 12] also inherit the shortcoming. Different from the above, in this paper, we consider the conjugate gradient method for solving the non-monotone problem (1.1). To this end, we briefly review the well-known conjugate gradient method for the unconstrained optimization problem

$$\underset{x\in R^n}{min}f(x).$$

The conjugate gradient method generates the sequence of iterates recurrently by

$$x_{k+1}=x_k+{\alpha }_k{d}_k,k=0,1,2,\dots ,$$

where $x_k$ is the current iterate, ${\alpha }_k>0$ is the step-size determined by some line search, and $d_k$ is the search direction defined by

$$d_k=\{\begin{array}{cc}-g_k\hfill & \text{if }k=0,\hfill \\ -g_k+{\beta }_k{d}_{k-1}\hfill & \text{if }k\ge 1,\hfill \end{array}$$

in which $g_k=\mathrm{\nabla }f(x_k)$ and ${\beta }_k>0$ is a parameter. The famous conjugate gradient methods include the Fletcher-Reeves (FR) method, the Polak-Ribière-Polyak (PRP) method, the Liu-Storey (LS) method, the Dai-Yuan (DY) method. Recently, Sun et al. [13] and Li et al. [14] proposed two variants of PRP method which possesses the following property:

$$|{\beta }_k|\le t\frac{\parallel g_k\parallel }{\parallel d_{k-1}\parallel },\mathrm{\forall }k\ge 1,$$

where $t>0$ is a constant.

In this paper, motivated by the projection methods in [3, 11] and the conjugate gradient methods in [13, 14], we propose a new family of conjugate gradient projection methods for solving nonlinear problem (1.1). The new designed method is derivative-free as it does not need to compute the Jacobian matrix or its approximation of the underlying function (1.1). Further, the new method does not need to solve any linear equations at each iteration, thus it is suitable to solve large scale problem (1.1).

The remainder of this paper is organized as follows. Section 2 describes the new method and presents its convergence. The numerical results are reported in Section 3. Some concluding remarks are drawn in the last section.

Algorithm and convergence analysis

Throughout this paper, we assume that the mapping $F(\cdot )$ is monotone, or more generally pseudo-monotone, on $R^n$ in the sense of Karamardian [15]. That is, it satisfies that

$$\langle F(y),y-x^{\ast }\rangle \ge 0,\text{for all }y\in R^n,x^{\ast }\in S,$$ 2.1

where $\langle \cdot ,\cdot \rangle$ denotes the usual inner product in $R^n$. Further, we use $P_C(x)$ to denote the projection of point $x\in R^n$ onto the convex set C, which satisfies the following property:

$${\parallel P_C(x)-P_C(y)\parallel }^2\le {\parallel x-y\parallel }^2-{\parallel P_C(x)-x+y-P_C(y)\parallel }^2,\mathrm{\forall }x,y\in R^n.$$ 2.2

Now, we describe the new conjugate gradient projection method for nonlinear constrained equations.

Algorithm 2.1

Step 0.

Given an arbitrary initial point $x_0\in R^n$, parameters $0<\rho <1,\sigma >0,t>0,\beta >0,ϵ>0$, and set $k:=0$.

Step 1.

If $\parallel F(x_k)\parallel <ϵ$, stop; otherwise go to Step 2.

Step 2.

Compute

$$d_k=\{\begin{array}{cc}-F(x_k)\hfill & \text{if }k=0,\hfill \\ -(1+{\beta }_k\frac{F{(x_k)}^{\mathrm{\top }}{d}_{k-1}}{{\parallel F(x_k)\parallel }^2})F(x_k)+{\beta }_k{d}_{k-1}\hfill & \text{if }k\ge 1,\hfill \end{array}$$ 2.3

where ${\beta }_k$ is such that

$$|{\beta }_k|\le t\frac{\parallel F(x_k)\parallel }{\parallel d_{k-1}\parallel },\mathrm{\forall }k\ge 1.$$ 2.4

Step 3.

Find the trial point $y_k=x_k+{\alpha }_k{d}_k$, where ${\alpha }_k=\beta {\rho }^{m_k}$ and $m_k$ is the smallest nonnegative integer m such that

$$-\langle F(x_k+\beta {\rho }^m{d}_k),d_k\rangle \ge \sigma \beta {\rho }^m{\parallel d_k\parallel }^2.$$ 2.5

Step 4.

Compute

$$x_{k+1}=P_{H_k}[x_k-{\xi }_{k}F(y_k)],$$ 2.6

where

$$H_k=\{x\in R^n|h_k(x)\le 0\},$$

with

$$h_k(x)=\langle F(y_k),x-y_k\rangle ,$$ 2.7

and

$${\xi }_k=\frac{\langle F(y_k),x_k-y_k\rangle }{{\parallel F(y_k)\parallel }^2}.$$

Set $k:=k+1$ and go to Step 1.

Obviously, Algorithm 2.1 is different from the methods in [13, 14].

Now, we give some comment on the searching direction $d_k$ defined by (2.3). We claim that it is derived from Schmidt orthogonalization. In fact, in order to make $d_k=-F(x_k)+{\beta }_k{d}_{k-1}$ satisfy the property

$$F{(x_k)}^{\mathrm{\top }}{d}_k=-{\parallel F(x_k)\parallel }^2,$$ 2.8

we only need to ensure that ${\beta }_k{d}_{k-1}$ is vertical to $F(x_k)$. As a matter of fact, by Schmidt orthogonalization, we have

$$d_k=-(1+{\beta }_k\frac{F{(x_k)}^{\mathrm{\top }}{d}_{k-1}}{{\parallel F(x_k)\parallel }^2})F(x_k)+{\beta }_k{d}_{k-1}.$$

Equality (2.8) together with the Cauchy-Schwarz inequality implies that $\parallel d_k\parallel \ge \parallel F(x_k)\parallel$. In addition, by (2.3) and (2.4), we have

$$\begin{array}{rl}\parallel d_k\parallel & \le \parallel F(x_k)\parallel +|{\beta }_k|\frac{\parallel F(x_k)\parallel \parallel d_{k-1}\parallel }{{\parallel F(x_k)\parallel }^2}\parallel F(x_k)\parallel +|{\beta }_k|\parallel d_{k-1}\parallel \\ & \le \parallel F(x_k)\parallel +t\parallel F(x_k)\parallel +t\parallel F(x_k)\parallel \\ & =(1+2t)\parallel F(x_k)\parallel .\end{array}$$

Therefore, for all $k\ge 0$, it holds that

$$\parallel F(x_k)\parallel \le \parallel d_k\parallel \le (1+2t)\parallel F(x_k)\parallel .$$ 2.9

Furthermore, it is easy to see that the line search (2.5) is well defined if $F(x_k)\ne 0$.

For parameter ${\beta }_k$ defined by (2.4), it has many choices such as ${\beta }_k^{\mathrm{S}1}=\parallel F(x_k)\parallel /\parallel d_{k-1}\parallel$, or [13, 14]

$$\begin{array}{rl}& {\beta }_k^{\mathrm{NWYL}}=\frac{\langle F(x_k),F(x_k)-\frac{\parallel F(x_k)\parallel }{\parallel F(x_{k-1})\parallel }F(x_{k-1})\rangle }{|F{(x_k)}^{\mathrm{\top }}{d}_{k-1}|+t\parallel F(x_k)\parallel \parallel d_{k-1}\parallel },\\ & {\beta }_k^{\mathrm{NPRP}}=\frac{\langle F(x_k),F(x_k)-F(x_{k-1})\rangle }{max\{t\parallel d_{k-1}\parallel ,{\parallel F(x_{k-1})\parallel }^2\}}.\end{array}$$

From the structure of $H_k$, the orthogonal projection onto $H_k$ has a closed-form expression. That is,

$$P_{H_k}(x)=\{\begin{array}{cc}x-\frac{\langle F(y_k),x-y_k\rangle }{{\parallel F(y_k)\parallel }^2}F(y_k)\hfill & \text{if }\langle F(y_k),x-y_k\rangle >0,\hfill \\ x\hfill & \text{if }\langle F(y_k),x-y_k\rangle \le 0.\hfill \end{array}$$

Lemma 2.1

For function $h_k(x)$ defined by (2.7), let $x^{\ast }\in S^{\ast }$, it holds that

$$h_k(x_k)\ge \sigma {\parallel x_k-y_k\parallel }^2\mathit{\text{and}}{h}_k\left(x^{\ast }\right)\le 0.$$ 2.10

In particular, if $x_k\ne y_k$, then $h_k(x_k)>0$.

Proof

From $x_k-y_k=-{\alpha }_k{d}_k$ and the line search (2.5), we have

$$\begin{array}{rl}{h}_k(x_k)& =\langle F(y_k),x_k-y_k\rangle \\ & =-{\alpha }_k\langle F(y_k),d_k\rangle \ge {\alpha }_k^2\sigma {\parallel d_k\parallel }^2=\sigma {\parallel x_k-y_k\parallel }^2.\end{array}$$

On the other hand, from condition (2.1), we can obtain

$$h_k\left(x^{\ast }\right)=\langle F(y_k),x^{\ast }-y_k\rangle \le 0.$$

This completes the proof. □

Lemma 2.1 indicates that the hyperplane $\partial H_k=\{x\in R^n|h_k(x)=0\}$ strictly separates the current iterate from the solutions of problem (1.1) if $x_k$ is not a solution. In addition, from Lemma 2.1, we also can derive that the solution set $S^{\ast }$ of problem (1.1) is included in $H_k$ for all k.

Certainly, if Algorithm 2.1 terminates at step k, then $x_k$ is a solution of problem (1.1). So, in the following analysis, we assume that Algorithm 2.1 always generates an infinite sequence $\{x_k\}$. Based on the lemma, we can establish the convergence of the algorithm.

Theorem 2.1

If F is continuous and condition (2.1) holds, then the sequence $\{x_k\}$ generated by Algorithm 2.1 globally converges to a solution of problem (1.1).

Proof

First, we show that the sequences $\{x_k\}$ and $\{y_k\}$ are both bounded. In fact, it follows from $x^{\ast }\in H_k$, (2.1), (2.2) and (2.6) that

$$\begin{array}{rl}{\parallel x_{k+1}-x^{\ast }\parallel }^2& \le {\parallel x_k-{\xi }_{k}F(y_k)-x^{\ast }\parallel }^2\\ & ={\parallel x_k-x^{\ast }\parallel }^2-2{\xi }_k\langle F(y_k),x_k-x^{\ast }\rangle +{\xi }_k^2{\parallel F(y_k)\parallel }^2\\ & \le {\parallel x_k-x^{\ast }\parallel }^2-2{\xi }_k\langle F(y_k),x_k-y_k\rangle +{\xi }_k^2{\parallel F(y_k)\parallel }^2\\ & ={\parallel x_k-x^{\ast }\parallel }^2-\frac{{\langle F(y_k),x_k-y_k\rangle }^2}{{\parallel F(y_k)\parallel }^2}\\ & \le {\parallel x_k-x^{\ast }\parallel }^2-\frac{{\sigma }^2{\alpha }_k^4{\parallel d_k\parallel }^2}{{\parallel F(y_k)\parallel }^4}.\end{array}$$

Thus the sequence $\{\parallel x_k-x^{\ast }\parallel \}$ is decreasing and convergent, and hence the sequence $\{x_k\}$ is bounded, and from (2.9), the sequence $\{d_k\}$ is also bounded. Then, by $y_k=x_k+{\alpha }_k{d}_k$, the sequence $\{y_k\}$ is also bounded. Then, by the continuity of $F(\cdot )$, there exists constant $M>0$ such that $\parallel F(y_k)\parallel \le M$ for all k. So,

$${\parallel x_{k+1}-x^{\ast }\parallel }^2\le {\parallel x_k-x^{\ast }\parallel }^2-\frac{{\sigma }^2{\alpha }_k^4{\parallel d_k\parallel }^4}{M^2},$$ 2.11

from which we can deduce that

$$\underset{k\to \mathrm{\infty }}{lim}{\alpha }_k\parallel d_k\parallel =0.$$ 2.12

If ${lim\hspace{0.17em}inf}_{k\to \mathrm{\infty }}\parallel d_k\parallel =0$, then from (2.9) it holds that ${lim\hspace{0.17em}inf}_{k\to \mathrm{\infty }}\parallel F(x_k)\parallel =0$. From the boundedness of $\{x_k\}$ and the continuity of $F(\cdot )$, $\{x_k\}$ has some accumulation point x̄ such that $F(\overline{x})=0$. Then from (2.11), $\{\parallel x_k-\overline{x}\parallel \}$ converges, and thus the sequence $\{x_k\}$ globally converges to x̄.

If ${lim\hspace{0.17em}inf}_{k\to \mathrm{\infty }}\parallel d_k\parallel >0$, from (2.9) again, we have

$$\underset{k\to \mathrm{\infty }}{lim\hspace{0.17em}inf}\parallel F(x_k)\parallel >0.$$ 2.13

By (2.12), it holds that

$$\underset{k\to \mathrm{\infty }}{lim}{\alpha }_k=0.$$ 2.14

Therefore, from the line search (2.5), we have

$$-\langle F(x_k+\beta {\rho }^{m_k-1}{d}_k),d_k\rangle <\sigma \beta {\rho }^{m_k-1}{\parallel d_k\parallel }^2.$$ 2.15

Since $\{x_k\}$ and $\{d_k\}$ are both bounded, then letting $k\to \mathrm{\infty }$ in (2.15) yields that

$$-\langle F(\overline{x}),\overline{d}\rangle \le 0,$$ 2.16

where $\overline{x},\overline{d}$ are limits of corresponding sequences. In addition, from (2.8) and (2.13), we get

$$-\langle F(\overline{x}),\overline{d}\rangle ={\parallel F(\overline{x})\parallel }^2>0.$$ 2.17

Obviously, (2.16) contradicts (2.17). This completes the proof. □

Numerical results

In this section, numerical results are provided to substantiate the efficacy of the proposed method. The codes are written in Mablab R2010a and run on a personal computer with 2.0 GHz CPU processor. For comparison, we also give the numerical results of the spectral gradient method (denoted by SGM) in [9], the conjugate gradient method (denoted by CGM) in [16]. The parameters used in Algorithm 2.1 are set as $t=1,\sigma =0.01,\rho =0.5,\beta =1$, and

$${\beta }_k=\frac{\parallel F(x_k)\parallel }{\parallel d_{k-1}\parallel }.$$

For SGM, we set $\beta =0.4,\sigma =0.01,r=0.001$. For CGM, we choose $\rho =0.1,\sigma ={10}^{-4}$ and $\xi =1$. Furthermore, the stopping criterion is set as $\parallel F(x_k)\parallel \le {10}^{-6}$ for all the tested methods.

Problem 1

The mapping $F(\cdot )$ is taken as $F(x)={(f_1(x),f_2(x),\dots ,f_n(x))}^{\mathrm{\top }}$, where

$$f_i(x)=e^{x_i}-1\text{for }i=1,2,\dots ,n.$$

Obviously, this problem has a unique solution $x^{\ast }={(0,0,\dots ,0)}^{\mathrm{\top }}$.

Problem 2

The mapping $F(\cdot )$ is taken as $F(x)={(f_1(x),f_2(x),\dots ,f_n(x))}^{\mathrm{\top }}$, where

$$\begin{array}{rl}& f_1(x)=x_1-e^{cos(\frac{x_1+x_2}{n+1})},\\ & f_i(x)=x_i-e^{cos(\frac{x_{i-1}+x_i+x_{i+1}}{n+1})},\text{for }i=2,3,\dots ,n-1,\\ & f_n(x)=x_n-e^{cos(\frac{x_{n-1}+x_n}{n+1})}.\end{array}$$

Problem 3

The mapping $F(x):R^4\to R^4$ is given by

$$F(x)=\left(\begin{array}{cccc}1& 0& 0& 0\\ 0& 1& -1& 0\\ 0& 1& 1& 0\\ 0& 0& 0& 0\end{array}\right)\left(\begin{array}{c}{x}_1\\ x_2\\ x_3\\ x_4\end{array}\right)+\left(\begin{array}{c}{x}_1^3\\ x_2^3\\ 2x_3^3\\ 2x_4^3\end{array}\right)+\left(\begin{array}{c}-10\\ 1\\ -3\\ 0\end{array}\right).$$

This problem has a degenerate solution $x^{\ast }=(2,0,1,0)$.

For Problem 1, the initial point is set as $x_0=(1,1,\dots ,1)$, and Table 1 gives the numerical results by Algorithm 2.1 with different dimensions, where Iter. denotes the iteration number and CPU denotes the CPU time in seconds when the algorithm terminates. Table 2 lists the numerical results of Problem 2 with different initial points. The numerical results given in Table 1 and Table 2 show that the proposed method is efficient for solving the given two test problems.

Table 1.

Numerical results with different dimensions of Problem 1

Dimension	10	50	100	500	1,000
Iter.	20	21	22	23	23
CPU	0.3594	0.8750	1.7031	41.4844	257.1406

Table 2.

Numerical results with different initial points of Problem 3

Initial point	Iter.	CPU	$\mathbf{\parallel }\mathit{F}\mathbf{(}{\mathit{x}}_{\mathit{k}}\mathbf{)}\mathbf{\parallel }$
(0,0,0,0)	20	0.9219	9.9320×10−7
(3,0,0,0)	22	0.8438	7.0942×10−7
(1,1,1,0)	41	1.6250	7.2649×10−7
(0,1,1,1)	28	1.1875	3.8145×10−7
(0,100,100,1)	34	1.4063	6.6320×10−7

Conclusion

In this paper, we extended the conjugate gradient method to nonlinear equations. The major advantage of the method is that it does not need to compute the Jacobian matrix or any linear equations at each iteration, thus it is suitable to solve large-scale nonlinear constrained equations. Under mild conditions, the proposed method possesses global convergence.

In Step 4 of Algorithm 2.1, we have to compute a projection onto the intersection of the feasible set C and a half-space at each iteration, which is equivalent to quadratic programming, quite time-consuming work. Hence, how to remove this projection step is one of our future research topics.

Authors’ contributions

DXF and MS organized and wrote this paper. XYW examined all the steps of the proofs in this research and gave some advice. All authors read and approved the final manuscript.

Competing interests

The authors declare that they have no competing interests.