Table 1.

The meaning of parameters and variables which are used in discussion of optimization algorithms

Variable name	Variable function
ϵ	Learning rate
θ	Performance criterion
$\tilde{\mathrm{\theta }}$	Updating performance criterion (θ)
α	Momentum parameter
υ	Speed of movement in parameters space
g	Calculation of gradient
$\widehat{g}$	Estimation of gradient vector
g ⨀ g	Calculation of partial derivatives squares
r	Integrating partial derivatives vector
$\widehat{r}$	Bias correlation of first order (r)
s	Integrating vector of square partial derivatives
$\widehat{s}$	Bias correlation of second order (s)
w	Weighted of neurons
t	Time slot
ρ	The damping rate in the weighted sum
δ	The numerical stability constant
△θ	Calculation of learning rate
m	Number of sample in dataset
O	Maximum value
x(1), …, x(m)	The samples of data set
y(1), …, y(m)	Labels of samples in dataset