Algorithm 1: A pseudocode of Adam optimization algorithm

Algorithm: Adam  Input: f, $\mathbf{w}$, $\alpha$, ${\beta }_1$, ${\beta }_2$  Output: ${\mathbf{w}}^{*}$  Begin Initialize $t=0,{\mathbf{m}}_1=\mathbf{0},{\mathbf{v}}_1=\mathbf{0}$  while not converged do:  $t=t+1$  ${\mathbf{g}}_t={\nabla }_{\mathbf{w}}f\left({\mathbf{w}}_t\right)$  ${\mathbf{m}}_{t+1}={\beta }_1{\mathbf{m}}_t+(1-{\beta }_1){\mathbf{g}}_t$  ${\mathbf{v}}_{t+1}={\beta }_2{\mathbf{v}}_t+(1-{\beta }_2){\mathbf{g}}_t^2$  ${\mathbf{m}}_{\widehat{t}+1}=\frac{{\mathbf{m}}_{t+1}}{1-{\beta }_1^t}$,   ${\mathbf{v}}_{\widehat{t}+1}=\frac{{\mathbf{v}}_{t+1}}{1-{\beta }_2^t}$  ${\mathbf{w}}_{t+1}={\mathbf{w}}_t-\alpha \frac{{\mathbf{m}}_{\widehat{t}+1}}{\sqrt{{\mathbf{v}}_{\widehat{t}+1}}}$  end while  ${\mathbf{w}}^{*}={\mathbf{w}}_t$  End Begin