Algorithm 1. Adam Hyper-parameter Tuning

Step 1. Start Algorithm Step 2. Initialise iteration counter, t = 0 Step 3. Initialise and assign values to hyper-parameters β1, β2, $\in$, ${\mathrm{L}}_{\mathrm{r}}$, ${\mathrm{w}}_{\mathrm{t}}$, ${\mathrm{w}}_{\mathrm{t}+1}$, ${{\mathrm{v}}_{\mathrm{t}},\mathrm{s}}_{\mathrm{t}}$ Step 4. Initialise parameters (weights) for the chosen classifier Step 5. Define the loss function to be minimised. Step 6. For each iteration t: Step 7. Compute the gradient of the loss function with respect to the hyper-parameters, $\frac{\mathsf{\partial }{\mathrm{L}}_{\mathrm{r}}}{\mathsf{\partial }{\mathrm{w}}_{\mathrm{t}}}$ Step 8. Update the exponential moving averages of the gradient and its square, ${\mathrm{v}}_{\mathrm{t}}$ and ${\mathrm{s}}_{\mathrm{t}}$ using Equations (30) and (31) Step 9. Compute bias-corrected estimates of the averages, $\widehat{{\mathrm{V}}_{\mathrm{t}}}$ and $\widehat{{\mathrm{S}}_{\mathrm{t}}}$ using Equations (28) and (29) Step 10. Update the parameters (weights) or the chosen classifier Step 11. Calculate $\mathrm{E}\mathrm{R}$ for the current equation Step 12. If $\mathrm{t}\mathrm{r}=1$, compute the gradient of the loss function with respect to the hyper-parameter ${\mathrm{w}}_{\mathrm{i}\mathrm{n}}$ Step 13. Else if $\mathrm{t}\mathrm{r}>1$, compute the gradient of the loss function with respect to the hyper-parameter ${\mathrm{w}}_{\mathrm{t}\mathrm{r}}$ Step 14. Update the hyper-parameter ${\mathrm{w}}_{\mathrm{t}+1}$ Step 15. If t = ConvCrit Step 16. Go to Step 19 Step 17. Else Step 18. Go to Step 7 Step 19. End Algorithm