Algorithm 2. RanAdam Hyper-parameter Tuning

Step 1. Start Algorithm Step 2. Initialise iteration counter, t = 0 Step 3. Initialise and assign values to hyper-parameters β1, β2, $\in$, ${\mathrm{L}}_{\mathrm{r}}$, ${\mathrm{w}}_{\mathrm{t}}$, ${\mathrm{w}}_{\mathrm{t}+1}$, ${{\mathrm{v}}_{\mathrm{t}},\mathrm{s}}_{\mathrm{t}}$ Step 4. Initialise parameters (weights) for the chosen classifier Step 5. Define the loss function to be minimised Step 6. For each iteration t: Step 7. Compute the gradient of the loss function with respect to the hyper-parameters, $\frac{\mathsf{\partial }{\mathrm{L}}_{\mathrm{r}}}{\mathsf{\partial }{\mathrm{w}}_{\mathrm{t}}}$ Step 8. Update the exponential moving averages of the gradient and its square, ${\mathrm{v}}_{\mathrm{t}}$ and ${\mathrm{s}}_{\mathrm{t}}$ using Equations (30) and (31) Step 9. Compute bias-corrected estimates of the averages, $\widehat{{\mathrm{V}}_{\mathrm{t}}}$ and $\widehat{{\mathrm{S}}_{\mathrm{t}}}$ using Equations (28) and (29) Step 10. Update the parameters (weights) or the chosen classifier Step 11. Calculate $\mathrm{E}\mathrm{R}$ for the current equation Step 12. If $\mathrm{t}\mathrm{r}=1$, compute the gradient of the loss function with respect to the hyper-parameter ${\mathrm{w}}_{\mathrm{i}\mathrm{n}}$ Step 13. Else if $>1$, compute the gradient of the loss function with respect to the hyper-parameter ${\mathrm{w}}_{\mathrm{t}\mathrm{r}}$. Step 14. Initialise random numbers for Rand1, Rand2, Rand3, Rand4 and specify bandwidth Step 15. if rand 1 < solution considering rate Step 16. ${{\mathrm{w}}^{\prime }}_{\mathrm{t}+1}={{\mathrm{w}}^{\prime }}_{\mathrm{t}}$ Step 17. End if Step 18. if rand 2 < solution adjusting rate Step 19. ${{\mathrm{w}}^{\prime }}_{\mathrm{t}+1}={{\mathrm{w}}^{\prime }}_{\mathrm{t}}$ * bandwidth * rand 3 Step 20. End if Step 21. If ${{\mathrm{w}}^{\prime }}_{\mathrm{t}+1}$ < Lower bound (LB) Step 22. ${{\mathrm{w}}^{\prime }}_{\mathrm{t}+1}=\mathrm{L}\mathrm{B}$ Step 23. End if Step 24. if ${{\mathrm{w}}^{\prime }}_{\mathrm{t}+1}$ > Upper bound (UB) Step 25. ${{\mathrm{w}}^{\prime }}_{\mathrm{t}+1}$ = UB Step 26. End if Step 27. if ${{\mathrm{w}}^{\prime }}_{\mathrm{t}+1}$ < UB Step 28. ${{\mathrm{w}}^{\prime }}_{\mathrm{t}+1}$ = LB + rand4 * bandwidth Step 29. End if Step 30. If (ER = minimum ER) Step 31. Optimum weight, ${{\mathrm{w}}^{\prime }}_{\mathrm{o}\mathrm{p}\mathrm{t}}$ = ${{\mathrm{w}}^{\prime }}_{\mathrm{t}+1}$ Step 32. Else Step 33. Go to Step 14 Step 34. If t = ConvCrit Step 35. Go to Step 38 Step 36. Else Step 37. Go to Step 7 Step 38. End Algorithm