Algorithm 1: Learning the best threshold level in the first learning hierarchy level.

1. Input:   Initialize time step (n), α, γ,$\mathit{T}{\mathit{h}}_{\mathit{l}}, \mathit{T}{\mathit{h}}_{\mathit{o}\mathit{l}\mathit{d}}$, Threshold, and ${\mathit{Q}}_{\mathit{T}\mathit{h}}$ matrices.   Make $\mathit{T}{\mathit{h}}_{\mathit{c}} \mathit{m}\mathit{a}\mathit{t}\mathit{r}\mathit{i}\mathit{x}$ as the accurate threshold matrix based on the transition states.

2. While not converged, do

1. Generate two binary random values as ${\mathit{z}}_{\mathit{n}} \mathit{a}\mathit{n}\mathit{d} {\mathit{z}}_{\mathit{m}}$

2. Determine the accurate threshold, Th, based on $\mathit{T}{\mathit{h}}_{\mathit{c}}$

3. With probability ɛ, randomly choose a value for the threshold from $\left({\mathbf{2}}^{\mathit{T}{\mathit{h}}_{\mathbf{l}}}\mathbf{-}\mathbf{1}\right)$ levels.

4. Otherwise, $\mathit{T}{\mathit{h}}_{\mathit{n}\mathit{e}\mathit{w}}=\mathit{a}\mathit{r}\mathit{g}\mathit{m}\mathit{a}{\mathit{x}}_{\mathit{j}\in \left({\mathbf{2}}^{\mathit{T}{\mathit{h}}_{\mathit{l}}}-\mathbf{1}\right) }\left({\mathit{Q}}_{\mathit{t}\mathit{h}}\left(\mathit{T}{\mathit{h}}_{\mathit{o}\mathit{l}\mathit{d}},\mathit{j}\right)\right)$

5. If $\mathit{T}\mathit{h}=\mathit{T}{\mathit{h}}_{\mathit{n}\mathit{e}\mathit{w}}$    ${\mathit{R}}_{\mathit{T}\mathit{h}}\left(\mathit{n}\right)=$    ${\mathit{N}}_{\mathit{T}\mathit{h}}-\mathit{t}\mathit{h}\mathit{e} \mathit{n}\mathit{u}\mathit{m}\mathit{b}\mathit{e}\mathit{r} \mathit{o}\mathit{f} \mathit{n}\mathit{o}\mathit{n}\mathit{z}\mathit{e}\mathit{r}\mathit{o} \mathit{e}\mathit{l}\mathit{e}\mathit{m}\mathit{e}\mathit{n}\mathit{t}\mathit{s} \mathit{o}\mathit{f} (\mathit{T}\mathit{h}\mathit{r}\mathit{e}\mathit{s}\mathit{h}\mathit{o}\mathit{l}\mathit{d}-\mathit{T}{\mathit{h}}_{\mathit{c}}$)

6. else    ${\mathit{R}}_{\mathit{T}\mathit{h}}\left(\mathit{n}\right)=\mathbf{0}$

7.  End

8. Update the ${\mathit{Q}}_{\mathit{T}\mathit{h}}$ matrix    ${\mathit{Q}}_{\mathit{T}\mathit{h}}\left({\mathit{z}}_{\mathit{n}},{\mathit{z}}_{\mathit{m}},\mathit{T}{\mathit{h}}_{\mathit{o}\mathit{l}\mathit{d}},\mathit{T}{\mathit{h}}_{\mathit{n}\mathit{e}\mathit{w}}\right)=\left(\mathbf{1}-\mathit{\alpha }\right){\mathit{Q}}_{\mathit{T}\mathit{h}}\left({\mathit{z}}_{\mathit{n}},{\mathit{z}}_{\mathit{m}},\mathit{T}{\mathit{h}}_{\mathit{o}\mathit{l}\mathit{d}},\mathit{T}{\mathit{h}}_{\mathit{n}\mathit{e}\mathit{w}}\right)+\mathit{\alpha } \left({\mathit{R}}_{\mathit{T}\mathit{h}}+\mathit{\gamma }\left(\mathit{m}\mathit{a}\mathit{x} \left({\mathit{Q}}_{\mathit{T}\mathit{h}}\left({\mathit{z}}_{\mathit{n}},{\mathit{z}}_{\mathit{m}},\mathit{T}{\mathit{h}}_{\mathit{o}\mathit{l}\mathit{d}},\mathit{T}{\mathit{h}}_{\mathit{n}\mathit{e}\mathit{w}}\right)\right)\right)\right)$

9. $\mathit{T}\mathit{h}\mathit{r}\mathit{e}\mathit{s}\mathit{h}\mathit{o}\mathit{l}\mathit{d}\left({\mathit{z}}_{\mathit{n}},{\mathit{z}}_{\mathit{m}},\mathit{T}{\mathit{h}}_{\mathit{o}\mathit{l}\mathit{d}}\right)=\mathit{T}{\mathit{h}}_{\mathit{n}\mathit{e}\mathit{w}}$

10. Increase the time step n by 1

3. End while