Algorithm 1 Training process in the reward learning process

Input: number of full episodes $K$, timesteps $T$, fixed parameters ${\mathsf{\theta }}_{\mathit{o}\mathit{l}\mathit{d}}$, target firing rate ${\mathit{f}}^{\mathit{0}}$, regularization hyper-parameters ${\mu }_v$, ${\mu }_e$, ${\mu }_{firing}$, bandwidth $\mathit{\sigma }$, predicted value function $V_{\mathsf{\theta }}\left(\mathit{t},\mathit{k}\right)$ and sum of future rewards $R\left(t,k\right)$ Output: total loss ${\mathit{L}}_{\mathsf{\theta }}$.     1.Parameters setting: ${\mathit{f}}^{\mathbf{0}}$, ${\mu }_v$, ${\mu }_e$, ${\mu }_{firing} \mathit{a}\mathit{n}\mathit{d}$ $\mathit{\sigma }$.     2.for n in batch size N:     3.       Set ${\mathit{e}}_{\mathit{n}}=R\left(t,k\right)-V_{\mathsf{\theta }}\left(\mathit{t},\mathit{k}\right)$     4.       if number of literation is 0:     5.                 $\left({\mathit{\phi }}_0, {\mathit{\phi }}_{-\mathbf{1}}, {\mathit{\phi }}_{\mathbf{1}}\right)=\left(\mathit{N}, \mathbf{0}, \mathbf{0}\right)$     6.       else:                 $\left({\mathit{\phi }}_{\mathbf{0}}, {\mathit{\phi }}_{-\mathbf{1}}, {\mathit{\phi }}_{\mathbf{1}}\right)=(\#\left\{{\mathit{e}}_{\mathit{n}}\in \left(-\mathbf{0.5}, \mathbf{0.5}\right)\right\},$                                       $\#\left\{{\mathit{e}}_{\mathit{n}}\in \left(-\mathbf{1},-\mathbf{0.5}\right)\right\},$                                         $\#\left\{{\mathit{e}}_{\mathit{n}}\in \left(\mathbf{0.5}, \mathbf{1}\right)\right\})$                 where #{·} indicates counting the samples that satisfy the condition     7.        $({\mathit{u}}_{\mathit{n}},{\mathit{v}}_{\mathit{n}},{\mathit{s}}_{\mathit{n}})=(-\mathit{e}\mathit{x}\mathit{p}(-\frac{{\mathit{e}}_{\mathit{n}}{}^{\mathbf{2}}}{\mathbf{2}{\mathit{\sigma }}^{\mathbf{2}}}),-\mathit{e}\mathit{x}\mathit{p}(-\frac{{({\mathit{e}}_{\mathit{n}}+\mathbf{1})}^{\mathbf{2}}}{\mathbf{2}{\mathit{\sigma }}^{\mathbf{2}}}),-\mathit{e}\mathit{x}\mathit{p}(-\frac{{({\mathit{e}}_{\mathit{n}}-\mathbf{1})}^{\mathbf{2}}}{\mathrm{2}{\mathit{\sigma }}^{\mathbf{2}}}))$     8.        ${\mathit{L}}_{\mathit{n}}^{\mathit{R}\mathit{M}\mathit{E}\mathit{E}}$=${\mathit{\phi }}_{\mathbf{0}}{\mathit{u}}_{\mathit{n}}{\mathit{e}}_{\mathit{n}}{}^{\mathbf{2}}+{\mathit{\phi }}_{-\mathbf{1}}{\mathit{v}}_{\mathit{n}}{({\mathit{e}}_{\mathit{n}}+\mathbf{1})}^{\mathbf{2}}+{\mathit{\phi }}_{\mathbf{1}}{\mathit{s}}_{\mathit{n}}{({\mathit{e}}_{\mathit{n}}-\mathbf{1})}^{\mathbf{2}}$     9. end for    10. for k in K:    11.        for t in T:              ${\mathit{L}}_{\left(\mathit{t},\mathit{k}\right)}^{\mathit{P}\mathit{P}\mathit{O}}={\mathit{O}}^{\mathit{P}\mathit{P}\mathit{O}}\left({\mathit{\theta }}_{\mathit{o}\mathit{l}\mathit{d}},\mathit{\theta }, \mathit{t}, \mathit{k}\right)$    12.        end for    13. end for    14. Calculate the total loss:        $\mathit{L}\left(\mathit{e}\right)={\mathit{L}}_{\mathit{p}}\left(\mathit{e}\right)+{\mathit{J}}_{\mathit{k}}\left(\mathit{e}\right)$    15.return $\mathit{L}\left(\mathit{e}\right)$