Algorithm 3.

Experience Replay for Stochastic Environment (ExperienceReplay)

(1)	g ← 0
(2)	Choose a simulated continuous state, ${\underset{\_}{\mathit{x}}}_t$, from the state-space
(3)	while g < K do
(4)	${\underset{\_}{s}}_t\leftarrow \varphi \left({\underset{\_}{x}}_t\right)$
(5)	${\underset{\_}{a}}_t\leftarrow arg{\max }_{a^{\prime }\in A_p}\left((1-\lambda )e^{\mathrm{\Delta }Q\left({\underset{\_}{s}}_t,a^{\prime }\right)}+(\lambda )\kappa \right)$
(6)	$\left(\mathrm{\Delta }{\overline{x}}_{N\underset{\_}{{}_{c_l}}}^a,{\overline{r}}_{N_{\underset{\_}{c_l}}}^a\right)\leftarrow M\left({\underset{\_}{s}}_t,{\underset{\_}{a}}_t\right)$
(7)	${\underset{\_}{x}}_{t+1}\leftarrow {\underset{\_}{x}}_t+\mathrm{\Delta }{\overline{x}}_{N\underset{\_}{c_l}}^a$
(8)	${\underset{\_}{\mathit{s}}}_{t+1}\leftarrow \varphi ({\underset{\_}{\mathit{x}}}_{t+1})$
(9)	Use (1) and (2) to update the Q-value and ΔQ-value
(10)	${\underset{\_}{x}}_t\leftarrow {\underset{\_}{x}}_{t+1}$
(11)	g ← g + 1
(12)	end while