Algorithm 1: CR-DRQN pseudocode

Initialize replay memory D with capacity N Initialize online Q network with parameters $\omega$ randomly Initialize target Q network with parameters ${\omega }^{-}=\omega$ For episode =1:M do  Initialize observed state $o_1=O(s_1)$  For t =1:T do   With probability $\epsilon$ select random action $a_t$, otherwise select $a_t{=\mathrm{argmax}}_{a}Q(\varphi (o_t),a;\omega )$   Execute action $a_t$ in emulator and get reward $r_{t+1}$ and next observed state $o_{t+1}$   Store transition $(o_t,a_t,r_{t+1},o_{t+1})$ in D   Set $y_j=\{\begin{array}{c}{r}_{j+1},\mathrm{if}\mathrm{episode}\mathrm{terminates}\mathrm{at}\mathrm{step}\mathrm{j}+1\\ r_{j+1}+\gamma {\max }_{a^{\prime }}{Q}^{\prime }({\varphi }_{j+1},a^{\prime };{\omega }^{-}),otherwise\end{array}$   Update network parameters $\omega$ by using the gradient descent of ${(y_j-Q({\varphi }_j,a_j;\omega ))}^2$   Every C steps reset $Q^{\prime }=Q$  End for End for