Algorithm 1: DQN-PTR to solve $\mathcal{P}3$

01: Initialize the Q-Network Q with random weights $\theta$ 02: Initialize the Target Q-Network $\widehat{Q}$ with weights ${\theta }^{{}^{\prime }}=\theta$ 03: Initialize replay memory D to capacity N 04: Initialize $ϵ=0,{ϵ}_{increment}=0.0001,{ϵ}_{max}=0.9999$ 05: Initialize optimal trajectory $\mathit{O}$ to empty and maximum total return $R=0$ 06: For $episode=1,\mathcal{M}$ do 07:  Initialize sum reward $R^{{}^{\prime }}=0$ 08:  Initialize state $s_0$, $\mu =rand\in [0,1]$ 09:  For each step t do 10:   If $episode\le 100$ or $\mu >0.9$ then 11:    If $rand\in [0,1]\ge ϵ$ then 12:     Select a random action $a_t$ 13:    else 14:     Set $a_t=arg{max}_{a}Q(s_t,a;\theta )$ 15:    end if 16:    Execute action $a_t$, observe next state $s_{t+1}$ and reward $r_t$ according to environment 1 17:   else 18:    Set $a_t$ according $\mathit{O}\left[t\right]$ 19:    Execute action $a_t$, observe next state $s_{t+1}$ and reward $r_t$ according to environment 2 20:   end if 21:   $R^{{}^{\prime }}=R^{{}^{\prime }}+r_t$ 22:   Store transition $(s_t,a_t,r_t,s_{t+1})$ in D 23:   If episode terminates at step $t+1$ then 24:    If $R^{{}^{\prime }}>R$ then 25:     Replace the trajectory in $\mathit{O}$ with $\{a_1,\dots ,a_t\}$ 26:     $R=R^{{}^{\prime }}$ 27:    else if $R^{{}^{\prime }}=R$ and $t<len\left(\mathit{O}\right)$ then 28:     Replace the trajectory in $\mathit{O}$ with $\{a_1,\dots ,a_t\}$ 29:    end if 30:    break 31:   end if 32:   Sample random mini-batch of transitions $(s_i,a_j,r_j,s_{j+1})$ from D 33:   Set $y_j=\left\{\begin{array}{cc}& r_j,\mathrm{if}\mathrm{episode}\mathrm{terminates}\mathrm{at}\mathrm{step}j+i,\hfill \\ & r_j+\gamma {max}_{a^{{}^{\prime }}}\widehat{Q}(s_{j+1},a^{{}^{\prime }};{\theta }^{{}^{\prime }}),\mathrm{otherwise},\hfill \end{array}\right.$ 34:   Perform a gradient descent step on ${(y_j-Q(s_j,a_j;\theta ))}^2$ with respect to the network parameters $\theta$ 35:   If $ϵ<{ϵ}_{max}$ then 36:    $ϵ=ϵ+{ϵ}_{increment}$ 37:   end if 38:   Every C steps reset $\widehat{Q}=Q$ 39:  end for 40: end for