Algorithm 1 Learning process, out-of-policy

Require:$Q(s_t,A_t)$, $\forall s_t\in S,\forall A_t\in A$ arbitrarilty, and $Q(terminalstate,·)=0$     for each $s_t$ do     Initialize agent a with sates s at time $t+1$         for each $s_{t+1}$ do           Choose A from S using $\mathrm{\Lambda }$ derived from Q           Take action $A_t$, observe R, $s_{t+1}$           $Q(s_t,A_t)\leftarrow Q(s_t,A_t)+\alpha [R(s_{t+1},A_{t+1})+\mathrm{\Phi }\mathbf{max}Q(s_{t+1}+A_{t+1})-Q(s_t,A_t)]$           $s_t\leftarrow s_{t+1}$         end for     until $s_t$ is terminal, hence the PE is fully mutated     end for