Algorithm 1 DQN training procedure.

Input:  Total training episodes ${\Gamma }_{max}$, experience pool capacity D, target network update frequency F, training batch size B, attenuation coefficient $\gamma$, greedy value $\epsilon$ and maximum total penalty $R_{min}$.

Output:  Target neural network $\widehat{Q}$ with $\widehat{\omega }$.

1: Initialization: set initial episode $\Gamma =0$, initial state $s_t$, total reward of each episode $R=0$, value neural network Q with random weight $\omega$, target network $\widehat{Q}$ with $\widehat{\omega }=\omega$.

2: while $\Gamma <{\Gamma }_{max}$ do

3: while $∥(x,y)-(x_d,y_d)∥⩾ϵ$

$R⩾R_{min}$ do

4:  Select an action through $a_t=argmaxQ(s_t,a_t;\omega )$.

5:  Execute $a_t$ and obtain $r_t$ and $s_{t+1}$.

6:  Store $(s_t,a_t,r_t,s_{t+1})$ into D.

7:  Randomly select B-size data from D, and perform $$y=\left\{\begin{array}{cc}{r}_t,\hfill & \Gamma ={\Gamma }_{max},\hfill \\ r_t+\gamma max\widehat{Q}(s_{t+1},a_{t+1};\widehat{\omega }),\hfill & \mathrm{else}.\hfill \end{array}\right.$$ (11)

8:  Compute mean square error loss through $$loss={(y-Q(s_t,a_t;\omega ))}^2.$$ (12)

9:  Utilize gradient descent algorithm to update $\omega$.

10:  After F steps, perform $Q=\widehat{Q}$.

11:  Perform $R=R+r_t$.

12: end while

13: Perform $\Gamma =\Gamma +1$ and $R=0$.

14: end while

15: return target neural network $\widehat{Q}$ with $\widehat{\omega }$.