Algorithm 1: MDRLAT

1: Initialize experience replay buffer $D$ to capacity $N$, parameters of online network $\theta$, parameters of target network ${\theta }^{-}=\theta$, frequency $N_t$ to update target network

2: for $t=1, 2,\dots , T$ do

3:   Obtain $s_k=\left[l_k,d_k\right]$ from the environment, with probability $\epsilon$ select a random action $a_k$, otherwise select $a_k=\underset{a}{\mathrm{argmax}}Q\left(s_k,a;\theta \right)$

4:   Execute action $a_k$, transit to next state $s_{k+1}$ and receive a reward $r_k$

5:   Store transition $\left(s_k,a_k,r_k,s_{k+1}\right)$ in $D$

6:   Randomly sample a batch of $N_B$ transition $\left(s_t,a_t,r_t,s_{t+1}\right)$ from $D$

7:   if episode terminates at step $t+1$ then

8:    Set $y_t=r_t$

9:   else

10:    Set $y_t=r_t+\gamma Q(s_{t+1},\underset{a_{t+1}}{\mathrm{argmax}}Q\left(s_{t+1},a_{t+1};\theta \right);{\theta }^{-})$

11:   end if

12:   Compute loss ${\mathcal{L}}_{DRL}={\left(y_t-Q\left(s_t,a_t;\theta \right)\right)}^2$

13:   Feed $s_t$ into the online network to get estimated velocity $\tilde{v}$, $\tilde{\omega }$

14:   Compute loss ${\mathcal{L}}_{AT}={\displaystyle \sum }_{i=t-2}^t\left({\left({\overline{v}}_i-{\tilde{v}}_i\right)}^2+{\left({\overline{\omega }}_i-{\tilde{\omega }}_i\right)}^2\right)$

15:   Update the parameters of the online network $\theta$

16:   if $t$ mod $N_t=0$ then

17:    Update the parameters of the target network ${\theta }^{-}\leftarrow \theta$

18:   end if

19: end for