Algorithm 1. SAC-Based Training Algorithm

1.Initialize the learning rate of each network ${\lambda }_V,{\lambda }_Q,{\lambda }_{\pi }$;

2.Initialize the target network soft update rate $\tau$, entropy regularization weights $\alpha$;

3.Initialize each network parameter $\theta ,\varphi ,\psi ,\stackrel{-}{\psi }$;

4.Initialize the replay buffer;

5. For each episode: (1) Initialize the UAV starting position; (2) Reset various parameters in the interactive environment; (3) Receive initial observation of the image state ${\mathrm{s}}_0$; (4) For each time step $\mathrm{t}=1,2,3\dots$: Take the current state as the input of the current actor network, and generate actions ${\mathrm{a}}_t$; Normalize the actions and convert them into speed control commands; Control the UAV with control commands and observe reward $r_{t+1}$ and observe new image state ${\mathrm{s}}_{t+1}$; Store the piece of experience $(s_t,a_t,s_{t+1},r_{t+1})$ into the replay buffer; Sample replayed experience $(s_b,a_b,s_b^{\prime },r_b)$ from the replay buffer; Update the behavior value network: $\psi \leftarrow \psi -{\lambda }_V{\widehat{\nabla }}_{\psi }{J}_V(\psi )$; Update the Q network: $\varphi \leftarrow \varphi -{\lambda }_Q{\widehat{\nabla }}_{\varphi }{J}_Q(\varphi )$; Update the policy network: $\theta \leftarrow \theta -{\lambda }_{\pi }{\widehat{\nabla }}_{\theta }{J}_{\pi }(\theta )$; Update the target value network: $\overline{\psi }\leftarrow \tau \psi +(1-\tau )\overline{\psi }$; If the terminal condition is satisfied, start a new episode. Or, continue for next time step.  The end of a time step; The end of an episode;