Algorithm 1 DIDDPG-based UAV formation algorithm.

1:Initialize system parameters $\overline{P},F_j,H_j,D_j^1,D_j^2$ and the replay memory buffer R.   2:Randomly initialize ${\theta }^{\mu }$, ${\theta }^Q$, ${\mu }^{\prime }$ and $Q^{\prime }$.   3:Initialize online actor and critic networks $\mu \left(s\left|{\theta }^{\mu }\right.\right)$ and $Q\left(s,a\left|{\theta }^Q\right.\right)$, respectively.   4:for episode = $0:1:N-1$ do   5:   Initialize the random noise $\omega$ and state $s_0$.   6:   for $j=0:1:M-1$ do   7:     Update the action $a_j=\mu \left(s_j\left|{\theta }^{\mu }\right.\right)+\omega$.   8:     Update the next state $s_{j+1}$ based on (11) that $s_{j+1}=s_j{F}_j+a_j{H}_j+\left[G_j^T,0_{3\times 1}\right]$.   9:     Derive the reward $r_j$ by (12) that $r_j=-s_j\overline{P}{s}_j^T-a_{j+1}Q{a}_{j+1}^T$. 10:     Store transition $(s_j,a_j,r_j,s_{j+1})$ in R. 11:     Randomly Select a mini-batch of K experience samples $(s_j,a_j,r_j,s_{j+1})$ from R. 12:     Update target Q value based on (15) that $y_j=r_j+\gamma Q^{\prime }\left(s_{j+1},{\mu }^{\prime }\left(s_{j+1}|{\theta }^{{\mu }^{\prime }}\right)|{\theta }^{Q^{\prime }}\right)$. 13:     Update ${\theta }^Q$ by minimizing the mean quadratic error function based on (14). 14:     Update ${\theta }^{\mu }$ by sampled policy gradient ${\nabla }_{{\theta }^{\mu }}J$ given by (16). 15:     Update the target networks: 16:     $\begin{array}{l}{\theta }^{Q^{\prime }}\leftarrow \delta {\theta }^Q+\left(1-\delta \right){\theta }^{Q^{\prime }},\\ {\theta }^{{\mu }^{\prime }}\leftarrow \delta {\theta }^{\mu }+\left(1-\delta \right){\theta }^{{\mu }^{\prime }}.\end{array}$ 17:   end for 18:end for