Algorithm 1 DDPG algorithm

Randomly initialize critic network $Q(s,a|{\theta }^Q)$ and actor $\mu \left(s|{\theta }^{\mu }\right)$ with weights ${\theta }^{\theta }$ and ${\theta }^{\mu }$ Initialize target network $Q^{\prime }$ and ${\mu }^{\prime }$ with weights ${\theta }^{Q^{\prime }}\leftarrow {\theta }^Q$, ${\theta }^{{\mu }^{\prime }}\leftarrow {\theta }^{\mu }$ Initialize replay buffer R for episode = 1, M do    Initialize a random process N for action exploration    Receive initial observation state $s_1$    for t = 1, T do      Select action $a_t=\mu \left(s_t|{\theta }^{\mu }\right)+$ Nt      according to the current policy and exploration noise      Execute action $a_t$ and observe reward $r_t$ and observe new state $s_{t+1}$      Store transition $<s_t,a_t,r_t,s_{t+1}>$ in R      Sample a random minibatch of N transitions $<s_i,a_i,r_i,s_{i+1}>$ from R      Set $y_i=r_i+\gamma Q^{\prime }(s_{i+1},{\mu }^{\prime }(s_{i+1}|{\theta }^{{\mu }^{\prime }})|{\theta }^{Q^{\prime }})$      Update critic by minimizing the loss: $L=\frac{1}{N}{\sum }_i\left(y_i-Q{\left(s_i,a_i|{\theta }^Q\right)}^2\right)$      Update the actor policy using the sampled gradient:          ${\nabla }_{{\theta }_{o_{\mu }}}\mu {|}_{s_i}\approx \frac{1}{N}{\sum }_i{\nabla }_{a}Q\left(s,a|{\theta }^Q\right){|}_{s=s_i,a=\mu \left(s_i\right)}{\nabla }_{{\theta }^{\mu }}\mu \left({\mathrm{s}|\theta }^{\mu }\right){|}_{s=s_i}$     Update the target networks:                  ${\theta }^{Q^{\prime }}\leftarrow \tau {\theta }^Q+\left(1-\tau \right){\theta }^{Q^{\prime }}$                  ${\theta }^{{\mu }^{\prime }}\leftarrow \tau {\theta }^{\mu }+\left(1-\tau \right){\theta }^{{\mu }^{\prime }}$ end for end for