Algorithm 1 The process noise adaption based on DDPG

1 Initialize the parameters of the actor network and critic network

2 Initialize target networks by copying the actor and critic network

3 Initialize the replay memory buffer $ℛ$

For each episode, perform the following steps

4 Initialize the estimation state and its covariance matrix

For each timestep, perform the following steps

5   Generate an action based on the actor network and the current state $a_k=A^{\mu }(s_k)+{𝒩}_k$, where the random noise is generated by Ornstein-Uhlenbeck process

6   Execute the action, i.e., the compensation factor in the filter to obtain a new state $s_{k+1}$ and a new reward $r_k$

7   Store the sample $(s_k,a_k,s_{k+1},r_k)$ in the buffer $ℛ$

8   Randomly select $N_{Sample}$ samples from the buffer

9   Calculate the temporal difference error ${\delta }_p$ of each sample

${\delta }_p=r_p+\xi Q^{{\omega }^{’}}(s_{p+1},A^{{\mu }^{’}}(s_{p+1}))-Q^{\omega }(s_p,A^{\mu }(s_p))$

10   Calculate the policy gradient

${\nabla }_{\mu }J(A^{\mu })=\frac{1}{N_{Sample}}{\displaystyle \sum _{p=1}^{N_{Sample}}{\nabla }_{\mu }{A}^{\mu }(s_p){\nabla }_{a_t}{Q}^{\omega }(s_p,a_p)}$

11   Update the actor network by Adam optimizer:

${\mu }_{k+1}=f_{Adam}({\nabla }_{\mu }J(A^{\mu }))$

12   Update the critic network

${\nabla }_{\omega }ℒ(\omega )=\frac{1}{N_{Sample}}{\displaystyle \sum _{p=1}^{N_{Sample}}-2{\delta }_p{\nabla }_{\omega }{Q}^{\omega }(s_p,a_p)}$

${\omega }_{k+1}=f_{Adam}({\omega }_t,{\nabla }_{\omega }ℒ(\omega ))$

13   Update the two target networks by soft update

$\mu {’}_{k+1}=\tau {\mu }_{k+1}+(1-\tau )\mu {’}_k$

$\omega {’}_{k+1}=\tau {\omega }_{k+1}+(1-\tau )\omega {’}_k$

End timestep

End episode