Algorithm 2 DDPG with HER and Adaptive Exploration

1:Initialize online policy network ${\mu }_{\theta }$ with weights $\theta$   2:Initialize target policy network ${\mu }_{{\theta }^{\prime }}$ with weights ${\theta }^{\prime }\leftarrow \theta$   3:Initialize online Q network $Q_{\varphi }$ with weights $\varphi$   4:Initialize target Q network $Q_{{\varphi }^{\prime }}$ with weights ${\varphi }^{\prime }\leftarrow \varphi$   5:Initialize experience replay pool $\mathcal{D}$ to capacity N   6:for episode = 1, M do   7:    Receive initial observation state $s_1$   8:    for t = 1, T do   9:       Obtain $\mu$ from the adaptive exploration adjustment unit 10:     Calculate exploration noise 11:     Select action $a_t={\mu }_{\theta }\left(s_t\right)+\mathrm{exploration}\mathrm{noise}$ 12:     Execute action $a_t$ in the environment 13:     Observe reward $r_t$ and new state $s_{t+1}$ 14:     Store transition $(s_t,a_t,r_t,s_{t+1})$ in $\mathcal{D}$ 15:     Store modified transitions with alternative goals 16:     Sample random mini-batch of K transitions $(s_i,a_i,r_i,s_{i+1})$ from $\mathcal{D}$ 17:     Set $y_i=r_i+\gamma Q_{{\varphi }^{\prime }}(s_{i+1},{\mu }_{{\theta }^{\prime }}\left(s_{i+1}\right))$ 18:     Update $\varphi$ by minimizing the loss: $L=\frac{1}{K}{\sum }_i{(y_i-Q_{\varphi }(s_i,a_i))}^2$ 19:      Update $\theta$ using the sampled policy gradient: $${\nabla }_{\theta }J\approx \frac{1}{K}\sum _i{\nabla }_a{Q}_{\varphi }(s,a){\left|s=s_i,a=\mu \theta \left(s_i\right){\nabla }_{\theta }{\mu }_{\theta }\left(s\right)\right|}_{s_i}$$ 20:      Update the target networks: $$\begin{array}{ccc}\hfill {\theta }^{\prime }& \leftarrow & \tau \theta +(1-\tau ){\theta }^{\prime }\hfill \\ {\varphi }^{\prime }& \leftarrow & \tau \varphi +(1-\tau ){\varphi }^{\prime }\end{array}$$ 21:  end for 22:end for