Algorithm 1: MEP–DDPG Algorithm

Randomly initialize critic network $Q\left(s,a|{\theta }^Q\right)$ and actor $\mu \left(s|{\theta }^{\mu }\right)$ with weights ${\theta }^Q$ and ${\theta }^{\mu }$

Initialize target network $Q^{\prime }$ and ${\mu }^{\prime }$ with weights ${\theta }^{Q^{\prime }}\leftarrow {\theta }^Q$, ${\theta }^{{\mu }^{\prime }}\leftarrow {\theta }^{\mu }$

Initialize experience pools $R_1,R_2\cdots R_X$

for episode = 1, $M$ do

Initialize a random process $\mathcal{N}$ for action exploration

Receive initial observation state $s_1$

for$t=1,T$do

According to probability $\eta$, select action $a_t=\mu \left(s_t|{\theta }^{\mu }\right)+{\mathcal{N}}_t$ based on current policy and exploration noise or select action $a_t$ by MPC-SA

Execute action $a_t$ and observe reward $r_t$ and observe new state $s_{t+1}$

Store experience $\left(s_t,a_t,r_t,s_{t+1}\right)$ in specific $R$ depending on the source of $a_t$.

Follow a specific sampling policy, sample a random minibatch of $N$ experiences $\left(s_i,a_i,r_i,s_{i+1}\right)$ from $R_1,R_2\cdots R_X$

Set $$y_i=r_i+\gamma Q^{\prime }\left(s_{i+1},{\mu }^{\prime }\left(s_{i+1}|{\theta }^{{\mu }^{\prime }}\right)|{\theta }^{Q^{\prime }}\right)$$

Update critic by minimizing the loss: $$L=\frac{1}{N}{\displaystyle \sum }_i{\left(y_i-Q\left(s_i,a_i|{\theta }^Q\right)\right)}^2$$

Update the actor policy using the sampled policy gradient: $${\nabla }_{{\theta }^{\mu }}J\approx \frac{1}{N}{\displaystyle \sum }_i{\nabla }_{a}Q\left(s,a|{\theta }^Q\right){|}_{s=s_i,a=\mu \left(s_i\right)}{\nabla }_{{\theta }^{\mu }}\mu \left(s|{\theta }^{\mu }\right){|}_{s_i}$$

Update the target networks: $${\theta }^{Q^{\prime }}\leftarrow \tau {\theta }^Q+\left(1-\tau \right){\theta }^{Q^{\prime }}$$ $${\theta }^{{\mu }^{\prime }}\leftarrow \tau {\theta }^{\mu }+\left(1-\tau \right){\theta }^{{\mu }^{\prime }}$$

end for

end for