Algorithm 1 Actor-dueling-critic algorithm

1: Initialize:  Initialize actor $\mu \left(s\right|{\theta }^{\mu })$ and dueling-critic $Q(s,a|{\theta }^Q,\alpha ,\beta )$  Initialize target actor ${\mu }^{\prime }$ with ${\theta }^{{\mu }^{\prime }}={\theta }^{\mu }$ and target dueling-critic $Q^{\prime }$ with ${\theta }^{Q^{\prime }}={\theta }^Q,{\alpha }^{\prime }=\alpha ,{\beta }^{\prime }=\beta$  Initialize replay memory $R=\varnothing$, random process $\mathcal{N}$.  Uniformly separate the action space to n intervals ($Z=\left\{z_1,z_2,\dots ,z_n\right\}$). 2: for episode=1 to M do 3:   Receive initial state $s_1$ 4:   for t=1 to N do 5:    With probability $ϵ$ select action $a_t=\mu \left(s_t\right|{\theta }^{\mu })+{\mathcal{N}}_t$, otherwise select $a_t=\mu \left(s_t\right|{\theta }^{\mu })$ 6:    Execute $a_t$ and observe reward $r_t$ and new state $s_{t+1}$ 7:    Store transition $(s_t,a_t,r_t,s_{t+1})$ in R 8:    Sample a random minibatch of N transitions $(s_i,a_i,r_i,s_{i+1})$ from R 9:    Implement target actor $a_{i+1}^{\prime }={\mu }^{\prime }\left(s_{i+1}\right|{\theta }^{{\mu }^{\prime }})$ 10:    Implement dueling-critic $Q_{i+1}^{\prime }=Q^{\prime }(s_{i+1},a_{i+1}^{\prime }|{\theta }^{Q^{\prime }},{\alpha }^{\prime },{\beta }^{\prime })$ (Equation (14)) with $a_{i+1}^{\prime }\in z_j$ 11:    Set $y_i=r_i+\gamma Q_{i+1}^{\prime }$ (set $y_i=r_i$ if $s_{t+1}$ is terminal) 12:    Update dueling-critic by minimizing the loss:          $L=\frac{1}{N}{\sum }_i{(y_i-Q(s_i,a_i|{\theta }^Q,\alpha ,\beta ))}^2$ 13:    Update actor using the sampled PG:          ${\nabla }_{{\theta }^{\mu }}J\approx \frac{1}{N}{\sum }_i{\nabla }_{a}Q(s,a|{\theta }^Q){|}_{s=s_i,a=\mu \left(s_i\right)}{\nabla }_{{\theta }^{\mu }}\mu \left(s\right|{\theta }^{\mu }){\left)\right|}_{s_i}$ 14:    Soft update target networks of dueling-critic and actor ($\tau \ll 1$):          $\begin{array}{cc}{\theta }^{{\mu }^{\prime }}\leftarrow \tau {\theta }^{\mu }+(1-\tau ){\theta }^{{\mu }^{\prime }}& {\theta }^{Q^{\prime }}\leftarrow \tau {\theta }^Q+(1-\tau ){\theta }^{Q^{\prime }}\end{array}$          $\begin{array}{cc}{\alpha }^{\prime }\leftarrow \tau \alpha +(1-\tau ){\alpha }^{\prime }& {\beta }^{\prime }\leftarrow \tau \beta +(1-\tau ){\beta }^{\prime }\end{array}$ 15:   end for 16: end for