Algorithm 1: Proposed SAC-based path planning algorithm for multi-arm manipulator.

1:Define MAMMDP and the augmented state $q_t$ and the state and goal state $q_{\mathrm{init}}$ and $q_{\mathrm{goal}}$ 2:Initialize network parameters $\psi ,{\theta }_{1,2},\varphi$ 3:Initialize the parameter values of the target network $\overline{\psi }\leftarrow \psi$ 4:Initialize global replay memory $\mathcal{D}$ 5:   6:for$e=1$ to Mdo 7:    Initialize local buffer $\mathcal{L}$                ▹ Memory for an episode      8:    for $t=0$ to $T-1$ do 9:        Randomly choose the goal and initial positions $q_{\mathrm{goal}},q_{\mathrm{init}}\in {\mathbf{Q}}_{\mathrm{free}}^a$ 10:        $a_t=f_{\varphi }({ϵ}_t,q_t\left|\right|q_{\mathrm{goal}}),{ϵ}_t\sim \mathcal{N}(0,{\sigma }_t)$ 11:        ${\widehat{q}}_{t+1}=q_t+\alpha ·a_t+{ϵ}_e,{ϵ}_e\sim \mathcal{N}(0,{\sigma }_e)$ 12:   13:        if ${\widehat{q}}_{t+1}\in {\mathbf{Q}}_{\mathrm{free}}^a$then                ▹ Get next state and reward 14:           $q_{t+1}\leftarrow {\widehat{q}}_{t+1}$ 15:           $r_{t+1}=-1$ 16:        else if ${\widehat{q}}_{t+1}\in {\mathbf{Q}}_{\mathrm{collide}}^a$ then 17:           $q_{t+1}\leftarrow q_t$ 18:           $r_{t+1}=-1$ 19:        else if $|q_{t+1}-q_{\mathrm{goal}}|\le \eta ·\alpha$ then 20:           $r_{t+1}=0$ 21:           Terminate due to goal arrival 22:        end if 23:   24:        Store the transition $(q_t||q_{\mathrm{goal}},a_t,r_{t+1},q_{t+1}||q_{\mathrm{goal}})$ in $\mathcal{D},\mathcal{L}$ 25:                              ▹ Parameters update 26:        Sample mini-batch of m transitions $(q_l||q_{\mathrm{goal}},a_l,r_{l+1},q_{l+1}||q_{\mathrm{goal}})$ from $\mathcal{D}$ 27:        $J_V(\psi )={\mathbb{E}}_{q_l}[\frac{1}{2}(V_{\psi }(q_l\left|\right|q_{\mathrm{goal}})-{\mathbb{E}}_{a_l}[{min}_{k=1,2}{Q}_{{\theta }_k}(q_l\left|\right|q_{\mathrm{goal}},a_l)-\beta log{\pi }_{\varphi }(a_l|q_l\left|\right|q_{\mathrm{goal}}{)])}^2]$ 28:        $J_Q({\theta }_{k=1,2})={\mathbb{E}}_{q_l,a_l}[\frac{1}{2}(Q_{{\theta }_{k=1,2}}(q_l\left|\right|q_{\mathrm{goal}},a_l)-(r_{l+1}+V_{\overline{\psi }}(q_{l+1}\left|\right|q_{\mathrm{goal}}{)))}^2]$ 29:        $J_{\pi }(\varphi )={\mathbb{E}}_{q_l,a_l}[\beta log{\pi }_{\varphi }(a_l|q_l\left|\right|q_{\mathrm{goal}})-{min}_{k=1,2}{Q}_{{\theta }_k}(q_l\left|\right|q_{\mathrm{goal}},a_l)]$ 30:   31:        Each network parameters $\psi ,{\theta }_{1,2},\varphi$ are updated by gradient descent 32:        using ${\nabla }_{\psi }{J}_V(\psi ),{\nabla }_{{\theta }_1}{J}_Q({\theta }_1),{\nabla }_{{\theta }_2}{J}_Q({\theta }_2),{\nabla }_{\varphi }{J}_{\pi }(\varphi )$ 33:   34:        Update state value target $\overline{\psi }\leftarrow \tau \psi +(1-\tau )\overline{\psi }$ 35:    end for 36:   37:    if $q_T\ne q_{\mathrm{goal}}$then                      ▹ HER 38:        Set additional goal $q_{\mathrm{goal}}^{\prime }\in \{q_1,q_2,\cdots ,q_T\}$ 39:        for $t=0$ to $T-1$ do 40:           Sample a transition $(q_t||q_{\mathrm{goal}},a_t,r_t,q_{t+1}||q_{\mathrm{goal}})$ from $\mathcal{L}$ 41:           if $|q_{t+1}-q_{\mathrm{goal}}^{\prime }|\le \eta ·\alpha$ then 42:               $r_{t+1}^{\prime }=0$ 43:           else  $r_{t+1}^{\prime }=-1$ 44:           end if 45:           Store the transition $(q_t||q_{\mathrm{goal}}^{\prime },a_t,r_{t+1}^{\prime },q_{t+1}||q_{\mathrm{goal}}^{\prime })$ in $\mathcal{D}$ 46:        end for 47:    end if 48:end for