Algorithm 1. Training algorithm using the MADDPG framework.

Parameters: batch size $\beta$, training episodes. $M$, training step $T$, action noise $\mathcal{N}$ and the number of USVs $N$, actor networks’ weights for each agent ${\theta }_i^{{\mu }^{’}}$

1: For episode = 1 to $M$ do

2:  Initialize observations $O_{\mathrm{init} }$, $O_{\mathrm{new} }\leftarrow O_{\mathrm{init} }$.

Initialize: Actor network $\mu$, critic network $\mathrm{Q}$ with weights ${\theta }^{\mu }$, ${\theta }^Q$.

Initialize: Target actor, critic network: ${\mu }^{\prime }$, $Q^{\prime }$, with weights ${\theta }^{{\mu }^{\prime }}\leftarrow {\theta }^{\mu }$, ${\theta }^{Q^{\prime }}\leftarrow {\theta }^Q$.

Initialize: Replay buffer $D$ with capacity $C$, exploration counter. $\mathrm{Counter} =0$

3:  for step $t=1$ to $T$ do

4:   if $\mathrm{Counter}< \mathrm{C}$ then

5:    each USV $i$ randomly chooses $a_i$;

6:

7:    else

8:     $a_i={\mu }_{{\theta }_i}\left(o_{\mathrm{new} }^i\right)+\mathcal{N}$

9:    end if

10:    Execute actions $a=\left[a_1,a_2,\dots a_N\right]$, and observe reward $R$, new states $o_{new}$;

11:    Store transition $\left(o,a,r,o_{\mathrm{new} }\right)$ into experience replay buffer $D$;

12:    Sample a mini-batch of $\beta$ transitions $\left(o^m,a^m,r^m,o_{\mathrm{new}}^m\right)$ from replay. Buffer $D$;

13:    Set $y^m=R^m+{\left.\gamma Q_i^{{\mu }^{\prime }}\left(o_{\mathrm{new}}^m,a_1^{\prime },\dots ,a_N^{\prime }\right)\right|}_{a_i^{\prime }={\mu }_i^{\prime }\left(o_i\right)}$

14:    Update critic by minimizing the loss

$L\left({\theta }_i\right)=\frac{1}{S}{\displaystyle \sum _{m=1}^S{\left(y^m-Q_i^{\mu }\left(o_{\mathrm{new} }^m,a_1,\dots ,a_N\right)\right)}^2}$

15:    Update actor using the sampled policy gradient:

${\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em} \nabla }_{{\theta }_i}J=\frac{1}{S}{\displaystyle \sum _{m=1}^S{\left.{\nabla }_{{\theta }_i}{\mu }_{{\theta }_i}\left(o_i^m\right){\nabla }_{a_i}{Q}_i^{\mu }\left(o^m,a_1,a_2,\dots ,a_N\right)\right|}_{a_i={\mu }_{{\theta }_i}\left(o_i^m\right)}}$

16:    Updating actor networks              Updating critic networks              Update target networks with updating rate $\tau$:             ${\theta }_i^{{\mu }^{\prime }}\leftarrow {\theta }_i^{\mu }+(1-\tau ){\theta }_i^{{\mu }^{\prime }}$             ${\theta }_i^{Q^{\prime }}\leftarrow {\theta }_i^Q+(1-\tau ){\theta }_i^{Q^{\prime }}$

17:  end for

18: end for