Algorithm 2 Adversarial attack trick and adversarial learning MADDPG (A2-MADDPG) algorithm.

1: for N agents, randomly initialize their critic network $Q_i\left(s_i,a_i;{\theta }_i^Q\right)$ and actor network ${\mu }_i\left(s_i|{\theta }_i^{\mu }\right)$

2: synchronize target networks $Q_i’\left(s_i,a_i;{\theta }_i^{Q’}\right)$ and ${\mu }_i’\left(s_i|{\theta }_i^{\mu ’}\right)$ with ${\theta }_i^{Q’}\leftarrow {\theta }_i^Q,{\theta }_i^{\mu ’}\leftarrow {\theta }_i^{\mu }$

3: initialize hyperparameters: experience buffer $\mathcal{D}$, mini-batch size m, max episode M, max step T, actor learning rate la, critic learning rate lc, discount factor γ, soft update rate τ

4: for episode = 1, M do

5:  reset environment, and receive the initial state x

6:  initialize exploration noise of action ${\mathbb{N}}_{\mathit{action}}$

7:  for t = 1, T do

8:  $s_{i\left(\mathit{noise}\right)}\leftarrow StateAttack\left(Q_i^{\mu ’}\left(x,a_1,a_2,\dots ,a_n\right),{\mu }_i\left(s_i|{\theta }_i^{\mu }\right),s_i\right)$

9:   for each agent i, select action $a_i=\mu \left(s_{i\left(\mathit{noise}\right)};{\theta }_i^{\mu }\right)+{\mathbb{N}}_{\mathit{action}}$

10:   execute $a_1,a_2,\dots ,a_n$, rewards $r_1,\dots ,r_n$, and next state x’

11:   store sample $\left(x,a_1,a_2,\dots ,a_n,r_1,\dots ,r_n,x’\right)$ in $\mathcal{D}$

12:   for agenti = 1, n do

13:    randomly extract m samples $\left(x^k,a^k,r^k,{x’}^k,\right)$

14:    update critic network by Equation (27)

15:    update actor network by: $${\nabla }_{{\theta }_i^{\mu }}J=m^{-1}{\sum }_j\left[{\nabla }_{{\theta }_i^{\mu }}{\mu }_i\left(a_i\left|s_i\right.\right){\nabla }_{a_i}{Q}_i^{\mu }\left(\mathrm{x},a_1\dots a_i\dots a_N;{\theta }_i^Q\right)\left|{}_{a_i={\mu }_i\left(s_i\right),a_{j\ne i}^{*}=a_j^k+{\tilde{\eta }}_j}\right.\right]$$

16:   end for

17:   update target networks by ${\theta }_i^{Q’}\leftarrow \tau {\theta }_i^Q+(1-\tau ){\theta }_i^{Q’},{\theta }_i^{\mu ’}\tau \leftarrow {\theta }_i^{\mu }+(1-\tau ){\theta }_i^{\mu ’}$

18:  end for

19: end for