Algorithm 1 MAAC learning algorithm process

1: Initialize the communication matrix of all agents $C_0$

2: Initialize the parameters of the agent ${\theta }_{Sender}^i$ and ${\theta }_{Receiver}^i$

3: repeat

4:  Receiver of $Agen{t}^i$: uses attention mechanism to generate communication matrix $\widehat{C_t}$

5:  Sender of $Agen{t}^i$: chooses an action $a_{t+1}^i$ from policy selection network, or randomly chooses   action a (e.g.,$ϵ$-greedy exploration)

6:  Sender of $Agen{t}^i$: generates its own information through the receiver’s communication matrix   $\widehat{C_t}$$c_{t+1}^i$

7:  Collect all the joint actions of Agent and execute the actions $a_{t+1}^1,\dots ,a_{t+1}^N$, get the reward from   the environment $R_{t+1}$ and next state $X_{t+1}$

8:  Update the strategic value function of each Agent:   ${\nabla }_{{\theta }_i}V\left({\theta }_i\right)=\mathbb{E}\left[{\nabla }_{{\theta }_i}log{\pi }_{{\theta }_i}\left(a_i^t|c_i^t\right){\widehat{Q}}_t\left(,a_t^1,\dots ,a_t^N\right)\right]$

9: until End of Round Episode

10: returns ${\theta }_{Sender}^i$ and ${\theta }_{Receiver}^i$ for each Agent