Algorithm 1 . Soft adversarial asynchronous actor-critic intrusion detection (SA3C-ID).

Input: Train Dataset $D_{Train}$, Test Dataset $D_{Test}$, Number of workers $N_{workers}$

Output: Detection results of policy ${\pi }_{\theta }$ on test dataset $D_{Test}$

01: Preprocess train and test datasets

02: Initialize datasets by SABPIO algorithm: $D_{Train}^{\prime }\leftarrow D_{Train}$, $D_{Test}^{\prime }\leftarrow D_{Test}$

03: Initialize global experience replay pool: $B_{globle}^a\leftarrow 0$, $B_{globle}^d\leftarrow 0$

04: Initialize ${\pi }_{\theta }$ (defender policy) and ${\pi }_{\phi }$ (attacker policy) randomly

05: Initialize shared parameters $\theta$ and $\phi$ for ${\pi }_{\theta }$ and ${\pi }_{\phi }$ respectively

06: Initialize worker threads based on $N_{workers}$

07: foreach worker in worker threads to do

08:            for epoch = 1 to epochs do

09:                Initialize the state $s_0$ $\leftarrow$ random sampling from $D_{Train}^{\prime }$

10:                Choose a set of actions Aa,t based on ${\pi }_{\phi }(s_0)$

11:                Initialize local Mini-Batch buffer $M$ $\leftarrow$ random sampling from $D_{Train}^{\prime }$, and labels all belong to $A_{a,t}$

12:                              while not done do

13:                                     Choose action $a_{d,t}^i$ based on ${\pi }_{\theta }(s_t)$ for each state $s_t^i$ in $M$

14:                                     Obtain rewards for the attacker and the defender: $r_{a,t+1}$, $r_{d,t+1}$

15:                                     Choose new set of actions $A_{a,t+1}$ based on ${\pi }_{\phi }(s_t)$

16:                                     Update local Mini-Batch buffer $M$ $\leftarrow$ random sampling from $D_{Train}$, and labels all belong to $A_{a,t+1}$

17:                                     Calculate the priority weight $w_t$ of each attacker experience $⟨s_t,a_{a,t},r_{a,t+1},s_{a,t+1}⟩$and each defender experience $⟨s_t,a_{d,t},r_{d,t+1},s_{d,t+1}⟩$

18:                                     Store attacker experience $B_{globle}^a\leftarrow ⟨s_t,a_{a,t},r_{a,t+1},s_{a,t+1},d_t,w_t⟩$

19:                                     Store defender experience $B_{globle}^d\leftarrow ⟨s_t,a_{d,t},r_{d,t+1},s_{d,t+1},d_t,w_t⟩$

20:                                     Update worker attacker agent by sampling $M$ experience from $B_{globle}^a$ based on their weights $w_t$

21:                                     Update worker defender agent by sampling $M$ experience from $B_{globle}^d$ based on their weights $w_t$

22:                             end while

23:                             Update global attacker agent by worker attacker agent

24:                             Update global defender agent by worker defender agent

25:            end for

26: end foreach

27: Test policy ${\pi }_{\theta }$ on test dataset $D_{Test}^{\prime }$

28: Return Detection results