Algorithm 2 Learning process, out-of-policy for mutating DoS cyber-attacks frames

Require:$Q(s_t,A_t)$, $\forall s_t\in S,\forall A_t\in A$ arbitrarilty, and $Q(terminalstate,·)=0$     for each network frame do     Sort the start of the protocol conversation     Build an action space S for each frame         for each $s_t$ do     Initialize agent a with sates s at time $t+1$             for each $s_{t+1}$ do           Choose A from S using $\mathrm{\Lambda }$ derived from Q           Take action $A_t$, observe R, $s_{t+1}$           $Q(s_t,A_t)\leftarrow Q(s_t,A_t)+\alpha [R(s_{t+1},A_{t+1})+\mathrm{\Phi }\mathbf{max}Q(s_{t+1}+A_{t+1})-Q(s_t,A_t)]$           $s_t\leftarrow s_{t+1}$             end for     until $s_t$ is terminal, hence the DoS cyber-attack frame is fully mutated.     end for end for