Algorithm 1 Q-Learning for ${\mathcal{V}}_R$, ${\mathcal{V}}_I$, and ${\mathcal{V}}_E$ tests.

1:Initialization:   2:Initialize the Q-table $\mathcal{Q}(s,a)$ with zeros for all state-action pairs $(s,a)$   3:Set the learning rate $\alpha$, discount factor $\gamma$, and the parameter $ϵ$ for the policy ${\varphi }_{ϵ}$   4:for each episode do   5:    Initialize the state s with the initial configuration of $\mathcal{D}$. In the initial state s, $\mathcal{A}$ performs a reconnaissance process of all available virtual machines   6:    while the state s is not terminal do   7:        Select the action a based on the policy ${\varphi }_{ϵ}$   8:        Execute action a, observe reward r and the new state $s^{\prime }$   9:        if a pertains to ${\mathcal{V}}_R$ then 10:           Perform reconnaissance using Nmap 11:        else if a pertains to Vulnerability ${\mathcal{V}}_I$ then 12:           Conduct vulnerability identification using Nmap Vulners 13:        else if a pertains to ${\mathcal{V}}_E$ then 14:           Conduct exploitation using Metasploit 15:        end if 16:        Select $a^{\prime }$ as the action that maximizes $\mathcal{Q}(s^{\prime },a^{\prime })$ 17:         Update $\mathcal{Q}(s,a)$ using the Bellman equation: $$\mathcal{Q}(s,a)\leftarrow \mathcal{Q}(s,a)+\alpha \left[r+\gamma \mathcal{Q}(s^{\prime },a^{\prime })-\mathcal{Q}(s,a)\right]$$ 18:        Update the state $s\leftarrow s^{\prime }$ 19:    end while 20:end for