Algorithm 1 The Q-learning based resource allocation algorithm

initialize the table entry $\mathrm{Q}\left(\mathrm{s},\mathrm{a}\right)$ arbitrarily for each state-action pair $\left(\mathrm{s},\mathrm{a}\right)$ observe the current state $\mathrm{s}$, initialize the value of $\mathsf{\alpha }$ and $\mathsf{\gamma }$ forepisode = 1 to Mdo   from the current state-action pair $\left(\mathrm{s},\mathrm{a}\right)$, execute action $\mathrm{a}$ and obtain   the immediate reward $r$ and a new state $s^{\prime }$   select an action $a^{\prime }$ based on the state $s^{\prime }$ and update the table entry for   $\mathrm{Q}\left(\mathrm{s},\mathrm{a}\right)$ as expressed in Equation (18)   replace $\mathrm{s}\leftarrow s^{\prime }$ end for Output:${\pi }^{\ast }\left(s\right)=argma{x}_a{Q}^{\ast }\left(s,a\right)$