Algorithm 1: Pseudocode for centralized Q-learning

Initialization:

for each $s_{BBU}^t\in S_{BBU}^t, a_{BBU}^t\in A_{BBU}^t$ do

initialize Q-table and policy ${\pi }_{BBU}^{\ast }\left(s_{BBU}^t\right)$

end for

Learning:

loop

estimate the state $s_{BBU}^t$

generate a random real number x$\in \left[0,1\right]$

if $x<\epsilon$ // for exploration

select the action $a_{BBU}^t$ randomly

else

select the action $a_{BBU}^t$ according to ${\pi }_{BBU}^{\ast }\left(s_{BBU}^t\right)$

recommend action $a_{BBU}^t$ to the devices in the cell

calculate reward $r_{BBU}^t$

update Q-value $Q\left(s_{BBU}^t, a_{BBU}^t\right)$ and ${\pi }_{BBU}^{\ast }\left(s_{BBU}^t\right)$

end loop