Algorithm 2: Pseudocode for distributed Q-learning

Initialization:

for each $s_u^t\in S_u^t, a_u^t\in A_u^t$ do

initialize Q-table and policy ${\pi }_u^{\ast }\left(s_u^t\right)$

end for

Learning:

loop

estimate state $s_u^t$

generate a random real number x$\in \left[0,1\right]$

if $x<\epsilon$ // for exploration

elect action $a_u^t$ randomly

else

select action $a_u^t$ according to ${\pi }_u^{\ast }\left(s_u^t\right)$

receive action $a_{BBU}^t$ from algorithm1

determine action $a_u^{\ast }$ by comparing $a_u^t$ and $a_{BBU}^t$

execute action $a_u^t$

calculate reward $r_u^t$

update Q-value $Q\left(s_u^t, a_u^t\right)$ and ${\pi }_u^{\ast }\left(s_u^t\right)$

end loop