Algorithm 2 Dueling DQN Algorithm for Lower Tier Bandwidth Allocation
1:	Initialize the action-value function $Q$, target action-value function $\widehat{Q}$ the replay memory D to capacity N
2:	Each MVNO receives a bandwidth $c_{m }$ from the BS;
3:	Each MVNO creates an action space $A_l$;
4:	Form = 1 to M, do
5:	MVNO randomly chooses an action $a\in A_l$ and performs $a$;
6:	MVNO allocates the bandwidth $c_j^m$ to users which are connected with it;
7:	Calculate the $q_j^m$ as state $s$;
8:	For t = 1, to T, do
9:	The agent gets the current state $s$;
10:	Choose an action $a\in A_l$ according to the policy of Dueling DQN;
11:	Calculate the total system utility $f_m$ according to (15);
12:	Calculate the total reward;
13:	The agent allocates the bandwidth to users and calculates the state after the selection action of this iteration as $s^{\prime }$;
14:	#Train Dueling DQN
15:	The agent i.e., each MVNO inputs $\left(s,a, s^{\prime },r\right)$ into the Dueling DQN;
16:	The agent store transition $\left(s,a, s^{\prime },r\right)$ in D;
17:	The agent sample random minibatch of transitions $\left(s\_,a\_, s^{\prime }\_,r\_\right)$ from D;
18:	Set $$y\_=\{\begin{array}{c}{r}_{-}\\ r\_+\gamma ma{x}_{a^{*}}\widehat{Q}\left(s^{\prime }\_,a^{*};{\theta }^{-},\alpha ,\beta \right)\end{array}\begin{array}{c}\mathrm{if} \mathrm{episode} \mathrm{terminates} \mathrm{at} \mathrm{step} \_+1\\ \mathrm{otherwise}\end{array}$$
19:	The agent perform a gradient descent step on ${\left(y\_-Q\left(s\_,a\_;\theta ,\alpha ,\beta \right)\right)}^2$ with respect to the network parameters θ, $\alpha$ and $\beta$;
20:	Every steps reset $\widehat{Q}=Q$;
21:	End for
22:	End for