Algorithm 1 DQN and Joint Bidding Algorithm for Upper Tier Bandwidth Allocation
1:	Initialize the Bidding pool B of MVNO and corresponding lower tier action selection table A;
2:	Initialize the action-value function $Q$, target action-value function $\widehat{Q}$ the replay memory D to capacity N
3:	Each MVNO $m\in M$ estimates the maximum total needed rate and minimum total needed rate of linked users, then create the Bidding pool B;
4:	For $b_m$ in B do
5:	Find the lower tier optimal allocation action and store it in table A;
6:	end for
7:	Random choose an action $a_t$ i.e., bidding value $b_mϵB$ and BS distributes $c_{m }$ to each MVNO according to (2);
8:	Repeat
9:	For t = 1, to T, do
10:	Calculate the ratio of the allocated bandwidth to its required minimum rate, and take it as the current state S = s of the last iteration;
11:	For m = 1 to M, do
12:	Each MVNO m allocates optimal bandwidth $c_j^m$ to its users according to table A;
13:	Each MVNO m calculates the $v_m$ by (9) and (10);
14:	Each MVNO m calculates the penalty $q_m$ by (4);
15:	Each MVNO m and calculates the profit $y_m$ by (3) and get the reward ${\mathrm{r}}_{\mathrm{m}}$;
16:	End for
17:	Calculate the total system utility F according to (5);
18:	Calculate the total reward r;
19:	Choose an action $a_t$ i.e., bidding value $b_mϵB$ according to the policy of DQN;
20:	InP distributes $c_{m }$ to each MVNO according to (2);
21:	Get the state S = s’ after the selection action of this iteration;
22:	#Train DQN
23:	The agent i.e., each MVNO inputs $\left(s,a, s^{\prime },r\right)$ into the DQN;
24:	The agent stores transition $\left(s,a, s^{\prime },r\right)$ in D;
25:	The agent sample random minibatch of transitions $\left(s\_,a\_, s^{\prime }\_,r\_\right)$ from D;
26:	Set $$y\_=\{\begin{array}{c}{r}_{-}\\ r\_+\gamma ma{x}_{a^{*}}\widehat{Q}\left(s^{\prime }\_,a^{*};{\theta }^{-}\right)\end{array}\begin{array}{c}\mathrm{if} \mathrm{episode} \mathrm{terminates} \mathrm{at} \mathrm{step} \_+1\\ \mathrm{otherwise}\end{array}$$
27:	The agent perform a gradient descent step on ${\left(y\_-Q\left(s\_,a\_;\theta \right)\right)}^2$ with respect to the network parameters θ;
28:	Every steps reset $\widehat{Q}=Q$;
29:	End for
30:	Until The predefined maximum number of iterations has been completed.