Algorithm 1 Super Arm Exploration

1:Initialize: Total number of trials K 2:for $k=1:K$ 3:   Solve problem (16) for a given $B_n^{\left[\mathrm{ahead}\right]}\left(k\right)$, 4:    CU calculates $B_n^{\left[\mathrm{total}\right]}\left(k\right)$ as per (2), $\mathcal{R}\left(B_n^{\left[\mathrm{ahead}\right]}\left(k\right)\right)$ as per (17), and $\mathcal{R}\left({\mathcal{A}}_k^{\left[\mathrm{set}\right]}\right)$ as per (18). 5:       If $k=1$ 6:             then                   $B_n^{\left[\mathrm{ahead}\right]}(k+1)=B_n^{\left[\mathrm{ahead}\right]}\left(k\right)+\Delta \mathcal{E},n\in {\mathcal{L}}_b$. 7:       else if the super arm reward of all the RRHs                   $\mathcal{R}\left({\mathcal{A}}_k^{\left[set\right]}\right)\le \mathcal{R}\left({\mathcal{A}}_{k-1}^{\left[set\right]}\right)$, 8:             then                   $B_n^{\left[\mathrm{ahead}\right]}(k+1)=B_n^{\left[\mathrm{ahead}\right]}(k-1)$, $\forall n\in {\mathcal{L}}_b$, 9:       else if the individual reward for the n-th RRH, $n\in N$                   $\mathcal{R}\left(B_n^{\left[\mathrm{ahead}\right]}\left(k\right)\right)\ge \mathcal{R}\left(B_n^{\left[\mathrm{ahead}\right]}(k-1)\right)$                                     and                   $B_n^{\left[\mathrm{ahead}\right]}\left(k\right)\ne {\mathcal{E}}^J$, 10:             then    $B_n^{\left[\mathrm{ahead}\right]}(k+1)=B_n^{\left[\mathrm{ahead}\right]}\left(k\right)+\Delta \mathcal{E}$, 11:       else                   $B_n^{\left[\mathrm{ahead}\right]}(k+1)=B_n^{\left[\mathrm{ahead}\right]}\left(k\right)$. 12:       end If 13:    Calculate the total energy cost of all the RRHs, ${\beta }^{[k,f,t]}$ as                   ${\beta }^{[k,f,t]}=\sum _{n\in {\mathcal{L}}_b}{B}_n^{\left[\mathrm{total}\right]}\left(k\right)$. 14:    Calculate the energy package index p at all RRHs from                   $p=\frac{B_n^{\left[\mathrm{ahead}\right]}\left(k\right)}{\Delta \mathcal{E}},n\in {\mathcal{L}}_b$. 15:    Update                   ${\mu }_{n,p}^{[k,f,t]}=\mathcal{R}\left(B_n^{\left[\mathrm{ahead}\right]}\left(k\right)\right),\forall p\in \mathcal{J},n\in {\mathcal{L}}_b$; 16:    Update                   ${\mathcal{A}}_{k+1}^{\left[\mathrm{set}\right]}=\{B_1^{\left[\mathrm{ahead}\right]}(k+1),\cdots ,B_N^{\left[\mathrm{ahead}\right]}(k+1)\}$; 17: end for 18:       Estimated mean reward for K trials                                     ${\widehat{\mu }}_{n,p}^{[f,t]}=\frac{{\sum }_{k=1}^K{\mu }_{n,p}^{[k,f,t]}}{K},\forall p\in \mathcal{J},n\in {\mathcal{L}}_b$.