Algorithm 2 Main Online Learning Algorithm

1:Initialize: Time slot count: $t=0$; 2: while $t\ne T$ do 3: Increment the iteration index $t=t+1$; 4:       for $f=1:F$ 5:             if $t=1$ (initial time slot) 6:                   then Initialize the super arm for the first trial ($k=1$) as                   ${\mathcal{A}}_1^{\left[\mathrm{set}\right]}=\{0_1,\cdots ,0_N\}$, 7:             else                   ${\mathcal{A}}_1^{\left[\mathrm{set}\right]}=S^{*}$, 8:             end if 9:       Exploration Stage: Run Algorithm 1 10:       Estimation Stage: 11:       Calculate the mean reward vector for the frame                   ${\widehat{\mu }}_{\mathbf{n}}^{[\mathbf{f},\mathbf{t}]}=({\widehat{\mu }}_{n,1}^{[f,t]},{\widehat{\mu }}_{n,2}^{[f,t]},\cdots ,{\widehat{\mu }}_{n,\mathcal{J}}^{[f,t]})$, where                   ${\widehat{\mu }}_{n,p}^{[f,t]}=\frac{{\sum }_{k=1}^K{\mu }_{n,p}^{[k,f,t]}}{K},\forall p\in \mathcal{J},n\in {\mathcal{L}}_b$. 12:       Adjustment Stage: 13:             if ${\Psi }_p$ (number of times the p-th arm is played) $\ne 0$ 14:                   then                   adjust ${\overline{\mu }}_{n,p}^{[f,t]}={\widehat{\mu }}_{n,p}^{[f,t]}+\sqrt{\frac{3\ln K}{2{\Psi }_p}}$, 15:             else                   ${\overline{\mu }}_{n,p}^{[f,t]}={\widehat{\mu }}_{n,p}^{[f,t]},\forall p\in \mathcal{J},n\in {\mathcal{L}}_b$. 16:             end if 17:       end for 18: Average adjusted mean reward vector over all frames                   ${\overline{\mu }}_{\mathbf{n}}^{\left[\mathbf{t}\right]}=\left(\frac{\sum _{f\in \mathcal{F}}{\overline{\mu }}_{n,1}^{[f,t]}}{F},\frac{\sum _{f\in \mathcal{F}}{\overline{\mu }}_{n,2}^{[f,t]}}{F},\cdots ,\frac{\sum _{f\in \mathcal{F}}{\overline{\mu }}_{n,J}^{[f,t]}}{F}\right),n\in {\mathcal{L}}_b$. 19: Exploitation Stage: 20: Average ${\overline{\mu }}_{\mathbf{n}}^{\left[\mathbf{t}\right]}$ over accumulated number of time slots, as                    ${\overline{\mu }}_{\mathbf{n}}=\frac{{\sum }_{t^{\prime }=1}^t{\overline{\mu }}_{\mathbf{n}}^{\left[{\mathbf{t}}^{\prime }\right]}}{t}=[{\overline{\mu }}_{n,1},{\overline{\mu }}_{n,2},\cdots ,{\overline{\mu }}_{n,J}],n\in {\mathcal{L}}_b$. 21: For the next time slot: find N optimum arm indexes as                   $p_n^{*}=\underset{p}{argmax}\left({\overline{\mu }}_{n,p}\right),p\in \mathcal{J},\forall n\in {\mathcal{L}}_b$, and the updated super arm as                   $S^{*}=\Delta \mathcal{E}[p_1^{*},p_2^{*},\cdots ,p_N^{*}]$. 22: end while