Algorithm 2: Second-level algorithm of the learning hierarchy.

1. Initialization

2. While not converged, do

1. For each element i of the learner’s flag, do

1. If flag(i) is converted from 1 to 0, do     Find device u that helped device i among all devices, including neighbors’ helpers, and return the borrowed bandwidth from di to du

2. Else if flag(i) is converted from 0 to 1, do

a. Make a feasible device set, including the nodes’ own feasible devices and neighbors’ feasible help lists.

b. With probability ɛ, randomly choose device k among the feasible device set, otherwise:       ${\mathit{a}}_{\mathit{i}}=\mathit{a}\mathit{r}\mathit{g}\mathit{m}\mathit{a}{\mathit{x}}_{\mathit{j}\in \mathit{f}\mathit{e}\mathit{a}\mathit{s}\mathit{i}\mathit{b}\mathit{l}\mathit{e}}\left(\mathit{Q}\left(\mathit{i},{\mathit{z}}_{\mathit{n}},{\mathit{z}}_{\mathit{m}},\mathit{T}{\mathit{h}}_{\mathit{n}\mathit{e}\mathit{w}},\mathit{j}\right)\right)$, where j is the index of the selected.

c. Increase the bandwidth of di and decrease the bandwidth of dj, as much as di needs.

d. Calculate penalty or punish(i) or blame(i) based on the learner’s action and then calculate RnQ:       ${\mathit{R}}_{\mathit{n}\mathit{Q}}=\mathit{o}\mathit{p}\mathit{e}\mathit{r}\mathit{a}\mathit{t}\mathit{i}\mathit{o}\mathit{n}-\mathit{p}\mathit{u}\mathit{n}\mathit{i}\mathit{s}\mathit{h}\left(\mathit{i}\right)-\mathit{p}\mathit{e}\mathit{n}\mathit{a}\mathit{l}\mathit{t}\mathit{y}-\mathit{b}\mathit{l}\mathit{a}\mathit{m}\mathit{e}$

e. Update Q matrix:       ${\mathit{Q}}_{\mathit{n}\mathit{e}\mathit{w}}\left(\mathit{i},{\mathit{z}}_{\mathit{n}},{\mathit{z}}_{\mathit{m}},\mathit{T}{\mathit{h}}_{\mathit{n}\mathit{e}\mathit{w}},{\mathit{a}}_{\mathit{i}}\right)=\left(\mathbf{1}-\mathit{\alpha }\right){\mathit{Q}}_{\mathit{o}\mathit{l}\mathit{d}}\left(\mathit{i},{\mathit{z}}_{\mathit{n}},{\mathit{z}}_{\mathit{m}},\mathit{T}{\mathit{h}}_{\mathit{n}\mathit{e}\mathit{w}},{\mathit{a}}_{\mathit{i}}\right)+$       $\mathit{\alpha } \left({\mathit{R}}_{\mathit{n}\mathit{Q}}+\mathit{\gamma }\left(\mathit{m}\mathit{a}{\mathit{x}}_{{\mathit{k}}^{\prime }\mathit{ϵ} \mathit{f}\mathit{e}\mathit{a}\mathit{s}\mathit{i}\mathit{b}\mathit{l}\mathit{e}} \left({\mathit{Q}}_{\mathit{o}\mathit{l}\mathit{d}}\left({\mathit{i}}^{\prime },{\mathit{z}}_{\mathit{n}},{\mathit{z}}_{\mathit{m}},\mathit{T}{\mathit{h}}_{\mathit{n}\mathit{e}\mathit{w}},{\mathit{a}}_{\mathit{i}}^{’}\right)\right)\right)\right)$

3. End if

2. End for

3. Make the next flag based on the learner’s device priorities.

4. Repeat lines 3 to 26 for neighboring fog nodes

5. Calculate SBMn using countpun considering all fragments

6. Calculate the average SBM

7. Increase the time step n by 1

3. End while