Algorithm 1: The routing mechanism.

Initialize $Q$();

While xi. next_hop ! = source node

for xj in Ni do

calculate ce, ct, P;

nodes satisfy $ce\left({\mathrm{e}}_{\mathrm{wth}}\right)<ce\left(e_{res}^j\right)$ are saved in set $N_{i0}$;

end for

if $N_{i0}! = \varnothing$ then

set ${\phi }_{ih}$ to ${\phi }_{ih}^{\prime }$, $\mathrm{h}\in N_i/\left(N_{i0}\right)$;

calculate the direct reward r;

select the node xj with maximum $Q$ value in set Ni;

calculate $Q$(xi, ah),$\mathrm{h}\in N_i/\left(N_{i0}\right)$;

aj = argmax($Q$(xi, ah));

else aj = argmax($Q$(xi, ah))

end if

xi = xj