Algorithm 3 Proposed reinforcement learning algorithm

Initialize $Q(s,a)=0$ where s is the set of states and a is the set of actions while Battery lifetime is not equal to zero do     Determine current state     Select action a based on policy $$\frac{e^{Q(s,a)}/\omega }{{\sum }_a\left(e^{Q(s,a)/\omega }\right)}$$     Execute the selected action     Calculate the reward $$Rf=\left\{\begin{array}{cc}\frac{LQ{M}_l}{1-LQ{M}_l}+LQ{M}_l\times A\hfill & \mathrm{for}n\le N.\hfill \\ \frac{LQ{M}_l}{1-LQ{M}_l}\hfill & n>N.\hfill \end{array}\right.$$     Calculate the learning rate $$\phi =\frac{Z}{visited(s,a)}$$     Calculate Q value for the executed action $$Q_{t+1}(s_t,a_t)=(1-\phi )Q_t(s_t,a_t)+\phi (Rf\left(s_{t+1}\right)+\mathsf{\Gamma }{V}_t\left(s_{t+1}\right))$$     Calculate the value function for the executed action $$V_{t+1}\left(s_t\right)=\underset{a\in A}{max}{Q}_{t+1}(s_t,a)$$     Update the utility table of the scheduler agent $$U\left(q\right)=(1-\mathsf{{\rm Y}})U\left(q\right)+\mathsf{{\rm Y}}\sum _{i}R{f}_i$$     Move to the next state based on the executed action end while