Algorithm 1 Q–Learning Algorithm

Input: State set S, Action set A, Reward function R Output: Q table   1: Initialize Q table   2: while $i\le$ the number of iterations do   3:    Return s to the initial state   4:    while s is not terminal do   5:        Choose a from A using policy derived from $Q\left(s\right)$ (e.g., $ϵ-$ greedy)   6:        Take action a, observe r, $s^{{}^{\prime }}$   7:        $Q(s,a)\leftarrow Q(s,a)+\alpha [r+\gamma ma{x}_{a^{{}^{\prime }}}Q(s^{{}^{\prime }},a^{{}^{\prime }})-Q(s,a)]$   8:        $s\leftarrow s^{{}^{\prime }}$   9:    end while 10: end while 11: Return Q