Algorithm 1. The Q-learning method pseudocode.

Initialize  Set the state $s$ and the action $a$  For each state $s_i$ and action $a_i$ Set $Q(s_i,a_i)=0$  End For  Randomly choose an initial state $s_t$ While the terminal condition is not reached do Choose the best action $a_t$ from the current state $s_t$ from Q-table Execute action $a_t$, then get the immediate reward  Find out the new state $s_{t+1}$  Acquire the corresponding maximum Q-value of $s_{t+1}$  Update the Q-table by (19)  Update the state $s_t\leftarrow s_{t+1}$ End While