Algorithm 1:Q-learning algorithm.

1. Initialize Q arbitrarily Q (terminal) =0 Repeat       initialize s       Repeat             choose $a^{\prime }\in ϵ-greedily$             take action a, observe $r,s^{\prime }$             $Q(s_t,a_t)\leftarrow Q(s_t,a_t)+\alpha [r_{t+1}+\gamma Q(s_{t+1},a_{t+1}]-Q(s_t,a_t)$             $s\leftarrow s^{\prime }$       s is terminal until convergence