Algorithm 1 Q-learning

initialize $Q(s,a)$ arbitrarily, where s denotes the state of the agent and a denotes the action for each episode do     initialize s     while s is not terminal state and steps number < max steps number do         choose a from s using policy derived from Q         take action a, observe reward r, and next state $s^{\prime }$         $Q(s,a)\leftarrow Q(s,a)+\alpha \left[r+\gamma {max}_{a^{\prime }}Q(s^{\prime },a^{\prime })-Q(s,a)\right]$         $s\leftarrow s^{\prime }$     end while end for