Algorithm 2 SARSA

initialize $Q(s,a)$ arbitrarily, where s denotes the state of the agent and a denotes the action for each episode do     initialize s     choose a from s using policy derived from Q     while s is not terminal state and steps number < max steps number do         take action a, observe reward r, and next state $s^{\prime }$         choose the next action $a^{\prime }$ from $s^{\prime }$ using policy derived from Q         $Q(s,a)\leftarrow Q(s,a)+\alpha \left[r+\gamma Q(s^{\prime },a^{\prime })-Q(s,a)\right]$         $s\leftarrow s^{\prime }$     end while end for