Algorithm 1 Q-λ (and related models): For SARSA-λ we replace the expression ${\max }_{\tilde{a}}Q(s_{n+1},\tilde{a})$ in line 9 by $Q(s_{n+1},a_{n+1})$ where $a_{n+1}$ is the action taken in the next state $s_{n+1}$. For Q-0 and SARSA-0 we set λ to zero.
1:	Algorithm Parameters: learning rate ${\displaystyle \alpha \in (0,1]}$, discount factor ${\displaystyle \gamma \in [0,1]}$, eligibility trace decay factor ${\displaystyle \lambda \in [0,1]}$, temperature ${\displaystyle T\in (0,\mathrm{\infty })}$ of softmax policy ${\displaystyle p}$, ${\displaystyle \mathrm{b}\mathrm{i}\mathrm{a}\mathrm{s}\mathrm{b}\in [0,1]}$ for preferred action ${\displaystyle a_{pref}\in \mathbf{A}}$.
2:	Initialize ${\displaystyle Q(s,a)=0}$ and ${\displaystyle e(s,a)=0}$ for all ${\displaystyle s\in \mathit{S},a\in \mathit{A}}$ For preferred action ${\displaystyle a_{pref}\in \mathit{A}}$ set ${\displaystyle Q(s,a_{pref})=b}$
3:	for each episode do
4:	Initialize state ${\displaystyle s_n\in \mathit{S}}$
5:	Initialize step ${\displaystyle n=1}$
6:	while ${\displaystyle s_n}$ is not terminal do
7:	Choose action ${\displaystyle a_n\in \mathbf{A}}$ from ${\displaystyle s_n}$ with softmax policy ${\displaystyle p}$ derived from ${\displaystyle Q}$
8:	Take action ${\displaystyle a_n}$, and observe ${\displaystyle r_{n+1}\in \mathbb{R}}$ and ${\displaystyle s_{n+1}\in \mathbf{S}}$
9:	${\displaystyle \mathit{R}\mathit{P}\mathit{E}(n\to n+1)\leftarrow r_{n+1}+\gamma {\mathrm{m}\mathrm{a}\mathrm{x}}_{\tilde{a}}Q(s_{n+1},\tilde{a})-Q(s_n,a_n)}$
10:	${\displaystyle e_n(s_n,a_n)\leftarrow 1}$
11:	for all ${\displaystyle s\in \mathbf{S},a\in \mathbf{A}}$ do
12:	${\displaystyle Q(s,a)\leftarrow Q(s,a)+\alpha RPE(n\to n+1)e_n(s,a)}$
13:	${\displaystyle e_{n+1}(s,a)\leftarrow \gamma \lambda e_n(s,a)}$
14	${\displaystyle n\leftarrow n+1}$