Algorithm 1 Updating $R_{𝓉}^{\lambda }$

1: For transition $\left(S_{𝓉},a_{𝓉},r_{𝓉},R_{𝓉}^{\lambda },S_{𝓉+1}\right)$, from back to front do

2:   If $S_{𝓉+1}$ is terminal

3:     Update $R_{𝓉}^{\lambda }\leftarrow r_{𝓉}$

4:   Else

5:     Obtain the next transition from $\mathcal{T}$ $$\left(S_{𝓉+1},a_{𝓉+1},r_{𝓉+1},R_{𝓉+1}^{\lambda },S_{𝓉+2}\right)$$

6:     Update $R_{𝓉}^{\lambda }\leftarrow r_{𝓉}+\gamma \left[\lambda R_{𝓉+1}^{\lambda }+\left(1-\lambda \right)ma{x}_{a\in \mathcal{A}}Q\left(s_{𝓉+1},a\right)\right]$

7:   End If

8: End For