Algorithm 1.

Non-deterministic fitted-Q

Learn Q̂T = (Q̂T[1], …, Q̂T[D]), set 𝒬T = {Q̂T}

for t = T − 1, T − 2, …, 1 do

for all $s_t^i$ in the data do

Generate ${\mathrm{\Pi }}_{\prec }^{\exists }(s_t^i)$ using 𝒬t+1

𝒬t ← ∅

for all ${\pi }_t\in {𝒞}_{\varphi }({\mathrm{\Pi }}_{\prec }^{\exists })$ do

for all Q̂t+1 ∈ 𝒬t+1 do

Learn (Q̂t[1](·, ·, πt, …), …, Q̂t[D] (·, ·, πt, …)) using Q̂t+1, add to 𝒬t