Algorithm 2 Contextual bandit with Energy Based Models

Input: $N,{\theta }_0,{\mathcal{X}}_0,K,c,\alpha ,a_{max},a_{min},\eta ,\sigma$ for  $i=1,\dots ,N$  do     Choose $a_i\sim {\pi }_i$ with SGLD, ${\tilde{a}}^0\sim U(a_{min},a_{max})$     for $k=1,\dots ,K$ do         Draw sample for noise $\omega \sim \mathcal{N}(0,\sigma )$         ${\tilde{a}}^k\leftarrow {\tilde{a}}^{k-1}-\eta {\nabla }_x{E}_{{\theta }_{i-1}}({\tilde{a}}^{k-1},{\mathit{s}}_i)/\alpha +\omega$     end for     Play action ${\tilde{a}}^K$, receive $r_i$, update $\mathcal{X}$     Every c steps train $E_{\theta }$ in batches:     ${\theta }_i={arg\; min}_{\theta }{\sum }_{X}log(1+e^{E_{\theta }(a^{+},{\mathit{s}}_j)-E_{\theta }(a^{-},{\mathit{s}}_j)})$ end for