Algorithm 1 Maximum entropy exploration with neural networks

Input: $\alpha ,N,{\theta }_0,{\mathcal{X}}_0,k$ for  $i=1,\dots ,N$  do     Receive context ${\mathit{s}}_i$ and choose $a_i\sim {\pi }_i$ where     ${\pi }_i\left(a\right|{\mathit{s}}_i)=\frac{e^{{\widehat{r}}_{{\theta }_{i-1}}(a,{\mathit{s}}_i)/\alpha }}{{\int }_{a^{\prime }}{e}^{{\widehat{r}}_{{\theta }_{i-1}}(a^{\prime },{\mathit{s}}_i)/\alpha }d{a}^{\prime }}$     Agent receives reward $r_i$     Add the triplet $\{{\mathit{s}}_i,a_i,r_i\}$ to the dataset $\mathcal{X}$     Every k steps train the model ${\widehat{r}}_{\theta }$:     ${\theta }_i={arg\; min}_{\theta }{\sum }_{\{s_j,a_j,r_j\}\in X}|r_j-{\widehat{r}}_{\theta }(a_j,s_j)|$ end for