Algorithm 1 Decision Transformer–based Hierarchical Reinforcement Learning (DT-HRL)

Input: Offline dataset 𝒟, action primitive set 𝒫, task embedding ${\varphi }_{\mathrm{task}}\left(·\right)$, goal embedding ${\varphi }_{\mathrm{goal}}\left(·\right)$, learning rate $\eta$, path-efficiency loss term $L_{\mathrm{PE}}$

Output: Hierarchical policy $\pi =\{ {\pi }_{\theta }^{high},{\pi }_{\theta }^{low} \}$

// Train High-Level Decision Transformer //

1.  Initialization

2.  for episode do

3.       Sample a task $\tau$ and trajectory $\{{\left(s_t,a_t,r_t,g_t\right)}_{t=0}^T$ from 𝒟

4.       $S=\left[s_{t-H},\dots ,s_t\right], A=\left[a_{t-H},\dots ,a_t\right]$

5.      ${â}_{t+1}\leftarrow {\pi }_{\theta }^{high}\left(S,A,{\varphi }_{task}\left(\tau \right),{\varphi }_{goal}\left(g_t\right)\right)$

6.      $a_H={â}_{t+1}$             ▷ Delivered to low-level policy

7.      $L\leftarrow \mathrm{CrossEntropy}\left({â}_{t+1},a_{t+1}\right)+\lambda L_{PE}$

8.       $\mathsf{\theta }\leftarrow \mathsf{\theta }-\mathsf{\eta }{\nabla }_{\mathsf{\theta }}L$

9.  end for

// Low-Level Execution //

10.  function ${\pi }_{\theta }^{low}\left(a_H\right)$

11.       $s_0\leftarrow \mathrm{initial}\mathrm{state}$

12.       for $k = 0$$\mathrm{to} K$ do

13.             $a_{k+1}\leftarrow {\pi }_{\theta }^{low}\left(s_k,a_H\right)$       ▷ Execute low-level controller

14.             $s_{k+1}\leftarrow Environment\left(s_k,a_{k+1}\right)$

15.              if done then

16.                   break

17.              end if

18.        end for

19.  end function