Algorithm 1 TBC-HRL: Strategy Optimization and Training Procedure

1:Init: high-level SAC ${\pi }_1$, low-level SAC ${\pi }_0$ (NDBCNet), twin critics $Q_{1,1},Q_{1,2}$ and $Q_{0,1},Q_{0,2}$, target critics $Q_{1,i}^{\mathrm{tgt}},Q_{0,i}^{\mathrm{tgt}}$, replay buffers $D_1,D_0$, optimizers with LR $({\eta }_1,{\eta }_0)$, discounts $(\gamma ,{\gamma }^{\tau })$, temperatures $({\alpha }_1,{\alpha }_0)$, Polyak factor ${\tau }_{\mathrm{polyak}}$, batch sizes $(B_1,B_0)$, update steps $(K_1,K_0)$  2:for each episode do  3:   Reset env; get $s_t$  4:   while episode not terminated do  5:     Sample high-level action $(g,\Delta t)\sim {\pi }_1(·|s_t)$; set $\Delta t_{\mathrm{rem}}\leftarrow \Delta t$, $\tau \leftarrow 0$, $R_{\mathrm{sum}}\leftarrow 0$, $s_{\mathrm{start}}\leftarrow s_t$  6:     for $k=0$ to $\Delta t-1$ do  7:        Sample $a\sim {\pi }_0(·|s_t,g,\Delta t_{\mathrm{rem}})$  8:        Execute a; observe $s_{t+1}$, reward $r_0$, done  9:        Store $(s_t,g,\Delta t_{\mathrm{rem}},a,r_0,s_{t+1},\Delta t_{\mathrm{rem}}-1)$ in $D_0$ 10:        $R_{\mathrm{sum}}\leftarrow R_{\mathrm{sum}}+r_0$;   $\Delta t_{\mathrm{rem}}\leftarrow \Delta t_{\mathrm{rem}}-1$;   $\tau \leftarrow \tau +1$ 11:        // low-level updates 12:        for $u=1$ to $K_0$ do 13:          Sample batch ${\left\{(s,g,\delta ,a,r,s^{\prime },{\delta }^{\prime })\right\}}_{B_0}\sim D_0$;   sample $a^{\prime }\sim {\pi }_0(·|s^{\prime },g,max({\delta }^{\prime },0))$ 14:          $y_0\leftarrow r+\gamma \left({min}_i{Q}_{0,i}^{\mathrm{tgt}}(s^{\prime },g,max({\delta }^{\prime },0),a^{\prime })-{\alpha }_{0}log{\pi }_0\left(a^{\prime }\right|s^{\prime },g,max({\delta }^{\prime },0))\right)$ 15:          Update $Q_{0,i}$ towards $y_0$;   update ${\pi }_0$ with respect to ${\alpha }_{0}log{\pi }_0-{min}_i{Q}_{0,i}$ 16:          Soft-update $Q_{0,i}^{\mathrm{tgt}}\leftarrow {\tau }_{\mathrm{polyak}}{Q}_{0,i}+(1-{\tau }_{\mathrm{polyak}})Q_{0,i}^{\mathrm{tgt}}$ 17:          (optional) adjust ${\alpha }_0$ by target entropy 18:        end for 19:        $t\leftarrow t+1$;   $s_t\leftarrow s_{t+1}$ 20:        if done or GoalReached($s_t,g$) or $\Delta t_{\mathrm{rem}}=0$ then 21:          break 22:        end if 23:     end for 24:     // aggregate and store high-level SMDP transition 25:     $r_1\leftarrow R_{\mathrm{sum}}$ (plus success bonus/penalty) 26:     Store $(s_{\mathrm{start}},g,\tau ,r_1,s_t,\mathrm{done})$ in $D_1$ 27:     if not GoalReached($s_t,g$) then 28:        $g^{\prime }\leftarrow s_t$ Hindsight Goal Relabeling (HGR) 29:        recompute $r_1^{\prime }$ 30:        Store $(s_{\mathrm{start}},g^{\prime },\tau ,r_1^{\prime },s_t,\mathrm{done})$ in $D_1$ 31:     end if 32:     // high-level updates 33:     for $u=1$ to $K_1$ do 34:        Sample batch ${\left\{(s,g,\tau ,r_1,s^{+},d)\right\}}_{B_1}\sim D_1$;   sample $(g^{+},\Delta t^{+})\sim {\pi }_1(·|s^{+})$ 35:        $y_1\leftarrow r_1+{\gamma }^{\tau }\left({min}_i{Q}_{1,i}^{\mathrm{tgt}}(s^{+},g^{+},\Delta t^{+})-{\alpha }_{1}log{\pi }_1(g^{+},\Delta t^{+}|s^{+})\right)$ 36:        Update $Q_{1,i}$ towards $y_1$;   update ${\pi }_1$ with respect to ${\alpha }_{1}log{\pi }_1-{min}_i{Q}_{1,i}$ 37:        Soft-update $Q_{1,i}^{\mathrm{tgt}}\leftarrow {\tau }_{\mathrm{polyak}}{Q}_{1,i}+(1-{\tau }_{\mathrm{polyak}})Q_{1,i}^{\mathrm{tgt}}$ 38:        (optional) adjust ${\alpha }_1$ by target entropy 39:     end for 40:   end while 41:end for 42:return trained ${\pi }_1\left({\theta }_1\right)$, ${\pi }_0\left({\theta }_0\right)$