Vision-Based Robot Navigation through Combining Unsupervised Learning and Hierarchical Reinforcement Learning

. 2019 Apr 1;19(7):1576. doi: 10.3390/s19071576

Algorithm 1 Hierarchical Reinforcement Learning

1:
Initialize experience replay memories ${D_{H}, D_{L}}$ and the exploration probability $ϵ_{H} = 1$ and $ϵ_{L} = 1$ for the high-level and low-level policies, respectively.
2:
Initialize parameters $θ_{H}, θ_{L}$ for the online networks and $θ_{H}^{-}, θ_{L}^{-}$ for the target networks.
3:
For i = 1, num_episodes do
4:
Initialize the navigation and get the start state s.
5:
While s is not teriminal do
6:
$R \leftarrow 0$
7:
$s_{0} \leftarrow s$
8:
With probability $ϵ_{H}$ select a random subgoal g from the PC-GWR nodes
9:
Otherwise select $g = {max}_{g} Q_{H} (s, g; θ_{H})$
10:
while not (s is terminal or goal g is reached) do
11:
With probability $ϵ_{L}$ select a random action a
12:
Otherwise select $a = {max}_{a} Q_{L} (s, g, a; θ_{L})$
13:
Execute a and obtain the next state $s^{'}$ and extrinsic reward $r^{e}$ from the environment
14:
Obtain the intrinsic reward $r^{i}$
15:
Store transition $({s, g}, a, r^{i}, {s^{'}, g})$ in $D_{L}$
16:
Sample random mini-batches from $D_{L}$
17:
Update $θ_{L}$ according to Equation (7)
18:
$R \leftarrow R + r_{e}$
19:
$s \leftarrow s^{'}$
20:
end while
21:
Store transition $(s_{0}, g, R, s^{'})$ in $D_{L}$
22:
Sample random mini-batches from $D_{H}$
23:
Update $θ_{H}$ according to Equation (6)
24:
Every C steps copy $θ_{H}$ to $θ_{H}^{-}$ and $θ_{L}$ to $θ_{L}^{-}$
25:
end while
26:
Anneal $ϵ_{H}$ and $ϵ_{L}$ adaptively
27:
end for