Algorithm 2 Hybrid Reinforcement Learning Algorithm

1: Initialize: State space $S$, and action space $A$

2: Apply discrete actions randomly to the robot and collect data set ${\mathbb{D}}_1$

3: Using ${\mathbb{D}}_1$ to generate ${\mathbb{D}}_2$, and then obtain the transition model $ℳ$

4: Obtain the reduced action space $a_{\mathrm{reduced}}^i$

5: Initialize $\mathcal{A}$ using $a_{\mathrm{rough}}^i\in [a_{\mathrm{rough}(\min )}^i,a_{\mathrm{rough}(\max )}^i]$

6: Initialize replay memory $D$, to capacity $N$, parameter vector $\theta$

7: For $\mathrm{episode}=1,K$ do

8:   Reset the robot and the platform to their initial position, transition temporary table $\mathcal{T}=\varnothing$

9:   For $t=1, T$ do

10:     Obtain current state $S_t$ from sensors’ reading

11:     Select a random action from ${\mathcal{A}}_t$ with probability $ϵ$, otherwise select $a_t^k$       $a_t^k={\mathrm{argmax}}_{a\in \mathcal{A}}Q(S_t,a|\theta )$, observe the next state $s_{t+1}$, and receive an immediate reward

$r_{t+1}$

12:     Append transition $(S_t,a_t,r_t,R_t^{\lambda },S_{t+1})$ to $\mathcal{T}$

13:     If $S_{𝓉+1}$ is within stable region $S_s$

14:       Update $R_{\mathcal{T}}^{\lambda }$ using Algorithm 1

15:       Store $\mathcal{T}$ in $D$, refresh $\mathcal{T}$: $\mathcal{T}=\varnothing$

16:     End If

17:     Sample random minibatch of transitions from $D(S_j,a_j,r_j,R_j^{\lambda },S_{j+1})$, $j=1,2,\cdots ,P$

18:     Apply a gradient descent on $\theta$ to improve Q-function $\theta \leftarrow \theta +\alpha {\nabla }_{\theta }{(R_j^{\lambda }-Q(S_j,a_j|\theta ))}^2$

19:     Every $C$ step, update $R_D^{\lambda }$ using Algorithm 1

20:    End For

21: End For