Algorithm 1 Visuomotor Control for Humanoid Loco-manipulation Tasks

Require: batch size M, trajectory length N, number of actors K, replay size R, exploration       constant $ϵ$, initial learning rates ${\alpha }_0$ and ${\beta }_0$

Learner

1: Initialize visual encoder weights v, which is shared by actor and critic networks $(\theta ,w)$

2: Initialize the remaining network weights $(\theta ,w)$ at random

3: Initialize target weights $\left({\theta }^{\prime },w^{\prime }\right)\leftarrow (\theta ,w)$

4: Launch K actors and replicate network weights $(\theta ,w)$ to each actor

5: for  $t=1,\dots ,T$ do

6:       Sample M transitions $\left({\mathbf{s}}_{i:i+N},{\mathbf{a}}_{i:i+N-1},r_{i:i+N-1}\right)$ of length N from replay with priority $p_i$

7:       Construct the target distributions $Y_i=\left({\sum }_{n=0}^{N-1}{\gamma }^n{r}_{i+n}\right)+{\gamma }^N{Z}_{w^{\prime }}\left({\mathbf{s}}_{i+N},{\pi }_{{\theta }^{\prime }}\left({\mathbf{s}}_{i+N}\right)\right)$

8:       Compute the actor and critic updates

${\delta }_{\theta }={\left.{\displaystyle \frac{1}{M}}{\sum }_i{\nabla }_{\theta }{\pi }_{\theta }\left({\mathbf{s}}_i\right)\mathbb{E}\left[{\nabla }_{\mathbf{a}}{Z}_w\left({\mathbf{s}}_i,\mathbf{a}\right)\right]\right|}_{\mathbf{a}={\pi }_{\theta }\left({\mathbf{s}}_i\right)}$

${\delta }_w={\displaystyle \frac{1}{M}}{\sum }_i{\nabla }_w{\left(R{p}_i\right)}^{-1}d\left(Y_i,Z_w\left({\mathbf{s}}_i,{\mathbf{a}}_i\right)\right)$

9:       Update network parameters $\theta \leftarrow \theta +{\alpha }_t{\delta }_{\theta },w\leftarrow w+{\beta }_t{\delta }_w$

10:       If $t=0mod{t}_{\mathrm{target}}$, update the target networks $\left({\theta }^{\prime },w^{\prime }\right)\leftarrow (\theta ,w)$

11:       If $t=0mod{t}_{\mathrm{actors}}$, replicate network weights to the actors

12: end for

13: return policy parameters $\theta$

Actor

1: repeat

2:     wait until current movement step finishes

3:     Sample action $\mathbf{a}={\pi }_{\theta }\left(\mathbf{s}\right)+ϵ\mathcal{N}(0,1)$

4:     Esecute action $\mathbf{a}$, observe reward r and state ${\mathbf{s}}^{\prime }$, which includes an one-hot vector       representing current FSM state

5:     Calculate priority p using Equation (6)

6:     Store $\left(\mathbf{s},\mathbf{a},r,{\mathbf{s}}^{\prime },p\right)$ in replay

7: until Learner finishes