Algorithm 1 Full training process for proposed ISSA

Inputs: ${\pi }_{{\varphi }^T},Q_{{\theta }_1^T},Q_{{\theta }_2^T},Q_{{\overline{\theta }}_1^T},Q_{{\overline{\theta }}_2^T}$: Teacher’s parametric networks for policy and Q functions, both based on combination of visual observation and internal states. ${\pi }_{{\varphi }^S},Q_{{\theta }_1^S},Q_{{\theta }_2^S},Q_{{\overline{\theta }}_1^S},Q_{{\overline{\theta }}_2^S}$: Student’s parametric networks for policy and Q functions, both solely based on visual observation. $aug$: image augmentation method inherited from DrQ-v2. $f_{\xi },T_s,T_r,B,\alpha ,\tau$: parametric network for image encoder, training steps for transfer learning and reinforcement learning, mini-batch size, learning rate and target update rate. 1:procedure  Main 2:    ${\mathbf{s}}_0^{\prime }\leftarrow f_{\xi }\left({\mathbf{x}}_0\right)$                 ▹ Get encoded state for initial observations 3:    for t:=0 to $T_s$ do                 ▹ Transfer learning part 4:        ${\mathbf{a}}_t\sim {\pi }_{{\varphi }^T}(·|{\mathbf{s}}_t^{\prime },{\mathbf{i}}_t)$                 ▹ Sample action based on teacher’s policy 5:        ${\mathbf{x}}_{t+1}\leftarrow p(·|{\mathbf{i}}_t,{\mathbf{a}}_t)$                 ▹ Get consecutive observations 6:        ${\mathbf{s}}_{t+1}^{\prime }\leftarrow f_{\xi }\left({\mathbf{x}}_{t+1}\right)$                 ▹ Get encoded state for timestep $t+1$ 7:        $\mathcal{D}\leftarrow \mathcal{D}\cup ({\mathbf{x}}_t,{\mathbf{i}}_t,{\mathbf{a}}_t,r({\mathbf{i}}_t,{\mathbf{a}}_t),{\mathbf{x}}_{t+1},{\mathbf{i}}_{t+1})$                 ▹ Store transition 8:        UpdateCriticTansfer($\mathcal{D}$)                 ▹ Update critic for teacher and student 9:        UpdateActorTransfer($\mathcal{D}$)                 ▹ Update policy for teacher and student 10:    end for                 ▹ End of Phase 1 (transfer learning phase) 11:    ${\mathbf{s}}_0^{\prime }\leftarrow f_{\xi }\left({\mathbf{x}}_0\right)$                 ▹ Get encoded state for initial observations 12:    for t:=0 to $T_r$ do                 ▹ Reinforcement learning part 13:        Traditional reinforcement learning based on ${\pi }_{{\varphi }^S},Q_{{\theta }_1^S},Q_{{\theta }_2^S},Q_{{\overline{\theta }}_1^S},Q_{{\overline{\theta }}_2^S}$ 14:    end for                 ▹ End of the entire training process 15:end procedure 16:procedure UpdateCriticTransfer($\mathcal{D}$) 17:    $\left\{({\mathbf{x}}_t,{\mathbf{i}}_t,{\mathbf{a}}_t,r_{t:t+n-1},{\mathbf{x}}_{t+n},{\mathbf{i}}_{t+n})\right\}\sim \mathcal{D}$                 ▹ Sample a mini batch of B transitions 18:    ${\mathbf{s}}_t^{\prime },{\mathbf{s}}_{t+n}^{\prime }\leftarrow f_{\xi }\left(aug\left({\mathbf{x}}_t\right)\right),f_{\xi }\left(aug\left({\mathbf{x}}_{t+n}\right)\right)$                 ▹ Data augmentation and encoding 19:    ${\mathbf{a}}_{t+n}\sim {\pi }_{{\varphi }^T}(·|{\mathbf{s}}_{t+n}^{\prime },{\mathbf{i}}_{t+n})$                 ▹ Sample action for timestep $t+n$ 20:    Compute ${\mathcal{L}}_{{\theta }_1^T,\xi },{\mathcal{L}}_{{\theta }_2^T,\xi },{\mathcal{L}}_{{\theta }_1^S,\xi },{\mathcal{L}}_{{\theta }_2^S,\xi }$ using Equation (1). 21:    $\xi \leftarrow \xi -\alpha {\nabla }_{\xi }({\mathcal{L}}_{{\theta }_1^T,\xi }+{\mathcal{L}}_{{\theta }_2^T,\xi }+{\mathcal{L}}_{{\theta }_1^S,\xi }+{\mathcal{L}}_{{\theta }_2^S,\xi })$                 ▹ Update encoder 22:    ${\theta }_k^i\leftarrow {\theta }_k^i-\alpha {\nabla }_{{\theta }_k^i}{\mathcal{L}}_{{\theta }_k^i,\xi }k\in \{1,2\},i\in \{T,S\}$                 ▹ Update Q functions 23:    ${\overline{\theta }}_k^i\leftarrow (1-\tau ){\overline{\theta }}_k^i+\tau {\theta }_k^{i}k\in \{1,2\},i\in \{T,S\}$                 ▹ Soft update target-Q functions 24:end procedure 25:procedure UpdateActorTransfer($\mathcal{D}$) 26:    $\left\{({\mathbf{x}}_t,{\mathbf{i}}_t)\right\}\sim \mathcal{D}$                 ▹ Sample a mini batch of B observations and internal states 27:    ${\mathbf{s}}_t^{\prime }\leftarrow f_{\xi }\left(aug\left({\mathbf{x}}_t\right)\right)$                 ▹ Data augmentation and encoding 28:    ${\mathbf{a}}_t\sim {\pi }_{{\varphi }^T}(·|{\mathbf{s}}_t^{\prime },{\mathbf{i}}_t)$                 ▹ Sample action for timestep $t+n$ 29:    Compute ${\mathcal{L}}_{{\varphi }^T},{\mathcal{L}}_{{\varphi }^S}$ using Equations (3) and (4). 30:    ${\varphi }^i\leftarrow {\varphi }^i-\alpha {\nabla }_{{\varphi }^i}{\mathcal{L}}_{{\varphi }^i}i\in \{T,S\}$                 ▹ Update teacher’s and student’s policy 31:end procedure