Algorithm 4 Dynamic Sampling with D4PG

Input: Environment, replay buffer D, Q-network parameters ${\theta }_{\mathrm{online}}$ and ${\theta }_{\mathrm{target}}$, exploration rate $ϵ$, learning rate $\alpha$, discount factor $\gamma$, target network update frequency C, target network update rate $\tau$

Output: Optimal action sequence for task offloading and resource allocation

1:Initialize ${\theta }_{\mathrm{online}}$ and ${\theta }_{\mathrm{target}}$  2:for each global iteration g do  3:  Observe current state $s_g$  4:  Select action $a_g$:  5:  if $\mathrm{random}(0,1)<ϵ$ then  6:   Choose a random subset of UEs as $a_g$  7:  else  8:   Choose $a_g=arg{max}_{a}Q(s_g,a;{\theta }_{\mathrm{online}})$  9:  end if 10: Sample subset $K_{\mathrm{sampled}}$ from $a_g$ 11: Record $(s_g,a_g)$ in D 12: Conduct local training and obtain reward R 13: Sample mini-batch $(s,a,r,s^{\prime })$ from D 14: Set target: 15: if $s^{\prime }$ is terminal then 16:  $target=r$ 17: else 18:  $target=r+\gamma \mathbb{E}\left[Q(s^{\prime },a^{\prime };{\theta }_{\mathrm{target}})\right]$ 19: end if 20:  Update ${\theta }_{\mathrm{online}}$ by minimizing the loss: $$L={\left(Q(s,a;{\theta }_{\mathrm{online}})-target\right)}^2$$ 21:  Every C steps, update ${\theta }_{\mathrm{target}}$ by: $${\theta }_{\mathrm{target}}\leftarrow \tau {\theta }_{\mathrm{online}}+(1-\tau ){\theta }_{\mathrm{target}}$$ 22: end for