Algorithm 1 FDRT’s Mobile Device Agent Training.

INPUT: maximum number of training episodes $MAX\_EPISODE$, maximum number of training steps per episode $MAX\_STEP$, learning rate $\alpha$, discount factor $\gamma$, exploration rate $\epsilon$, experience replay pool $\mathcal{M}$ capacity $CAPACITY$, number of batch samples $BATH\_SIZE$, current network $Q$, target network $\widehat{Q}$, target network update frequency $C$, numbers of training in a round of federated learning $F$ OUTPUT: parameter ${\theta }_t$ of current network // $random\left(a,b\right)$ is a function that generates random numbers in range $\left[\left.a,b\right)\right.$ // $randint\left(a,b\right)$ is a function that generates random integers in range $\left[\left.a,b\right)\right.$ // $memory.isfull()$ indicates whether the experience replay pool $\mathcal{M}$ is full // ${\theta }^e$ denotes the global model parameters of the MEC server Initialization: Wireless communication model between MDs and BSs; task model and queue model of MD$,\mathrm{parameter}\theta$ of current$\mathrm{network}Q$$,\mathrm{parameter}\widehat{\theta }=\theta$$\mathrm{of}\mathrm{target}\mathrm{network}\widehat{Q}$$,\mathrm{experience}\mathrm{replay}\mathrm{pool}\mathcal{M}$. 1:  for $episode=0:MAX\_EPISODE$ by 1 do 2:    Get initial state $s_0$; 3:    for $t=0:MAX\_STEP$ by 1 do 4:       $x\leftarrow random\left(0,1\right)$; 5:      if($x>\epsilon$) then 6:         $a_t\leftarrow randint\left(0,2\right)$; 7:      else 8:        $a_t\leftarrow \underset{a}{\mathrm{argmax}}Q\left(s_t,a;{\theta }_t\right)$; 9:      end if 10:      Perform action $a_t$ in the system model, get reward $r\left(s_t,a_t\right)$ and next state $s_{t+1}$; 11:      $s_t\leftarrow s_{t+1}$; 12:      Put $I_t:=\left(s_t,a_t,r\left(s_t,a_t\right),s_{t+1}\right)$ in$\mathcal{M}$; 13:      if$(memory.isfull())$ then 14:        continue; 15:      end if 16:      if$\left(\left(episode*MAX\_STEP+t\right) mod F==0\right)$ then 17:         Upload ${\theta }_t$ to connected MEC server; 18:         ${\theta }_t\leftarrow {\theta }^e$; 19:      else 20:        Randomly choose a batch sample from $\mathcal{M}$ to update parameter, ${\theta }_t\leftarrow {\theta }_t-\alpha \nabla L\left({\theta }_t\right)$; 21:      end if 22:      if$\left(t mod C==0\right)$ then 23:        ${\widehat{\theta }}_t\leftarrow {\theta }_t$; 24:      end if 25:    end for 26:  end for 27:  return ${\theta }_t$;