Algorithm 3 Federated Learning Algorithm with Dynamic UE Selection using D4PG

Input: Global model parameters $w_{\mathrm{global}}$, UE dataset $D_k$ for each UE k, hyperparameter $\lambda$ for all UEs, maximum number of global iterations G, local training iterations L.

Output: Final global model parameters $w_{\mathrm{global}}^{\left(G\right)}$, trained D4PG agent, performance metrics.

1:Initialization: Initialize global model parameters $w_{\mathrm{global}}$. Initialize the hyperparameter $\lambda$ for all UEs.  2:D4PG Agent Initialization: Initialize the D4PG agent with appropriate observation and action spaces, neural networks for policy and value functions, and experience replay buffer.  3:Dynamic Sampling: At each global iteration g:  4: D4PG agent dynamically selects a subset $K_{\mathrm{sampled}}$ of $K_g$ UEs based on learned policies  5:Global Model Broadcast: Broadcast the global model parameters $w_{\mathrm{global}}$ to all UEs in $K_{\mathrm{sampled}}$.  6:Local Training: Each UE $k\in K_{\mathrm{sampled}}$ performs local training:  7: Initialize local model $w_{\mathrm{local}}$ with $w_{\mathrm{global}}$.  8:for $(l=0,1,\dots ,L-1)$ do  9:   Randomly select a data point ${\xi }_{k,g,l}$ from local dataset $D_k$. 10:  Update local model: As shown in Equation (45) 11:end for 12:Send Local Models: Each UE sends its updated local model $w_{\mathrm{local}}^{(g,L)}$ to the base station. 13:Global Model Aggregation: The base station aggregates the local models to update the global model: As shown in Equation (46) 14:D4PG Agent Update: Update the D4PG agent using the reward based on the performance and fairness of the aggregated model, storing experiences in the replay buffer, and training the neural networks. 15:Repeat: Repeat the process for G global iterations.