Algorithm 1 Training process of the improved D3QN

1:Input: Total iteration rounds T, attenuation factor $\gamma$, and exploration rate $\epsilon$. 2:Initialize the evaluation network parameters $\theta$, target network parameters ${\theta }^{\prime }=\theta$, ${\theta }^{\prime }$ update frequency x, and replay buffer D; 3:for $t=1$ to T do 4:      Randomly select actions according to the probability of $\epsilon$ in the optimized exploration space e, otherwise select actions according to the greedy policy; 5:      Taking action, the agent gets new status $s_{t+1}$ and reward; 6:      Store combination $(s_t,{\overrightarrow{a}}_t,R_t,s_{t+1})$ to D; 7:      Randomly select a batch of $(s_t,{\overrightarrow{a}}_t,R_t,s_{t+1})$ from D and calculate Q-Value of the evaluation network; 8:      Calculate loss and update parameters $\theta$; 9:      Every x steps reset ${\theta }^{\prime }=\theta$, update the target Q-values, and reduce $\epsilon$ value; 10:     $s_t=s_{t+1}$; 11:end for