Algorithm 2 DualContextAtt DQN Algorithm

Step 1: Initialize Initialize Q-network and target Q-network with dueling architecture and context-aware self-attention. Initialize replay buffer D. for each episode do       Reset environment and observe initial state s.       for each time step do             Select action $a_t$ using $ϵ$-greedy: $$a_t=arg\underset{a}{max}Q(s,a;\theta )$$             Execute $a_t$, observe reward $r_t$ and next state $s_{t+1}$.             Store $(s_t,a_t,r_t,s_{t+1})$ in buffer D.             Sample mini-batch of transitions $(s,a,r,s^{\prime })$ from D.             Step 2: Compute Target Q-values $$y_i=\left\{\begin{array}{cc}{r}_i+\gamma \underset{a^{\prime }}{max}{Q}^{\prime }(s^{\prime },a^{\prime };{\theta }^{-}),\hfill & \mathrm{if}\mathrm{not}\mathrm{terminal}\hfill \\ r_i,\hfill & \mathrm{if}\mathrm{terminal}\hfill \end{array}\right.$$             Step 3: Compute Predicted Q-values with Attention Fusion             Compute value stream $V(s;\theta )$             Compute advantage stream $A(s,a;\theta )$             Compute attention fusion weight $w(s;\theta )$             Fuse: $$Q(s,a;\theta )=w·V\left(s\right)+(1-w)·\left(A(s,a)-\frac{1}{\left|A\right|}\sum _{a^{\prime }}A(s,a^{\prime })\right)$$             Step 4: Compute Loss (MSE) $$L\left(\theta \right)=\frac{1}{N}\sum _{i\in N}{\left(Q_{\theta }(s_i,a_i)-Q_{\theta }^{\prime }(s_i,a_i)\right)}^2$$ where $$Q_{\theta }^{\prime }=R(s_t,a_t)+\gamma \underset{a_i^{\prime }}{max}{Q}_{\theta }(s_i^{\prime },a_i^{\prime })$$             Step 5: Update Parameters             Perform gradient descent on $L\left(\theta \right)$             Update target network parameters: ${\theta }^{-}\leftarrow \theta$ periodically       end for end for Step 6: Repeat Until Convergence Repeat steps until convergence or maximum episodes.