Algorithm 1 Traffic Signal Control Using Parameterized Deep RL.

1:Initialize: Learning rates $\{l{r}_Q,l{r}_x\}$, exploration parameter $ϵ$, minibatch size B, a probability distribution $\zeta$, flow configurations, network weights ${\mathbf{\omega }}_0$ and ${\mathbf{\theta }}_0$. 2:for episode $e=1,\cdots E$ do 3:    Start simulation, observe initial state $s_0$ and take initial joint action $a_0$. 4:     for  $t=1,\cdots T$  do 5:         Compute action parameters ${d_P}_t\leftarrow {x_d}_P(s_t;{\mathbf{\theta }}_t)$. 6:          Select action $a_t=(P_t,{d_P}_t)$ according to $ϵ$-greedy policy. $$a_t=\left\{\begin{array}{c}\mathrm{a}\mathrm{sample}\mathrm{from}\zeta \mathrm{with}\mathrm{probability}ϵ,\hfill \\ (P_t,{d_P}_t)P_t=\underset{P}{\mathrm{argmax}}Q(s_t,P,{d_P}_t;{\mathbf{\omega }}_t)1-ϵ.\end{array}\right.$$ 7:         Perform $a_t$, observe next state $s_{t+1}$ and get ${\mathcal{R}}_t$. 8:         Store $<s_t,a_t,{\mathcal{R}}_t,s_{t+1}>$ in memory M. 9:         Sample random B experiences from M. 10: $$y_t=\left\{\begin{array}{c}{\mathcal{R}}_t\mathrm{if}t=T,\hfill \\ {\mathcal{R}}_t+\underset{P}{\max }\gamma Q(s_{t+1},P,{x_d}_P(s_{t+1};\mathbf{\theta });{\mathbf{\omega }}_t)\mathrm{otherwise}.\end{array}\right.$$ 11:         $\mathrm{Compute}{\nabla }_{\mathbf{\omega }}{\ell }_t^Q\left({\mathbf{\omega }}_t\right)\mathrm{and}{\nabla }_{\mathbf{\theta }}{\ell }_t^Q\left(\mathbf{\theta }\right)\mathrm{using}\{y_t,s_t,a_t\}$. 12:         update weights ${\mathbf{\omega }}_{t+1}\stackrel{}{\leftarrow }{\mathbf{\omega }}_t-l{r}_Q{\nabla }_{\mathbf{\omega }}{\ell }_t^Q\left({\mathbf{\omega }}_t\right)$ and ${\mathbf{\theta }}_{t+1}\stackrel{}{\leftarrow }{\mathbf{\theta }}_t-l{r}_x{\nabla }_{\mathbf{\theta }}{\ell }_t^Q\left(\mathbf{\theta }\right)$. 13:     end for 14:end for