Algorithm 1 PPO-Based Training Algorithm

Input: discounting factor $\mathsf{\gamma }$; clipping ratio $\mathsf{\epsilon }$; update epoch $L$; number of training steps $E$; critic network $v_{\varnothing }$; actor-network ${\pi }_{\theta }$, behavior actor-network ${\pi }_{{\theta }_{old}}$, where ${\theta =\theta }_{old}$; entropy loss coefficient $f_e$; value function loss coefficient $f_v$; policy loss coefficient $f_p$. 1    Initialize ${\pi }_{\theta }$, ${\pi }_{{\theta }_{old}}$, and $v_{\varnothing }$; 2    for e = 1 to E 3        Pick N independent scheduling instances from distribution D; 4        for n = 1 to N 5              for t = 1 to … 6                  sample $a_{n,t}$ based on ${\pi }_{{\theta }_{old}}\left(a_{n,t}|S_{n,t}\right)$; 7                  Receive reward $r_{n,t}$ and next state $S_{n,t+1}$; 8                  Compute the advantage function ${\widehat{A}}_{n,t}$ and probability ratio $r_{n,t}\left(\theta \right)$. 9                  ${\widehat{A}}_{n,t}$ = ${\sum }_0^t{Y}^t{r}_{n,t}-V_{\varnothing }(S_n,t)$; 10                    $r_{n,t}\left(\theta \right)$=$\frac{{\pi }_{\theta }\left(a_{n,t}|S_{n,t}\right)}{{\pi }_{{\theta }_{old}\left(a_{n,t}|S_{n,t}\right)}}$ 11                    while $s_{n,t}$ is terminal do 12                          break; 13                  end while 14            end for 15            Compute the policy loss $L_n^{PPO}\left(\theta \right)$, the value function loss $L_n^{critic}\left(\varnothing \right)$ and the entropy loss $L_n^S\left(\theta \right)$. 16            $L_n^{PPO}\left(\theta \right)={\sum }_0^t\mathrm{m}\mathrm{i}\mathrm{n}(r_{n,t}\left(\theta \right){\widehat{A}}_{n,t},clip(r_{n,t}\left(\theta \right),1-\mathsf{\epsilon },1+\mathsf{\epsilon }){\widehat{A}}_{n,t})$; 17            $L_n^{critic}\left(\varnothing \right)={\sum }_0^t{(v_{\mathrm{\varnothing }}\left(s_{n,t}\right)-{\widehat{A}}_{n,t})}^2$; 18            $L_n^S\left(\theta \right)={\sum }_0^t\mathrm{S}\left({\pi }_{\theta }\right(a_{n,t}\left|s_{n,t}\right))$, where $\mathrm{S}(·)$ is entropy; 19            Total Losses: $L_n\left(\theta ,\varnothing \right)=f_p{L}_n^{PPO}\left(\theta \right)-{f_{v}L}_n^{critic}\left(\varnothing \right)+{f_{e}L}_n^S\left(\theta \right)$; 20        end for 21        for l = 1 to L 22            Update $\theta$, $\varnothing$ with cumulative loss by Adam optimizer: 23            $\theta$,$\varnothing \leftarrow Adam\left({\sum }_{n=1}^N{L}_n\left(\theta ,\varnothing \right)\right)$ 24        end for 25    ${\mathit{\theta }}_{\mathit{o}\mathit{l}\mathit{d}}\leftarrow \mathit{\theta }$ 26    end for 27    Output: Trained parameter set of $\theta$.