Algorithm 1. The A2C-based training method.

1:Initialize actor and critic network ${\pi }_{\theta }$, $V_{\varphi }$ 2:for epoch = 1: EP do 3:Perform new job arrival at time zero (current system time) 4:Get current state st 5:whilestep = 1: n do \\ n is the number of jobs for the selected instance 6:Determine an action at based on probability ${\pi }_{\theta }(s_t)$ at state st 7:Select a job j from BF using action at, process job j in all machines, obtain the finished time of job j in each machine 8:Push forward the system time only to the time when job j is finished in M1 9:Perform new job arrival and update WIP at current system time 10:Get current state st+1 and reward rt 11:Store transition $\{s_t,a_t,r_t,s_{t+1}\}$ of this step 12:st ← st+1 13:if step % T == 0 then 14:Calculate discounted reward drt of the T steps in reverse order using data in transitions, $d{r}_t=\left\{\begin{array}{c}{r}_t+\gamma V_{\varphi }(s_{t+1}),\mathrm{for}\mathrm{the}T\mathrm{th}\mathrm{step}\\ r_t+\gamma d{r}_{t+1},\mathrm{for}\mathrm{the}\mathrm{first}T-1\mathrm{steps}\end{array}\right.$ 15:Update critic-network $V_{\varphi }$ using gradient ${\nabla }_{\varphi }{(d{r}_t-V_{\varphi }(s_t))}^2$ 16:Update actor-network ${\pi }_{\theta }$ using gradient ${\nabla }_{\theta }\log {\pi }_{\theta }(a_t|s_t)[d{r}_t-V_{\varphi }(s_t)]+\beta {\nabla }_{\theta }H({\pi }_{\theta }(s_t;\theta ))$ 17:end if 18:end while 19:end for