Algorithm 1 Deep deterministic policy gradient algorithm.

Input I: Status and reward $\left(s_t,r_t,s_{t+1}\right)$ Output I: Action $a_t\to \Delta t_i$ 1. for each bus, from $t=1$ to $t=T$; 2. Initial test system parameters, including network parameters, reward and punishment functions, etc. 3. Combine with adjustment constraints and random factor, this model determined the action $a_t$, $a_t\in A$, and transfer the action into the simulation part. 4. The simulation part executes action $a_t$, and it will receive the reward value $r_t$ and the new status $s_{t+1}$. 5. If the sample pool overflows, then delete the earliest sample records in chronological order. 6. The actor network will put the $\left(s_t,a_t,r_t,s_{t+1}\right)$ into experience playback, which supplies the train data for the online network. 7. Sampling from the experience pool, gain N sets of sample data $\left(s_t,a_t,r_t,s_{t+1}\right)$ as the training set for the online actor network and Q network. 8. Use the standard BP method to calculate the gradient of the online Q network. 9. Update the parameter ${\theta }^e$ of the online Q network. 10. Calculate the policy gradient (PG) of the actor network. 11. Update the parameter ${\theta }^n$ of the online actor network. 12. Update the parameters $n1,e1$ of the target network. 13. End for.