Algorithm 1 DDPG-based Learning Algorithm
1	Initialize weights ${\theta }^Q$ and ${\theta }^{\mu }$ of critic network $Q\left(s,a|{\theta }^Q\right)$ and actor network $\mu \left(s|{\theta }^{\mu }\right)$
2	Initialize weights ${\theta }^{Q\prime }\leftarrow {\theta }^Q$, ${\theta }^{\mu \prime }\leftarrow {\theta }^{\mu }$ of target network $Q\prime$ and $\mu \prime$
3	Initialize experience replay buffer$R$
4	for $episode$ = 1, 2, …, $M$ do
5		Receive initial observation state ${\mathit{s}}_1$
6		for $t$ = 1, 2, …, $T$ do
7			Choose $a_t=\mu \left(s_t|{\theta }^{\mu }\right)$ and do simulation using pandapower
8			Observe reward $r_t$ and the next state $s_{t+1}$
9			Store transition $\left(s_t,a_t,r_1,s_{t+1}\right)$ in $R$
10			Sample a random minibatch of $B$ transitions $\left(s_i,a_i,r_i,s_{i+1}\right)$ from $R$
11			Set $y_i=r_i+\gamma \cdot Q\prime \left(s_{i+1},\mu \prime \left(s_{i+1}|{\theta }^{\mu \prime }\right)|{\theta }^{Q\prime }\right)$ according to Equation (26)
12			Update critic network parameters by minimizing the loss, see Equation (27):      $Loss=\frac{1}{B}{{\displaystyle \sum }}_i{\left(y_i-Q\left(s_i,a_i|{\theta }^Q\right)\right)}^2$
13			Update the actor policy using the sampled policy gradient, see Equation (28):      ${\nabla }_{{\theta }^{\mu }}J\approx \frac{1}{B}{\displaystyle {\displaystyle \sum }_i}{\nabla }_{a}Q\left(s,a\left|{\theta }^Q\right.\right){|}_{s=s_i,a=\mu \left(s_i\right)}\cdot {\nabla }_{{\theta }^{\mu }}\mu \left(s|{\theta }^{\mu }\right){|}_{s_i}$
14			Softly update the target networks using the updated critic and actor network parameters:      ${\theta }^{Q\prime }\leftarrow \tau {\theta }^Q+\left(1-\tau \right){\theta }^Q$      ${\theta }^{\mu \prime }\leftarrow \tau {\theta }^{\mu }+\left(1-\tau \right){\theta }^{\mu }$
15		end for
16	end for