Algorithm 1: Proximal policy optimization (PPO) for a multivariable system

Initialization: actor function parameters $\theta$, critic function parameters $\varphi$, hyperparameters: truncation factor $\epsilon$, number of sub-iterations$M,B$ for k = 0, 1, 2, … do  Randomly initialize setpoints $SP$ within the range allowed for each loop of a multivariable coupling system.  Enforce policy ${\pi }_{{\theta }_k}$ in the environment, and save the track $D_k=\text{{}{\tau }_i\text{}}$  Calculated rewards $R_t$  Calculate the advantage function ${\stackrel{\wedge }{A}}_t$ based on the current critic function $V_{{\varphi }_k}$  Normalizing the ${\stackrel{\wedge }{A}}_t$ using the mean and standard deviation of the current small batch experience  for $m\in \text{{}1,\cdots ,M\text{}}$ do   Computing importance sampling weights:    ${\mathcal{l}}_t({\theta }^{’})=\frac{{\pi }_{\theta }\left(A_t\right|S_t)}{{\pi }_{{\theta }_{\mathrm{old}}}\left(A_t\right|S_t)}$   Using the Adam stochastic gradient ascent algorithm to maximize the objective function of PPO clip to update the policy:    ${\theta }_{k+1}=\underset{\theta }{\arg \max }\frac{1}{\left|D_k\right|T}{\displaystyle \sum _{\tau \in D_k}{\displaystyle \sum _{t=0}^T\min }}({\mathcal{l}}_t({\theta }^{’})A^{{\pi }_{{\theta }_{\mathrm{old}}}}(S_t,A_t),\mathrm{clip}({\mathcal{l}}_t({\theta }^{’}),1-\epsilon ,1+\epsilon )A^{{\pi }_{{\theta }_{\mathrm{old}}}}(S_t,A_t\left)\right)$  end for  for $b\in \text{{}1,\cdots ,B\text{}}$ do   The critic function is learned by minimizing the mean square error using the gradient descent algorithm:    ${\varphi }_{k+1}=\underset{\varphi }{\arg \min }\frac{1}{\left|D_k\right|T}{\displaystyle \sum _{\tau \in D_k}{\displaystyle \sum _{t=0}^T{\left(V_{\varphi }\right(S_t)-R_t)}^2}}$  end for end for