Algorithm 2 PPO-Based Deep Optimistic Linear Support

1:Initialization: 2:$\mathbb{S}$ = partial CSS, which is composed of $V_t$ obtained after the PPO learning. 3:$\mathbb{W}$ = corner weights, which is obtained from $\mathbb{S}$. 4:$\mathbb{Q}$ = priority queue of weights for the multi-objective, where the weights form a tuple along with their importance (i.e., ($[{\omega }_1^t,{\omega }_2^t],I$)). Instruction:    5:${\omega }_t=\mathbb{Q}$.pop() 6:for iteration=1,2, ….,do 7:    for iteration=1,2, …., T do 8:        $a_t={\pi }_{{\theta }_{old}}\left(s_t\right)$ 9:        $[r_e^t,r_d^t],s_{t+1}=Env\left(a_t\right)$ 10:        Reduce scaling of $[r_e^t,r_d^t]$ 11:        $M=M\cup \{s_t,a_t,[r_e^t,r_d^t],s_{t+1}\}$ 12:        $[A_e^t,A_d^t]=$ compute advantage estimate from Equation (26) 13:        $\widehat{A_t}=\widehat{A_t}\cup \{[A_e^t,A_d^t]\times [{\omega }_1^t,{\omega }_2^t]\}$ 14:    end for 15:    Optimize surrogate L and wrt $\theta$ from $\widehat{A_t}$, with K epochs    16:    Optimize $V_{\varphi }$ and wrt $\varphi$ from ${\widehat{V_t}}^{GAE(\gamma ,\lambda )}$, with K epochs 17:${\theta }_{old}=\theta ,{\varphi }_{old}=\varphi$ 18:end for when convergence 19:$V_t=V_{\varphi }\left(s\right)$ 20:$\mathbb{W}=\mathbb{W}\cup {\omega }_t$ 21:if${\omega }_t·V_t>{\sum }_{U\in \mathbb{S}}{\omega }_t·U$ then 22:    $\mathbb{S}$ = remove obsolete $V_{del}$ due to new $V_t$ 23:    ${\omega }_c$ = new corner weight from $\mathbb{S}$ 24:    $\mathbb{S}=\mathbb{S}\cup V_t$ 25:    $\mathbb{Q}$ = remove obsolete ${\omega }_{del}$ due to new ${\omega }_c$ 26:    for iteration=1,2, …., ${\omega }_c$ do 27:        if estimate improvement of $({\omega }^{\prime },\mathbb{W},\mathbb{S})>\tau$ then 28:           $\mathbb{Q}=\mathbb{Q}\cup {\omega }^{\prime }$ 29:        end if 30:    end for 31:end if 32:if$\mathbb{Q}$ is not empty then 33:    go back to line 1 34:end if