Algorithm 1 The Optimal Strategy Pairs

Input: input $CMDP$ quintuple $<S,A,\{Agen{t}_0,Agen{t}_1\},\{T_0,T_1\},\{R_0,R_1\}>$: error parameter, $\epsilon >0$ balance parameter $0<\alpha <1$, discount factor $0<\gamma <1$. Output: output Optimal strategy pair $({\pi }_{*}^0,{\pi }_{*}^1)=({\pi }_k^0,{\pi }_k^1)$, optimal value function $V_{k+1}=V_{k+1}^{({\pi }_{*}^0,{\pi }_{*}^1)}$. Initialization V-function $V_1^0,V_1^1$, take the initial value randomly. Such as $V_1{}^1(s)=0(s\in S)$; suppose $V_1=\alpha V_1^0+(1-\alpha )V_1^1$; Use Equation (19) to greedy improvement strategy pair $({\pi }_1^0,{\pi }_1^1)$: $(V_1^{({\pi }_1^0,{\pi }_1^1)}=V_1)$; Use the updated strategy pair $({\pi }_1^0,{\pi }_1^1)$ in step 2 and Formula (18) to find the V-function $V_2$; Repeat steps 2, 3; Step $k+1$: assuming $({\pi }_k^0,{\pi }_k^1)$, $V_k^{({\pi }_k^0,{\pi }_k^1)}$ has been obtained, at this step, do the following two steps of calculation: (a) Evolutionary calculation of $CMDP$ system: find $V_{k+1}^{}$ by $({\pi }_k^0,{\pi }_k^1)$ and Formula (18), when $\parallel V_{k+1}^{}-V_k^{}\parallel <\epsilon$, defined $V_{k+1}^{({\pi }_k^0,{\pi }_k^1)}(s,t)=V_k(s,t)((s,t)\in S\times S)$. (b) Greedy computing $({\pi }_{k+1}^0,{\pi }_{k+1}^1)$: $\begin{array}{c}{\pi }_{0,k+1}^{}(s,t)=arg{max}_a{\Sigma }_{(s^{\prime },a^{\prime })\in S\times A}^{}{P}_0(s,t,a,s^{\prime },a^{\prime })[R_0(s,t,a,s^{\prime },a^{\prime })+\gamma ·V_0^{({\pi }_{0k},{\pi }_{1k})}(s^{\prime },t)],\hfill \\ {\pi }_{1,k+1}^{}(s,t)=arg{max}_a{\Sigma }_{(t^{\prime },a^{\prime })\in S\times A}^{}{P}_1(s,t,a,t^{\prime },a^{\prime })[R_1(s,t,a,t^{\prime },a^{\prime })+\gamma ·V_0^{({\pi }_{0k},{\pi }_{1k})}(s,t^{\prime })].\hfill \end{array}$ If ${\pi }_{k+1}^0={\pi }_k^0$, ${\pi }_{k+1}^1={\pi }_k^1$, terminate calculation. Return result.