Algorithm 2 CGRL algorithm.

Input: ${\mathcal{F}}_n={\{(x^l,u^l,r^l,y^l)\}}_{l=1}^n,L_f,L_{\rho },x_0,T$

Initialization:

D ← n × (T − 1) matrix initialized to zero;

A ← n–dimensional vector initialized to zero;

B ← n–dimensional vector initialized to zero;

Computation of the Lipschitz constants ${\left\{L_{Q_N}^{\prime }\right\}}_{N=1}^T$:

$L_{Q_1}^{\prime }=L_{\rho }$;

for k = 2 … T do

$L_{Q_k}^{\prime }\leftarrow L_{\rho }+L_f{L}_{Q_{k-1}}^{\prime }$;

end for

t ← T − 2;

while t > −1 do

for i = 1 … n do

$j_0\leftarrow \underset{j\in \{1,\dots ,n\}}{\text{arg max}}{r}^j-L_{Q_{T-t-1}}^{\prime }{║y^i-x^j║}_{\mathcal{X}}+B(j)$;

$m_0\leftarrow \underset{j\in \{1,\dots ,n\}}{\text{max}}{r}^j-L_{Q_{T-t-1}}^{\prime }{║y^i-x^j║}_{\mathcal{X}}+B(j)$;

A(i) ← m0;

D(i, t + 1) ← j0; \\ best tuple at t + 1 if in tuple i at time t

end for

B ← A;

t = t −1;

end while

Conclusion:

S ← (T + 1)–length vector of actions initialized to zero;

$l\leftarrow \underset{j\in \{1,\dots ,n\}}{\text{arg max}}{r}^j-L_{Q_T}^{\prime }{║x_0-x^j║}_{\mathcal{X}}+B(j)$;

$S(T+1)\leftarrow \underset{j\in \{1,\dots ,n\}}{\max }{r}^j-L_{Q_T}^{\prime }{║x_0-x^j║}_{\mathcal{X}}+B(j)$; \\ best lower bound

S(1) ← ul; \\ CGRL action for t = 0.

for t = 0 … T − 2 do

l′ ← D(l, t + 1);

S(t + 2,:) ← ul′; other CGRL actions

l ← l′;

end for

Return: S