Algorithm 1:

Tuning parameters selection via cross-validation

1	Input: Data ${\left\{Z_h\right\}}_{h=1}^N$, a set of M policies {π1, · · ·, πM} ⊂ Π, a set of J candidate tuning parameters ${\left\{\left({\mu }_j,{\lambda }_j\right)\right\}}_{j=1}^J$ in the value function estimation, and a set of J candidate tuning parameters ${\{({\mu }_j^{\prime },{\lambda }_j^{\prime })\}}_{j=1}^J$ in the ratio function estimation.
2	Randomly split Data into K subsets: ${\left\{Z_h\right\}}_{h=1}^N={\left\{D_k\right\}}_{k=1}^K$
3	Denote e(1) (m, j) and e(2) (m, j) as the total validation error for m-th policy and j-th pair of tuning parameters in value and ratio function estimation respectively, for m = 1, · · · M and j = 1, · · ·, J. Set their initial values as 0.
4	Repeat for m = 1, · · ·, M,
5	Repeat for k = 1, · · ·, K,
6	Repeat for j = 1, · · ·, J
7	Use ${\left\{Z_h\right\}}_{h=1}^N\backslash D_k$ to compute $\left({\widehat{\eta }}_n^{{\pi }_m},\widehat{\alpha }\left({\pi }_m\right)\right)$ and $\widehat{\nu }({\pi }_m)$ by (6.2)–(6.3) and (6.4)–(6.5) using tuning parameters (μj, λj) and $({\mu }_j^{\prime },{\lambda }_j^{\prime })$ respectively;
8	Compute ${\delta }^{{\pi }_m}(\cdot ;\widehat{\eta }\left({\pi }_m\right),{\widehat{Q}}_n^{{\pi }_m})$ and ${\epsilon }^{{\pi }_m}(\cdot ;{\widehat{H}}_n^{{\pi }_m})$ and their corresponding squared Bellman errors mse(1) and mse(2) on the dataset Dk by Gaussian kernel regression;
9	Assign e(1) (m, j) = e(1) (m, j) + mse(1) and e(2) (m, j) = e(2)(m, j) + mse(2);
10	Compute j(1)* ∈ argminj maxm e(1) (m, j) and j(2)* ∈ argminj maxm e(2) (m, j)
11	Output: $({\mu }_{j^{(1)\ast }}^{(1)},{\lambda }_{j^{(1)\ast }}^{(1)})$ and $({\mu }_{j^{(2)\ast }}^{\prime },{\lambda }_{j^{(2)\ast }}^{\prime })$.