A Dynamic Programming Algorithm for Finding an Optimal Sequence of Informative Measurements

View full-text article in PMC

. 2023 Jan 30;25(2):251. doi: 10.3390/e25020251

Algorithm 3 Stochastic Rollout Algorithm

1:
Input: $(x_{0}^{'}, x_{0}) \in S_{k}^{'} \times S_{k}$
2:
for $k = 0$ to $N - 1$ do
3:
for each $(u_{k}^{'}, u_{k}) \in U_{k}^{'} (x_{k}^{'}) \times U_{k} (x_{k}^{'}, x_{k})$ do
4:
repeat
5:
$w_{k} \sim p_{W_{k}}$ , $m_{k} \sim p_{M_{k}}$
6:
$x_{k + 1}^{'} \leftarrow v_{k} (x_{k}^{'}, u_{k}^{'}, w_{k}), x_{k + 1} \leftarrow f_{k} (x_{k}, u_{k}, m_{k})$
7:
for $i = k + 1$ to $N - 1$ do
8:
${{\hat{μ}}_{i}^{'} (x_{i}^{'}), {\hat{μ}}_{i} (x_{i}^{'}, x_{i})} \leftarrow G e n e r a t e_b a s e_p o l i c i e s (x_{i}^{'}, x_{i})$
9:
$w_{i} \sim p_{W_{i}}$ , $m_{i} \sim p_{M_{i}}$
10:
$x_{i + 1}^{'} \leftarrow v_{i} (x_{i}^{'}, {\hat{μ}}_{i}^{'} (x_{i}^{'}), w_{i}), x_{i + 1} \leftarrow f_{i} (x_{i}, {\hat{μ}}_{i} (x_{i}^{'}, x_{i}), m_{i})$
11:
end for
12:
Store: $h_{k} (x_{k}, u_{k}, m_{k}) + {\tilde{J}}_{k + 1} (x_{k + 1}^{'}, x_{k + 1})$
13:
until a selected criterion is met
14:
${\tilde{Q}}_{k} (x_{k}^{'}, x_{k}, u_{k}^{'}, u_{k}) \leftarrow \underset{w_{k}}{E} \{\underset{m_{k}}{E} \{h_{k} (x_{k}, u_{k}, m_{k}) + {\tilde{J}}_{k + 1} (x_{k + 1}^{'}, x_{k + 1})\}\}$
15:
end for
16:
${\hat{J}}_{k} (x_{k}^{'}, x_{k}) \leftarrow max_{u_{k}^{'} \in U_{k}^{'} (x_{k}^{'})} max_{u_{k} \in U_{k} (x_{k}^{'}, x_{k})} {\tilde{Q}}_{k} (x_{k}^{'}, x_{k}, u_{k}^{'}, u_{k})$
17:
${\bar{μ}}_{k} (x_{k}^{'}, x_{k}) \leftarrow u_{k}^{*}, {\bar{μ}}_{k}^{'} (x_{k}^{'}) \leftarrow u_{k}^{' *}$
18:
$w_{k} \sim p_{W_{k}}$ , $m_{k} \sim p_{M_{k}}$
19:
$x_{k + 1}^{'} \leftarrow v_{k} (x_{k}^{'}, {\bar{μ}}_{k}^{'} (x_{k}^{'}), w_{k}), x_{k + 1} \leftarrow f_{k} (x_{k}, {\bar{μ}}_{k} (x_{k}^{'}, x_{k}), m_{k})$
20:
end for