Algorithm 2. Training the prediction neural network

Network structure:

input layer: $2n$ neurons ($n$ agent positions and $n$ target location probabilities, both relative to the size $n$ of the domain),

hidden layer: $2n$ neurons,

output layer: $9$ neurons (in accordance with the number of possible actions).

Activation function:

sigmoid function $f\left(x\right)=1/\left(1+e^{-x}\right)$.

Loss function:

mean square error (MSE) function.

Input: domain $C=\left\{c_1,c_2,\dots ,c_n\right\}$,

set $\mathbb{A}=\left\{\uparrow ,\nearrow ,\to ,\searrow ,\downarrow ,\swarrow ,\leftarrow ,\nwarrow ,\odot \right\}$ of possible actions,

probability $p_{TA}$ of true alarms (Equation (3)),

rate $\alpha$ of false alarms and their probability $p_{FA}=\alpha p_{TA}$ (Equation (4)),

sensor sensitivity $\lambda$,

discount factor $\gamma$,

objective probability map $P^{*}$ (obtained by using the value $\epsilon$),

number $r$ of iterations for updating the weights,

initial value $\eta$ (Equation (22)) and its discount factor $\delta$,

learning rate $\rho$ (with respect to the type of optimizer),

number $M$ of epochs,

initial weights $w$ of the prediction network and initial weights $w^{\prime }=w$ of the target network,

training data set (that is, the $L\times N$ table of $\left(c,P\right)$ pairs created by Procedure 1).

Output: The trained prediction network.

1. Create the prediction network.

2. Create the target network as a copy of the prediction network.

3. For each epoch $j=1,\dots ,M$ do:

4. For each pair $\left(c,P\right)$ from the training data set, do:

5. For each action $\mathbb{a}\in \mathbb{A}$ do:

6. Calculate the value $Q\left(c,P,\mathbb{a};w\right)$ with the prediction network.

7. Calculate the probability $p\left(\mathbb{a}|Q;\eta \right)$ (Equation (22)).

8. End for.

9. Choose an action according to the probabilities $p\left(\mathbb{a}|Q;\eta \right)$.

10. Apply the chosen action and set the next position $c^{\prime }=\mathbb{a}\left(c\right)$ of the agent.

11. Calculate the next probability map $P^{\prime }$ with Equations (20) and (21).

12. If $P=P^{*}$ or $c^{\prime }\notin C$, then

13. Set the immediate reward $R\left(\mathbb{a}\right)=0$.

14. Else

15. Calculate the immediate reward $R\left(\mathbb{a}\right)$ with respect to $P$ and $P^{\prime }$ (Equation (14)).

16. End if.

17. For each action $\mathbb{a}\in \mathbb{A}$ do:

18. If $P=P^{*}$ then

19. Set $Q\left(c^{\prime },P^{\prime },\mathbb{a};w^{\prime }\right)=0$.

20. Else

21. Calculate the value $Q\left(c^{\prime },P^{\prime },\mathbb{a};w^{\prime }\right)$ with the target network.

22. End if.

23. End for.

24. Calculate the target value $Q^{+}=R\left(\mathbb{a}\right)+\gamma \underset{\mathbb{a}\in \mathbb{A}}{\max }Q\left(c^{\prime },P^{\prime },\mathbb{a};w^{\prime }\right)$ (Equation (17)).

25. Calculate the temporal difference learning error as ${\Delta }_l\left(Q\right)=Q^{+}-Q\left(c,P,\mathbb{a};w\right)$ for the chosen action $\mathbb{a}$ (Equation (19)) and set ${\Delta }_l\left(Q\right)=0$ for all other actions.

26. Update the weights $w$ in the prediction network by backpropagation with respect to the error ${\Delta }_l\left(Q\right)$.

27. Every $r$ iterations, set the weights of the target network as $w^{\prime }=w$.

28. End for.