Alert-Driven Active Defense for IoT-Enabled CBTC Systems Using Bayesian Hypergame Modeling and Hierarchical Reinforcement Learning

. 2026 Jul 14;26(14):4475. doi: 10.3390/s26144475

Algorithm 1 HRL for Dynamic Defense Strategy Solving

Require: System state space S, belief space

M

, strategy sets

A_{A}, A_{D}

. Learning rates

α_{1}, α_{2}

, discount factor

γ

, convergence threshold

ξ

, exploration rate

ϵ_{e x p}

1:
Initialize:
2:
$Q^{(1)} (s, σ)$ and $Q^{(2)} (s, a_{D})$ with random weights;
3:
initial belief $μ_{0}^{D} (θ_{A})$ .
4:
for each episode $r = 1, 2, \dots, R$ do
5:
Observe initial system state $s_{0} \in S$ .
6:
while s is not terminal do
7:
High-level (Meta-Policy):
8:
Observe attacker behavior history $h^{k}$ .
9:
Update Bayesian belief $μ_{D}^{k + 1} (θ_{A} | h^{k + 1})$ using Equation (8).
10:
Select strategic posture $σ \in {σ_{p a s s i v e}, σ_{a c t i v e}, σ_{h y b r i d}}$ using $ϵ_{e x p}$ -greedy policy based on $Q^{(1)} (s, σ)$ .
11:
Low-level (Sub-Policy):
12:
Retrieve attacker cognitive error parameter $δ$ from hypergame model.
13:
Execute specific action $a_{D}$ based on

$Q^{(2)} (s, a_{D} | σ, δ)$ .
14:
Environment Interaction:
15:
Observe attacker action $a_{A}$ , next state $s^{'}$ , and transitions.
16:
Calculate High-level reward:
17:
$R^{(1)} = f (μ_{D} (θ_{A})) + U_{D} (s, σ)$
18:
Calculate Low-level reward:
19:
$R^{(2)} = g (δ) + U_{D} (s, a_{D})$
20:
Policy Update:
21:
$Q^{(1)} (s, σ) \leftarrow (1 - α_{1}) Q^{(1)} (s, σ) + α_{1} [R^{(1)} + γ \max_{σ^{'}} Q^{(1)} (s^{'}, σ^{'})]$ .
22:
$Q^{(2)} (s, a_{D}) \leftarrow (1 - α_{2}) Q^{(2)} (s, a_{D}) + α_{2} [R^{(2)} + γ \max_{a_{D}^{'}} Q^{(2)} (s^{'}, a_{D}^{'})]$ .
23:
$s \leftarrow s^{'}$ .
24:
end while
25:
Convergence Check:
26:
if $∥ π_{D}^{r + 1} - π_{D}^{r} ∥ < ξ$ and $∥ π_{A}^{r + 1} - π_{A}^{r} ∥ < ξ$ then Terminate: Stable hierarchical policy profile reached.
27:
end if
28:
end for