Deep Reinforcement Learning for Semantic Secure Energy Efficiency Optimization in IRS-Assisted UAV Communications

View full-text article in PMC

. 2026 Sep 11;26(18):5779. doi: 10.3390/s26185779

Algorithm 1 Hierarchical HG-SAC Algorithm for SSEE Maximization

1:
Initialize: candidate semantic symbol set $N_{s} = {1, \dots, 7}$ , $N_{s}^{★}$ , $π^{★}$ , $η_{SSEE}^{★} = - \infty$ .
2:
for each candidate semantic symbol number $N_{s} \in N_{s}$ do
3:
Initialize: actor network $π_{ϕ}$ , twin critic networks $Q_{ω_{1}}$ , $Q_{ω_{2}}$ , and target networks.
4:
Generate heuristic samples according to (41), initialize buffer $D$ .
5:
Pre-train the actor network using heuristic samples.
6:
for each training episode do
7:
Observe the initial state $s [1]$ .
8:
for each time slot $n = 1, \dots, N$ do
9:
Generate heuristic action $a_{h} [n]$ according to (41).
10:
Select action $a [n]$ according to the hybrid strategy (50).
11:
Execute action, obtain reward $r [n]$ via (33), observe $s [n + 1]$ .
12:
Store transition $(s [n], a [n], r [n], s [n + 1])$ into $D$ .
13:
Randomly sample a mini-batch from $D$ .
14:
Compute target Q-value according to (37).
15:
Update twin critic networks by minimizing (38).
16:
Update actor network according to (39).
17:
Softly update target networks according to (40).
18:
end for
19:
Decay heuristic guidance probability.
20:
end for
21:
Obtain the optimized policy $π_{N_{s}}$ .
22:
Evaluate the resulting SSEE $η_{SSEE} (N_{s})$ .
23:
if $η_{SSEE} (N_{s}) > η_{SSEE}^{★}$ then
24:
Update $η_{SSEE}^{★} \leftarrow η_{SSEE} (N_{s})$ , $N_{s}^{★} \leftarrow N_{s}$ , $π^{★} \leftarrow π_{N_{s}}$ .
25:
end if
26:
end for
27:
Output: Optimal semantic symbol number $N_{s}^{★}$ and optimized policy ${q^{★} [n], P^{★} [n], Θ^{*} [n]}$ .