-
1:
Output: Optimal policy parameters .
-
2:
Hyperparameters: training iterations K, rollouts per iteration , horizon T, PPO epochs E, trajectories per minibatch , discount , GAE parameter , entropy coefficient , PPO clip .
-
3:
Initialize M actor policies and a centralized critic ; set iteration .
-
4:
while
do
-
5:
Clear replay buffer D.
-
6:
Rollout: collect on-policy data.
-
7:
for rollout index to do
-
8:
Initialize environment state .
-
9:
Run Algorithm 1 (PA-BAC) using , , to obtain .
-
10:
for to T do
-
11:
Set global observation for all agents.
-
12:
Sample each agent action ; form joint action .
-
13:
Execute actions and evolve the environment.
-
14:
UAV positions .
-
15:
Update ship positions according to mobility models.
-
16:
Compute energy and update remaining energy according to energy model.
-
17:
Obtain next state , call Algorithm 1 (PA-BAC) to solve under C1–C5 and compute .
-
18:
Compute reward by (31)–(34).
-
19:
Store transition into D.
-
20:
end for
-
21:
end for
-
22:
GAE: compute advantages and value targets
-
23:
For each trajectory in D, compute by (35), by (36), and by (37).
-
24:
for epoch = 1 to E do
-
25:
Shuffle D and split into minibatches, each containing trajectories.
-
26:
for each minibatch B do
-
27:
Randomly generate an agent index permutation .
-
28:
Initialize sequential weights .
-
29:
for to M do
-
30:
Compute PPO ratio by (38).
-
31:
Update actor parameters by minimizing in (39).
-
32:
if then
-
33:
update sequential weights by (40).
-
34:
end if
-
35:
end for
-
36:
Update critic parameters by minimizing in (41).
-
37:
end for
-
38:
end for
-
39:
Set
-
40:
end while