Skip to main content
. 2026 May 17;28(5):561. doi: 10.3390/e28050561
Algorithm 2 E-MAPPO-PA-BAC training procedure.
  •   1:

    Output: Optimal policy parameters Θ∗={θ1∗,…,θM∗}.

  •   2:

    Hyperparameters: training iterations K, rollouts per iteration Broll, horizon T, PPO epochs E, trajectories per minibatch Btraj, discount γ, GAE parameter λ, entropy coefficient μ, PPO clip ϵ.

  •   3:

    Initialize M actor policies {πθm}m=1M and a centralized critic Vϕ; set iteration k=0.

  •   4:

    while  k<K  do

  •   5:

          Clear replay buffer D.

  •   6:

          Rollout: collect on-policy data.

  •   7:

          for rollout index j=1 to Broll do

  •   8:

                Initialize environment state sj(1).

  •   9:

                Run Algorithm 1 (PA-BAC) using {pbm,j(1)}, {pun,j(1)}, {w(un)} to obtain α∗(1).

  • 10:

                for t=1 to T do

  • 11:

                      Set global observation om,j(t)=sj(t) for all agents.

  • 12:

                      Sample each agent action am,j(t)∼πθmk(·∣sj(t)); form joint action aj(t).

  • 13:

                      Execute actions and evolve the environment.

  • 14:

                      UAV positions pbm,j(t+1)=pbm,j(t)+am,j(t)·Δt.

  • 15:

                      Update ship positions according to mobility models.

  • 16:

                      Compute energy and update remaining energy according to energy model.

  • 17:

                      Obtain next state sj(t+1), call Algorithm 1 (PA-BAC) to solve α∗(t+1) under C1–C5 and compute Hw(t+1).

  • 18:

                      Compute reward rj(t) by (31)–(34).

  • 19:

                      Store transition (sj(t),aj(t),rj(t),donej(t),sj(t+1),logπθk(aj(t)|sj(t))) into D.

  • 20:

                end for

  • 21:

          end for

  • 22:

          GAE: compute advantages and value targets

  • 23:

          For each trajectory in D, compute δj,t by (35), A^j,t by (36), and R^j,t by (37).

  • 24:

          for epoch = 1 to E do

  • 25:

                Shuffle D and split into minibatches, each containing Btraj trajectories.

  • 26:

                for each minibatch B do

  • 27:

                      Randomly generate an agent index permutation (i1,i2,…,iM).

  • 28:

                      Initialize sequential weights Mj,ti1=A^j,t.

  • 29:

                      for τ=1 to M do

  • 30:

                            Compute PPO ratio rj,tiτ by (38).

  • 31:

                            Update actor parameters by minimizing L(θiτ) in (39).

  • 32:

                            if τ<M then

  • 33:

                                  update sequential weights by (40).

  • 34:

                            end if

  • 35:

                      end for

  • 36:

                      Update critic parameters by minimizing LV(ϕ) in (41).

  • 37:

                end for

  • 38:

          end for

  • 39:

          Set k←k+1

  • 40:

    end while