Algorithm 1. PC-PPO Training Procedure

Input: Environment E, hyperparameters (Table 5)

Output: Pareto Policy Archive Π*

Initialize policy πθ, value heads Vφ, archive Π* ← ∅

for iteration = 1 to 5000 do

Sample preference ω~Uniform(Δ2)

Collect trajectory τ = {(s,a,R,s′)} for 2048 steps using πθ(·|s, ω)

for each objective i ∈ {1,2,3} do

Compute advantages Ai(t) via GAE(γ = 0.99, λ = 0.95)

end for

Scalarize: Aω(t) = ω1A1(t) + ω2A2(t) + ω3A3(t)

for epoch = 1 to K = 10 do

Sample minibatch B of size 256 from τ

Compute PPO clipped loss with ε = 0.2 using Aω

Compute value loss = ∑i ωi(Vi − Ri_target)2

Update θ, φ via Adam (lr = 3 × 10−4)

end for

if iteration mod 50 == 0 then

Evaluate πθ on 21 test preferences → {J1, J2, J3}

Add non-dominated policies to Π*; prune dominated

Compute hypervolume(Π*); check convergence

end if

end for

return Π*