Algorithm 1 Flow-Multi with Pareto Masking (mini-batch)

Require:  ${\pi }_{\theta }$: Online policy model (initialized from pretrained weights)   ${\pi }_{\mathrm{ref}}$: Reference model   $\mathcal{Q}$: prompt batch   K: group size (default 4)   ${\sigma }_t=a\sqrt{t/(1-t)}$: noise schedule   ${\mu }_{\theta }$ (Equation (3): drift; $\beta$: KL weight; LoRA rank r; reward weights $w\in {\mathbb{R}}^4$ (default $w_d=\frac{1}{4}$)) Ensure: updated policy ${\pi }_{\theta }$ (LoRA-only updates)   1: Initialize policy ${\pi }_{\theta }$ (LoRA on Q/K/V/O); set reference ${\pi }_{\mathrm{ref}}\leftarrow \mathrm{stopgrad}\left({\pi }_{\theta }\right)$   2: for epoch $=1,\dots$ do   3:       Sample a mini-batch of prompts ${\left\{q_i\right\}}_{i=1}^P$ from $\mathcal{Q}$   4:       for each prompt $q_i$ do   5:             Sampling (SDE rollout): generate K images ${\left\{x_{i,k}\right\}}_{k=1}^K$ using (2)   6:             Reward vector for each $x_{i,k}$: $R\left(x_{i,k}\right)=(r_{\mathrm{aesth}},r_{\mathrm{pref}},r_{\mathrm{align}},r_{\mathrm{comp}})$   7:             Group-wise Pareto non-dominated mask $M_{i,k}\in \{0,1\}$                 (Section 3.5)   8:             Scalarization $r_{i,k}^{\mathrm{comb}}={\sum }_{d=1}^4{w}^{\left(d\right)}{r}_{i,k}^{\left(d\right)}$                          (Section 3.6)   9:       end for 10:       Gather survivors $S=\{(i,k):M_{i,k}=1\wedge \mathrm{valid}(i,k)\}$ across devices; if $S=\varnothing$ then fallback to top-1 by $r^{\mathrm{comb}}$ 11:       Compute survivors-only stats ${\mu }_S$, ${\sigma }_S$ 12:       Advantage (masked) $A_{i,k}={\displaystyle \frac{r_{i,k}^{\mathrm{comb}}-{\mu }_S}{{\sigma }_S}}·M_{i,k}$ 13:       Compute likelihood ratios ${\rho }_{i,k}={\displaystyle \frac{{\pi }_{\theta }(o_{i,k}\mid q_i)}{{\pi }_{{\theta }_{\mathrm{old}}}(o_{i,k}\mid q_i)}}$ 14:       PPO-style objective with Pareto mask                   (Equation (9), Section 3.7) 15:       KL penalty to reference (stepwise or sequence):               (Equation (10), Section 3.7) 16:       Update $\theta \leftarrow \theta +\eta {\nabla }_{\theta }{J}_{\mathrm{GRPO}}\left(\theta \right)$                          ▹ LoRA params only 17:       Logging: Pareto keep ratio $\left({\sum }_{i,k}{M}_{i,k}/(P·K)\right)$, each reward mean/var, A stats, KL, loss terms, learning curves 18:       Periodically refresh reference ${\pi }_{\mathrm{ref}}\leftarrow \mathrm{stopgrad}\left({\pi }_{\theta }\right)$ 19: end for