Algorithm 1 pFL -GMCO Algorithm

Require:  Private datasets ${\{{\mathcal{D}}_i=(x_i^S,x_i^A,x_i^R),Y_i\}}_{i=1}^N$ in multiple sites, pFL models ${\{{\mathcal{M}}_i={\mathcal{M}}_i^E,{\mathcal{M}}_i^C,{\mathcal{M}}_i^{GM}\}}_{i=1}^N$, number of epochs L, local learning rate $\alpha$, federated learning rate $\eta$, variance ${\sigma }^2$ of Gaussian noise, local learning rate $\alpha$, federated learning rate $\eta$. Ensure:  Optimal weights of pFL model ${\mathcal{M}}_i$ in site ${\mathcal{F}}_i$;   1:for $t=1:L$ do   2:    Local updating:   3:    for $i=1:N$ do   4:        Compute local classification loss: ${\mathcal{L}}_i^{cls}$   5:        Update pFL models locally: ${\theta }_i^t\leftarrow {\theta }_i^{t-1}-\alpha {\nabla }_{{\theta }_i^{t-1}}{\mathcal{L}}_i^{cls}$   6:    end for   7:    Federated adaptive aggregation:   8:    for $i=1:N$ do   9:        Compute gradient matching loss: ${\mathcal{L}}_i^{GM}$ 10:        Compute the adaptive aggregating weights: ${\lambda }_{i,j}$ 11:        Federated aggregation: ${\theta }_i^{t^{\prime }}\leftarrow {\theta }_i^t+\zeta {\sum }_{j=1(j\ne i)}^N{\lambda }_{i,j}\mathcal{G}\left({\theta }_j^t\right)$ 12:    end for 13:    Federated optimization: 14:    for $i=1:N$ do 15:        Compute gradient matching loss: ${\mathcal{L}}_i^{Fed-GM}$ 16:        Compute model contrastive loss: ${\mathcal{L}}_i^{con}$ 17:        Compute classification loss: ${\mathcal{L}}_i^{con}$ 18:        Compute federated optimization loss: ${\mathcal{L}}_i^{Fed}={\mathcal{L}}_i^{Fed-cls}+\beta {\mathcal{L}}_i^{Fed-GM}+\gamma {\mathcal{L}}_i^{con}$ 19:        Update personalized federated parameters: ${\theta }_i^{t*}\leftarrow {\theta }_i^{t^{\prime }}-\eta {\nabla }_{{\theta }_i^{t^{\prime }}}{\mathcal{L}}_i^{Fed}$ 20:    end for 21:    Update multimodal classifier: ${\theta }_i^{ML{P}^t*}\leftarrow {\theta }_i^{ML{P}^t}-\alpha {\nabla }_{{\theta }_i^{ML{P}^t}}{\mathcal{L}}_{CE}({\mathcal{M}}_i^{MLP}\left(z_{MCB}\right),Y_i)$ 22:end for 23:Return the pFL model ${\mathcal{M}}_i$ for each federated site.