1:  Setting the mentor learning rate ηt and mentee learning rate ηs, client number N

2:  Setting the hyperparameters Tstart and Tend

3:  for each client i (in parallel) do

4:       Initialize parameters ${\mathrm{\Theta }}_i^t$, Θs

5:       repeat

6:           ${\mathbf{g}}_i^t$,gi=LocalGradients(i)

7:           ${\mathrm{\Theta }}_i^t\leftarrow {\mathrm{\Theta }}_i^t-{\eta }_t{\mathbf{g}}_i^t$

8:           gi ← Ui∑iVi

9:           Clients encrypt Ui, ∑i, Vi

10:         Clients upload Ui, ∑i, Vi to the server

11:         Server decrypts Ui, ∑i, Vi

12:         Server reconstructs gi

13:         Global gradients g ← 0

14:         for each client i (in parallel) do

15:               g = g + gi

16:         end for

17:         g ← U∑V

18:         Server encrypts U, ∑, V

19:         Server distributes U, ∑, V to user clients

20:         Clients decrypt U, ∑, V

21:         Clients reconstructs g

22:         Θs ← Θs − ηsg/N

23:     until Local models converges

24:  end for

LocalGradients(i):

25:  Compute task losses ${\mathcal{L}}_{t,i}^t$ and ${\mathcal{L}}_{s,i}^t$

26:  Compute losses ${\mathcal{L}}_{t,i}^d$, ${\mathcal{L}}_{s,i}^d$, ${\mathcal{L}}_{t,i}^h$, and ${\mathcal{L}}_{s,i}^h$

27:  ${\mathcal{L}}_i^t\leftarrow {\mathcal{L}}_{t,i}^t+{\mathcal{L}}_{t,i}^d+{\mathcal{L}}_{t,i}^h$

28:  ${\mathcal{L}}_i^s\leftarrow {\mathcal{L}}_{s,i}^t+{\mathcal{L}}_{s,i}^d+{\mathcal{L}}_{s,i}^h$

29:  Compute local mentor gradients ${\mathbf{g}}_i^t$ from ${\mathcal{L}}_i^t$

30:  Compute local mentee gradients gi from ${\mathcal{L}}_i^s$

31:  return ${\mathbf{g}}_i^t,{\mathbf{g}}_i$