Algorithm 1.

FedAvg for a small number of clients. K—number of clients, n_k—number of samples on kth client, R—number of communication rounds, E—number of local epochs in a round, η—local learning rate.

1:  procedure Server-side

2:       Initialize ω←ω0 ⊳ Initialize model weights

3:       for r∈1, .., R do

4:        for k∈1, .., K do < in parallel>

5:             ${\omega }_r^k\leftarrow$ Client-side(k, ω) ⊳ Parallel local optimization

6:        end for

7:        $\omega \leftarrow \sum _{k=1}^K\frac{n_k}{n}{\omega }_r^k$ ⊳ Weighted average of client weights

8:       end for

9:      end procedure

10:      procedure Client-side(k, ω)

11:        for i∈1, .., E do

12:        for batch b do

13:             ω←ω−η∇F(ω, b) ⊳ Batch gradient descent

14:        end for

15:        end for

16:        return ω

17:  end procedure