Communication-efficient federated learning

Significance

Federated learning (FL) is an emerging paradigm that enables multiple devices to collaborate in training machine learning (ML) models without having to share their possibly private data. FL requires a multitude of devices to frequently exchange their learned model updates, thus introducing significant communication overhead, which imposes a major challenge in FL over realistic networks that are limited in computational and communication resources. In this article, we propose a communication-efficient FL framework that enables edge devices to efficiently train and transmit model parameters, thus significantly improving FL performance and convergence speed. Our proposed FL framework paves the way to collaborative ML in large-scale networking systems such as Internet of Things networks.

Abstract

Federated learning (FL) enables edge devices, such as Internet of Things devices (e.g., sensors), servers, and institutions (e.g., hospitals), to collaboratively train a machine learning (ML) model without sharing their private data. FL requires devices to exchange their ML parameters iteratively, and thus the time it requires to jointly learn a reliable model depends not only on the number of training steps but also on the ML parameter transmission time per step. In practice, FL parameter transmissions are often carried out by a multitude of participating devices over resource-limited communication networks, for example, wireless networks with limited bandwidth and power. Therefore, the repeated FL parameter transmission from edge devices induces a notable delay, which can be larger than the ML model training time by orders of magnitude. Hence, communication delay constitutes a major bottleneck in FL. Here, a communication-efficient FL framework is proposed to jointly improve the FL convergence time and the training loss. In this framework, a probabilistic device selection scheme is designed such that the devices that can significantly improve the convergence speed and training loss have higher probabilities of being selected for ML model transmission. To further reduce the FL convergence time, a quantization method is proposed to reduce the volume of the model parameters exchanged among devices, and an efficient wireless resource allocation scheme is developed. Simulation results show that the proposed FL framework can improve the identification accuracy and convergence time by up to 3.6% and 87% compared to standard FL.

Machine learning (ML) uses data to realize intelligent and autonomous decision-making and inference. ML algorithms have been used in a wide variety of areas, such as computer vision, natural language processing, medical imaging, and communications (1–4). Data are often collected on devices at the edges of networks: Images and text messages are often generated and stored on smartphones; biomedical signals are collected by medical and wearable devices, and often stored on hospital servers; various forms of signals are recorded by Internet of Things systems and sensors.

As massive amounts of data are typically required to train an ML model, such as deep neural networks, centralized ML algorithms must collect training data from edge devices for training purposes. For example, to train an ML model for medical diagnosis, a central controller (CC) must collect the medical data from multiple hospitals. Nonetheless, in some applications, such as the aforementioned example of training an ML diagnosis system for medical data, the edge devices may not be willing to share their data, due to privacy concerns and regulations. Furthermore, conveying large volumes of aggregated data by many mobile devices may induce a notable burden on the communication infrastructure. These considerations gave rise to the need for ML algorithms that train an ML model in a distributed fashion, such that edge devices can contribute to the learning procedure without sharing their data.

Federated learning (FL) proposed in ref. 5 is a distributed learning algorithm that enables edge devices to jointly train a common ML model without being required to share their data. The FL procedure relies on the ability of each device to train an ML model locally, based on its data, while having the devices iteratively exchanging and synchronizing their local ML model parameters with each other in a manner orchestrated by a CC unit (6). Due to its unique features, FL has been applied in a wide variety of practical applications such as mobile keyboard prediction [e.g., Google (7)], speaker and command recognition [e.g., Apple (8)], and data silos for insurance companies [e.g., WeBank (9)].

However, implementation of FL in practical applications faces several challenges which stem from its distributed operation, which is fundamentally different from traditional centralized ML algorithms (10). These challenges include 1) communication overhead induced by the repetitive model parameter exchanges; 2) device hardware heterogeneity, as each device may have different computational capabilities; 3) data heterogeneity, as each device can access a relatively small and personalized dataset and may thus train an ML model which is biased toward its own data; and 4) privacy and security issues, which follow from the fact that the learning procedure is carried out over multiple individual devices. Among these challenges, the communication overhead constitutes a major bottleneck due to the following reasons. First, FL is trained by an iterative process, and hence the time it takes to learn, that is, the convergence time, depends on both the optimization procedure, for example, the number of training steps, as well as the FL parameter transmission delay per training step. Second, FL training is potentially implemented by millions of edge devices, and each device must iteratively share its large-size FL parameters with a CC. Therefore, for FL implemented over a realistic network with limited computational and communication resources, its FL parameter transmission delay may be much larger than the time it takes the devices to train their local ML models. Therefore, it is necessary to design a communication-efficient FL framework that can significantly improve both convergence speed and model accuracy, thus allowing its application to training large-scale ML models over millions of edge devices.

A number of existing works, including refs. 11–23, have studied the design of communication-efficient FL algorithms. However, the majority of these works focus on optimization of FL in a single aspect such as device selection and scheduling (11–13), FL model parameter update and transmission (14–18), or network resource management (20–22). In this study, we propose a communication-efficient FL framework that tackles multiple causes for communication delay, by jointly optimizing the device selection, FL model parameter transmission, and network resource management. In particular, we first propose a probabilistic device selection scheme which allows the devices that can significantly improve the convergence speed and training loss to have higher probabilities for ML model transmission. Then, a quantization method is designed to reduce the data size of the ML parameters exchanged among devices, thus improving FL convergence speed. In addition, for the selected devices, a wireless resource allocation scheme is developed to further improve their transmission data rates, thus reducing the FL transmission delay at each learning step. Finally, we analyze the convergence of our proposed FL framework. Simulation results based on real-world data demonstrate the performance of our proposed FL framework and its ability to allow accurate and fast collaborative training of multiple edge devices in a federated manner.

Communication-Efficient FL

Basic Concepts of FL.

Before introducing the proposed FL framework, we first explain the basic FL training process. Consider a set $\mathcal{U}$ of $U$ devices and a CC cooperatively implementing an FL algorithm (see SI Appendix, section 1 for further details). Letting $\mathit{b}$ be the model parameters, the objective of FL is given as follows:

$$\underset{\mathit{b}}{min}\frac{1}{N}\sum _{i=1}^U\sum _{n\in {\mathcal{N}}_i}f\left(\mathit{b},{\mathit{x}}_{i,n},{\mathit{y}}_{i,n}\right).$$ [1]

Here, ${\mathcal{N}}_i$ is a set of $N_i$ training data samples of each device $i$, where each training data sample $n$ of device $i$ consists of an input vector ${\mathit{x}}_{i,n}$ and output vector ${\mathit{y}}_{i,n}$, $f$ is a local objective function, and $N={\sum }_{i=1}^U{N}_i$ is the total number of training samples. FL tackles the problem in Eq. 1 using iterative distributed optimization orchestrated by the CC.

A common FL optimization algorithm is the local stochastic gradient descent (SGD) method (24), where, at every FL iteration, each device $i$ uses $\tau$ SGD iterations to train its local model ${\mathit{o}}_i$. In particular, the update process at iteration $t$ begins with the CC sharing the global model ${\mathit{b}}_t$ with the devices, which first set their local models to ${\mathit{o}}_{i,t+1}^0={\mathit{b}}_t$, followed by $\tau$ local SGD steps,

$${\mathit{o}}_{i,t+1}^k={\mathit{o}}_{i,t+1}^{k-1}-\frac{{\lambda }_{t+1}^k}{N_{i,t+1}^k}\sum _{n\in {\mathcal{N}}_{i,t+1}^k}\nabla f\left({\mathit{o}}_{i,t+1}^{k-1},{\mathit{x}}_{i,n},{\mathit{y}}_{i,n}\right),$$ [2]

where ${\mathit{o}}_{i,t+1}^k$ is the local FL model of device $i$ at $k$ local updates of FL iteration $t+1$, ${\lambda }_{t+1}^k$ is the learning rate, ${\mathcal{N}}_{i,t+1}^k$ is the training dataset of user $i$ at local SGD step $k$ of iteration $i$, which consists of $N_{i,t+1}^k$ training samples randomly selected from the local training dataset ${\mathcal{N}}_i$, and $\nabla f\left(\cdot \right)$ is the gradient of the objective function with respect to the model parameters.

When each device completes $\tau$ steps of local FL training via Eq. 2, it transmits its trained local FL model parameters, ${\mathit{o}}_{i,t+1}^{\tau }-{\mathit{b}}_t$, to the CC. The CC aggregates the received local FL parameters of all of the participating devices into a global FL model as follows:

$${\mathit{b}}_{t+1}=\frac{1}{N}\sum _{i=1}^U{N}_i\left({\mathit{o}}_{i,t+1}^{\tau }-{\mathit{b}}_t\right)+{\mathit{b}}_t.$$ [3]

The local SGD algorithm implements multiple iterations of the local FL model update in Eq. 2 and global FL model update in Eq. 3. At convergence, we have $\mathit{b}={\mathit{o}}_1=\cdots ={\mathit{o}}_U$.

Communication Model.

From the FL training process, we can see that the devices and the CC must iteratively exchange their FL parameters over wireless links.

The devices communicate with the CC over an uplink wireless channel, which comprises $R$ resource blocks (RBs), for example, frequency bins, each of bandwidth $B$. We consider frequency flat fading channels with Gaussian noise and interference. Let $s_i\left(t\right)$ be the signal transmitted by device $i$, and let $z_i\left(t\right)$ be the corresponding channel output at the CC. When RB $r$ is assigned to device $i$, the channel input–output relationship is given by

$$z_i\left(t\right)=h_i{s}_i\left(t\right)+w_i\left(t\right)+v_r\left(t\right).$$ [4]

Here, $h_i={𝜗}_i{d}_i^{-2}$ is the channel gain between device $i$ and the CC, where $d_i$ is the distance between device $i$ and the CC, and ${𝜗}_i$ is the Rayleigh fading parameter, assumed to be known to each device and the CC; $w_i\left(t\right)$ is additive white Gaussian noise with power spectral density $N_0$; and $v_r\left(t\right)$ is the interference over RB $r$, whose energy is $I_r$. Finally, the channel input $s_i\left(t\right)$ is constrained to maximal transmit power of $P$.

The channel model in Eq. 4 implies that FL is carried out over wireless channels which are limited in energy and bandwidth, as well as subject to noise and interference. The ML models conveyed over the channel in FL typically comprise a very large number of parameters, particularly when representing deep neural networks. Combining this with the limited wireless resources, such as energy and bandwidth, implies that the convergence time of FL implemented over realistic wireless networks depends not only on the number of iterations but also on the FL parameter transmission time per iteration. Therefore, it is necessary to design a communication-efficient FL algorithm that trains accurate models using minimal convergence time.

Next, we introduce our proposed communication-efficient FL algorithm that consists of three components: 1) probabilistic device selection for limiting the number of participating devices, 2) universal FL parameter compression method for reducing the volume of data conveyed at each FL iteration, and 3) a resource allocation scheme for optimizing the usage of the wireless channel.

Probabilistic Device Selection Scheme.

Due to limited network resource and energy-constrained devices, only a subset of devices can participate in FL at each iteration. Hence, it is necessary to design a device selection method that selects the devices that can significantly improve convergence speed and training loss to execute the FL. To introduce the proposed device selection scheme, we first define

$${\mathit{g}}_{i,t}\triangleq {\sum }_{k=1}^{\tau }{\sum }_{n\in {\mathcal{N}}_{i,t+1}^k}\nabla f\left({\mathit{o}}_{i,t+1}^{k-1},{\mathit{x}}_{i,n},{\mathit{y}}_{i,n}\right).$$ [5]

Then, at each iteration, the proposed selection method consists of the following three steps: 1) Each device calculates ${\mathit{g}}_{i,t}$ and its norm $‖{\mathit{g}}_{i,t}‖$, 2) device $i$ sends $‖{\mathit{g}}_{i,t}‖$ to the CC, and 3) the CC determines the device connection probability, which is given by

$$p_{i,t}=\frac{\alpha ‖{\mathit{g}}_{i,t}‖}{\sum _{i=1}^U‖{\mathit{g}}_{i,t}‖}+\left(1-\alpha \right)\frac{\underset{j\in \mathcal{U}}{max}{d}_j-d_i}{\sum _{i=1}^U\left(\underset{j\in \mathcal{U}}{max}{d}_j-d_i\right)},$$ [6]

where $d_i$ is the distance between device $i$ and the CC, and $\alpha \in \left[\mathrm{0,1}\right]$.

Using the model updates ${\mathit{g}}_{i,t}$ to determine the participation probability of each device in Eq. 6 implies that users with increased contribution to the global model are more likely to participate. This is because ${\mathit{g}}_{i,t}$ captures the change in the local FL model of device $i$ at learning step $t$. Consequently, the proposed setting is expected to increase the convergence speed of the global FL model to the optimal global FL model compared to uniform selection as in ref. 5. Furthermore, $d_i$ determines the FL parameter transmission distance, thus affecting transmission delay. As a result, the second term in Eq. 6 accounts for the transmission delay per learning step. The balance between these two terms is dictated by the setting of the hyperparameter $\alpha$: Increasing $\alpha$ enables the CC to select the devices that are more likely to contribute toward faster FL model updates, while decreasing $\alpha$ implies that the CC is expected to select the devices that transmit their parameters with reduced delay.

The CC determines the devices that transmit their model updates, based on the probability distribution ${\mathit{p}}_t=\left[p_{1,t},\dots ,p_{U,t}\right]$. The proposed device selection method guarantees all devices have a chance to participate in the FL training process, which enables the proposed FL algorithm to find the optimal FL model. Since the connection probability $p_{i,t}$ of each device $i$ at iteration $t$ depends on both its expected contribution to the global model and its transmission delay, a device with a large connection probability is likely to improve both convergence speed and training loss. An additional term that affects the convergence speed and is not accounted for in Eq. 6 is the time required by each device to carry out its local update. While this duration can change considerably between devices, the FL parameter transmission time, which is accounted for in Eq. 6, tends to be the dominant cause of delay in FL carried out over resource-constrained wireless networks. It is noted, however, that one can decrease the number of local SGD updates to further decrease the time used for local FL model updating, although possibly affecting the convergence accuracy, as discussed in ref. 25.

Universal FL Model Parameter Compression.

Due to limited energy and bandwidth resources, devices may not be able to directly transmit the large-sized FL parameters to the CC. This gives rise to the need to compress the local FL model updates transmitted from the devices to the CC, and, particularly, to represent their updates using a limited number of bits, such that the CC can still accurately generate the global model via Eq. 3. However, due to the heterogenous nature of the training data available at the users, a priori knowledge of the underlying distribution of the local FL model parameters is not available at the device side, which motivates compressing the local FL model as a form of universal quantization. In addition, the fact that the CC and the users communicate repeatedly allows them to share a source of common randomness, for example, a random seed, enabling the participating users to implement low-distortion discretization in a universal manner via random lattice quantization (15, 26).

The resulting model compression scheme consists of the following steps: As a preliminary step, an $L$-dimensional lattice $\mathcal{L}$ is fixed. Upon FL model parameter transmission, each device normalizes its local FL model parameter ${\mathit{o}}_{i,t}^{\tau }$ by $\zeta$ times its norm, where $\zeta >0$ is a given scaling factor. The normalization result is divided into $M\hspace{0.17em}L$-sized vectors, to which the users add a random dither signal randomized in an independent and identically distributed fashion from a uniform distribution over the basic cell of the lattice. The dithered signal is discretized by projecting to the nearest lattice point, and the discrete quantity is further compressed prior to transmission, using lossless entropy coding. The CC decompresses the model updates by recovering the lattice point, and subtracting the dither signal from it. The fact that the devices and the CC can generate the same dither signals relies upon their ability to share a source of common randomness, that is, a random seed (see SI Appendix, section 2 for further details).

Network Resource Management Scheme.

Since the compressed local FL models must be transmitted over wireless links, we also optimize the wireless resource allocation in order to further reduce transmission time. To that aim, recall that the total number of uplink RBs is $R$, which implies that only $R$ devices can participate in the FL training process at each iteration. The data rate of device $i$ transmitting its compressed local FL model to the CC is $c_i^{\mathrm{U}}\hspace{0.17em}\left({\mathit{\chi }}_{i,t}\right)=\sum _{r=1}^R{\chi }_{i,t}^{r}B{\log }_2\left(1\hspace{0.17em}+\hspace{0.17em}P{h}_i/\hspace{0.17em}\left(I_r\hspace{0.17em}+\hspace{0.17em}B{N}_0\right)\right)$ . Here, ${\chi }_{i,t}^r\in \left\{\mathrm{0,1}\right\}$ is the RB allocation index, with ${\chi }_{i,t}^r=1$ implying that RB $r$ is allocated to device $i$ at iteration $t$; otherwise, we have ${\chi }_{i,t}^r=0$, and ${\mathit{\chi }}_{i,t}=\left[{\chi }_{i,t}^1,\dots ,{\chi }_{i,t}^R\right]$. Then, the transmission delay at each FL iteration is given by $\underset{i\in {\mathcal{U}}_{{\mathit{p}}_t}}{max}Z/c_i^{\mathrm{U}}\left({\mathit{\chi }}_{i,t}\right)$, where ${\mathcal{U}}_{{\mathit{p}}_t}$ is the subset of $R$ devices that transmit their compressed local FL model parameters to the CC at iteration $t$, which is determined by ${\mathit{p}}_t$. Let $Z$ denote the number of bits that are used to represent the FL parameters that each device needs to transmit to the CC. Hence, the transmission delay minimization problem can be formulated as

$$\underset{{\mathit{\chi }}_{i,t},i\in {\mathcal{U}}_{{\mathit{p}}_t}}{min}\underset{i\in {\mathcal{U}}_{{\mathit{p}}_t}}{max}\frac{Z}{c_i^{\mathrm{U}}\left({\mathit{\chi }}_{i,t}\right)}$$ [7]

$$\mathrm{s}.\hspace{0.17em}\mathrm{t}.\hspace{0.17em}{\chi }_{i,t}^r\in \left\{\mathrm{0,1}\right\},\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\forall i\in {\mathcal{U}}_{{\mathit{p}}_t},r\in \mathcal{R},$$ [7a]

$$\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\sum _{i\in {\mathcal{U}}_{{\mathit{p}}_t}}{\chi }_{i,t}^r=1,\forall r\in \mathcal{R},$$ [7b]

$$\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\sum _{r=1}^R{\chi }_{i,t}^r=1,\forall i\in {\mathcal{U}}_{{\mathit{p}}_t},$$ [7c]

where $\mathcal{R}$ is the set of RBs. To solve this optimization problem, we first transform it to an equivalent integer linear programming problem (see SI Appendix, section 3 for further details). Then, we solve it using interior-point methods (27), to obtain the optimized allocation.

Convergence Analysis.

To analyze the convergence of the proposed FL algorithm, we first assume that the local loss functions $f_i\left({\mathit{o}}_i\right)\triangleq {\sum }_{n\in {\mathcal{N}}_i}f\left({\mathit{o}}_i,{\mathit{x}}_{i,n},{\mathit{y}}_{i,n}\right)$ satisfy the following conditions: 1) The expected squared ${\ell }_2$ norm of the random vector $\nabla f_i\left(\mathit{o}\right)$, is bounded by some ${\xi }^2>0$; 2) the local loss functions $\left\{f_i\left(\cdot \right)\right\}$ are ${\rho }_s$-smooth and ${\rho }_c$-strongly convex; and 3) the probabilistic device selection scheme selects the set ${\mathcal{U}}_{{\mathit{p}}_t}$ of devices, such that $\left|{\mathcal{U}}_{{\mathit{p}}_t}\right|=R$ and ${\mathcal{U}}_{{\mathit{p}}_t}$ is uniformly distributed over all $R$-sized subsets of $\mathcal{U}$. While assumptions 1 through 3 may not accurately reflect practical FL systems, particularly when training deep models, and can even be in conflict (28), they are often employed in distributed learning studies (24, 25). Therefore, analyzing the convergence of our proposed scheme when these assumptions are satisfied facilitates identifying the contribution of the introduced methods for improving communication efficiency to the convergence profile, compared to FL carried out over ideal communication channels, as studied in refs. 24 and 25.

We let ${\overline{\sigma }}_{\mathcal{L}}^2$ be the normalized second-order moment of the lattice $\mathcal{L}$ (29), and define the heterogeneity mismatch as $\psi \triangleq f\left({\mathbf{b}}^{*}\right)-{\sum }_{i=1}^U{\alpha }_i\underset{\mathbf{b}}{min}{f}_i\left(\mathbf{b}\right)$, where ${\alpha }_i=N_i/N$, and ${\mathit{b}}^{*}$ is the target optimal global FL model that minimizes the objective function in Eq. 1. The resulting convergence bound of the proposed communication efficient FL is stated in the following theorem:

Theorem 1. Set $\gamma =\tau max\left(\mathrm{1,4}{\rho }_s/{\rho }_c\right)$ and consider local SGD with $\tau$ local iterations, step-size ${\lambda }_t^k=\frac{\tau }{{\rho }_c\left(t\tau +k+\gamma \right)}$, and mini-batch size $\left|{\mathcal{N}}_{i,t}^k\right|=1$. Under this setting, when assumptions 1 through 3 are satisfied, it holds that

$$\hspace{0.17em}\mathbb{E}\left\{f\left({\mathbf{b}}_t\right)\right\}\hspace{0.17em}-\hspace{0.17em}f\left({\mathbf{b}}^{*}\right)\hspace{0.17em}\le \hspace{0.17em}\frac{{\rho }_{s}max\left(\frac{\underset{c}{\overset{2}{\rho }}\hspace{0.17em}+\hspace{0.17em}{\tau }^2\varpi }{\tau {\rho }_c},\gamma \Vert {\mathbf{b}}_0\hspace{0.17em}-\hspace{0.17em}{\mathbf{b}*\Vert }^2\right)}{2\left(t\tau \hspace{0.17em}+\hspace{0.17em}\gamma \right)},$$ [8]

where $\varpi \triangleq \left(1+4M{\zeta }^2{\overline{\sigma }}_{\mathcal{L}}^2{\tau }^2\right){\xi }^2{\sum }_{i=1}^U{\alpha }_i^2+6{\rho }_s\psi +8{\left(\tau -1\right)}^2{\xi }^2+\frac{4\left(U-R\right)}{R\left(U-1\right)}{\tau }^2{\xi }^2$.

The proof of Theorem 1 is detailed in SI Appendix, section 4. Theorem 1 proves that the asymptotic convergence rate of our proposed communication efficient FL is $\mathcal{O}\left(1/t\right)$. This is observed by noting that the nominator in Eq. 8 comprises constants which are determined by the FL scheme, and do not change with $t$, and thus the overall bound decays asymptotically as $1/t$, which is the same as FL over ideal error-free channels (e.g., ref. 25, theorem 1). This indicates the ability of the proposed communication-efficient FL scheme to overcome the challenges associated with the need to communicate over shared noisy and throughput-limited channels. Comparing the instantaneous bound in Eq. 8, that is, for a fixed $t$, to the corresponding bound for FL over ideal links in ref. 25, theorem 1, reveals the effect of the components of our proposed communication-efficient FL scheme on the convergence profile. In particular, each component induces an additive term to the constant $\varpi$, which can be written as $\varpi =\mathcal{O}\left({\tau }^2\left(1+{\overline{\sigma }}_{\mathcal{L}}^2{\sum }_{i=1}^U{\alpha }_i^2+\left(U-R\right)/R\left(U-1\right)\right)\right)$. Increasing $\varpi$ is translated to a decrease in the convergence rate. For instance, the requirement to limit the number of participants induces the additive term proportional to $\left(U-R\right)/R\left(U-1\right){\tau }^2$, that is, $4\left(U-R\right)/R\left(U-1\right){\tau }^2{\xi }^2$, whose contribution decreases as the number of participating devices $R$ increases. Quantizing the local FL model parameters results in the term proportional to ${\overline{\sigma }}_{\mathcal{L}}^2{\sum }_{i=1}^U{\alpha }_i^2$, namely, $4M{\zeta }^2{\overline{\sigma }}_{\mathcal{L}}^2{\tau }^2{\xi }^2{\sum }_{i=1}^U{\alpha }_i^2$, which is smaller for denser lattices, that is, when more bits are allowed to be conveyed. It is also observed that the quantization error is mitigated by averaging the local FL models at the CC side. This follows from the fact that this error term vanishes as $U$ grows when the datasets are of similar sizes, and, particularly, when $max{\alpha }_i\to 0$ as $U\to \infty$. Finally, by repeating the arguments in ref. 25, it can be shown that the number of communication rounds required to achieve a given accuracy, which dictates the convergence speed and is reduced by the proposed scheme for improving communication efficiency, has a finite minimum with respect to the number of local SGD steps $\tau$.

Extensions.

The device selection scheme, universal FL model parameter compression method, and RB allocation technique are designed for learning via the federated averaging algorithm introduced from Eqs. 1–3. An interesting extension of the proposed communication-efficient FL scheme would be to adopt the proposed techniques for alternative distributed optimization algorithms such as in refs. 11 and 30–37. Furthermore, one can also consider the queuing delays and FL parameter transmission for the FL algorithms implemented over communication networks. A further extension involves enhancing the proposed device selection scheme for scenarios with additional forms of diversity among the devices which may faithfully reflect some FL applications. Such scenarios include 1) devices with varying computational power, 2) devices which can be clustered based on the distribution of their local data as in ref. 17, and 3) devices implementing different numbers of local SGD updates. In particular, to extend the proposed device selection scheme for scenario 3, one only needs to consider the devices that complete the local model update in Eq. 6. Finally, an analysis of how the value of the hyperparameter $\alpha$ in Eq. 6 affects FL convergence and how to optimize $\alpha$ to improve FL performance is an interesting topic for further study.

Results

Next, we evaluate our proposed FL method for handwritten digit identification, object recognition, and finger movement detection. For comparison purposes, we use two baselines: baseline a, an FL algorithm that uses the proposed device selection and resource allocation schemes without the compression of local FL model parameters; and baseline b, a standard FL algorithm in ref. 6 that randomly determines device selection and resource allocation without compressing the local FL model parameters of each device. All simulation settings are detailed in Materials and Methods.

Handwritten Digit Identification.

We first evaluate the proposed communication-efficient FL scheme by identifying handwritten digits from 0 to 9, as shown in Fig. 1. In particular, Fig. 1A shows an example of implementing the proposed FL method for handwritten digit identification. In Fig. 1A, the black digits are the identification results of the proposed FL scheme, while the red digit is the wrong identification result. From Fig. 1A, we can see that the proposed FL algorithm can correctly identify the handwritten digits. Fig. 1B shows how the identification accuracy changes as the number of FL learning steps increases. From Fig. 1B, we can see that the identification accuracy achieved by the proposed algorithm is slightly lower than that of baselines a and b. This is because our proposed quantization-based FL transmission slightly affects the identification accuracy. In Fig. 1C, we show how the convergence time changes as the number of selected devices varies. From Fig. 1C, we can see that, as the number of selected devices increases, the convergence time of all considered FL algorithms increases. This is because the FL model transmission delay at each learning step depends on the device with the largest transmission delay. As the number of selected devices increases, the maximum transmission delay among all devices increases.

Fig. 1.

Implementation of FL for handwritten digit identification. (A) An example of implementing FL for handwritten digit identification. (B) Identification accuracy changes as the number of learning steps varies. (C) Convergence time changes as the number of selected devices varies. (D) Identification accuracy changes as the number of selected devices varies. (E) Sum energy consumption of all devices changes as the number of selected devices varies. (F) Convergence of the FL algorithms with various quantization methods.

From Fig. 1 B and C, we can also see that baseline a achieves better identification accuracy and convergence time than baseline b. This is due to the fact that the proposed device selection scheme can select the most appropriate devices to join the FL training process. We also observe, in Fig. 1 B and C, that, although the proposed scheme achieves lower identification accuracy, it significantly reduces the convergence time by up to 85% and 87% compared to baselines a and b. These gains stem from the fact that the proposed FL uses advanced quantization methods specifically tailored for its distributed operation to compress the local FL model parameters, thus significantly reducing the size of the local FL model that is transmitted over wireless links. Fig. 1C also shows that, for a simple handwritten digit identification task, the standard FL (baseline b) needs more than 800 s to converge, which further illuminates the need for designing a communication-efficient FL algorithm.

Fig. 1D shows how the identification accuracy changes as the number of selected devices varies. From Fig. 1D, we can see that baseline a can achieve better performance than baseline b. This is because baseline a uses the proposed device selection scheme, thus improving identification accuracy. Fig. 1D also shows that, as the number of selected devices increases, the gap between the proposed FL and baseline a decreases. This is due to the fact that, as the number of selected devices increases, the effects of compression errors on the global FL model update decrease, which verifies the analytical observation made based on Theorem 1.

In Fig. 1E, we show how the sum energy consumption of all devices changes as the number of selected devices varies. From Fig. 1E, we see that the proposed FL significantly reduces the energy that all devices used for FL parameter transmission. This is because we jointly optimize wireless resource allocation, device selection, and FL parameter compression.

Fig. 1F shows the convergence of the FL algorithms that use various quantization methods. Our scheme with two-dimensional lattices is compared to its implementation with conventional scalar quantizers, the Quantized SGD (QSGD) FL algorithm proposed in ref. 38, and the combination of uniform quantization with random rotations proposed in ref. 39. From Fig. 1F, we see that the proposed algorithm outperforms the other quantization-based FL methods. This is because the proposed FL algorithm implements substractive dithered lattice quantization, which reduces the distortion induced by quantizing the FL model parameters and mitigates its effect on the aggregated global model.

Object Recognition in Images.

Next, we demonstrate how the proposed FL algorithm identifies 10 class objects in images, as shown in Fig. 2. Fig. 2A shows an example of implementing the proposed FL for 10-class object identification. In this figure, the digits are the labels used to represent each class of objects. The black digits are the correct identification results, while the red digits are the wrong identification results. For example, for the last object, the proposed FL identifies a cat (label 4) as a dog (label 2). Fig. 2B shows how the identification accuracy changes as the number of learning steps varies. From Fig. 2B, we can see that the proposed FL algorithm achieves almost the same identification accuracy as baseline a, which is different from the case in Fig. 1B, that the proposed FL algorithm has a lower identification accuracy compared to baseline a. This is due to the fact that the size of weight matrices of a neural network model used for object recognition is much larger than that of a neural network model used for handwritten digit identification. As the size of the weight matrices of a neural network increases, the weight bias caused by quantization method decreases.

Fig. 2.

Implementation of FL for object recognition. (A) An example of implementing FL for object recognition. (B) Identification accuracy changes as the number of learning steps varies. (C) Convergence of the proposed FL changes as the value of $\alpha$ in Eq. 6 varies.

Fig. 2C shows how FL convergence changes as the value of $\alpha$ in Eq. 6 varies. From this figure, we see that the identification accuracy and convergence time both depend on $\alpha$. In particular, as $\alpha$ decreases, FL convergence time decreases. This is because the CC prefers to select the devices that can improve transmission time. However, identification accuracy does not decrease as $\alpha$ decreases. In particular, when $\alpha =0.6$, the proposed FL algorithm achieves a better identification accuracy compared to the proposed FL, with $\alpha =0.85$ or $\alpha =0.4$.

From Figs. 1C and 2C, we see that the convergence time of the FL algorithm used for objective identification is much larger than that of the FL algorithm used for handwritten digit identification. This is because the FL model used for objective identification is a convolutional neural network (CNN) that consists of 17 layers, while the FL model used for handwritten digit identification is a shallow neural network.

Finger Movement Detection.

Next, we introduce the use of the proposed FL method for processing actual medical data. In particular, the proposed FL method is implemented for detecting 15-class finger movements, as shown in Fig. 3A. Fig. 3B shows the signal samples used to represent the hand closed movement. In Fig. 3B, the signals have been processed by the windowing method, and, hence, the number of the signal samples used to represent each finger movement is 500.

Fig. 3.

Implementation of FL for finger movement detection. (A) Fifteen-class finger movements. (B) Signal samples used to represent the hand closed movement.

Fig. 4 shows the convergence of the FL methods used for finger movement detection. In this figure, non-FL is a method in which each device trains its ML model using its own dataset, and each device will not share its ML model with the CC. From Fig. 4, we can see that the proposed FL algorithm can improve the identification accuracy by up to 3.6%, 3.9%, and 30% compared to baseline a, baseline b, and non-FL algorithm. The 30% gain implies that using the FL algorithm can significantly improve the identification accuracy of an ML model.

Fig. 4.

Convergence of FL used for finger movement detection.

Materials and Methods

In our simulations, we consider a circular network area having a radius $r=500$ m with one CC at its center servicing $U=15$ uniformly distributed devices. The CC can allocate a total of $R=10$ RBs to all devices for local FL model transmission, and thus only 10 devices can transmit their trained local FL models to the CC at each FL iteration. Lattice quantization is carried out by setting $\mathcal{L}$ to be the two-dimensional hexagonal lattice (40), using four bits per sample, with a scale parameter of $2.8/\sqrt{M}$. The interference over RBs is given by $\mathit{I}=\left[0.02,0.03,0.04,0.05,0.06,0.07,0.08,0.09,0.1,0.11\right]\times 10^{-3}$. The bandwidth of each RB is 2 MHz, the transmit power of each device is 1 W, and the noise power spectral density $N_0=-174$ dBm/Hz. Optimal RB allocation vector is found by the Matlab intlinprog function. All simulations are implemented using Matlab 2018b. Neural networks used for data analysis are generated using Matlab machine learning toolbox. Identification accuracy, which is defined as the portion of training data samples that each device can correctly identify, is used to capture the FL performance. All code required to reproduce the results reported in this paper is available online at https://github.com/mzchen0/Communication-Efficient-Federated-Learning and https://code.ihub.org.cn/projects/4394/repository/revisions/master/show/PNAS. Next, we introduce the setup for the proposed FL and the simulation settings for all of the learning tasks.

Handwritten Digit Identification.

For handwritten digit identification, each device trains a fully connected neural network (FNN) that consists of 50 neurons (see SI Appendix, section 5 for further details) using 1,000 training data samples from MNIST dataset (41, 42). The 1,000 training data samples are distributed sequentially among the devices; that is, the first device has the first 1,000 samples in the dataset, while the second device has the second 1,000 samples, thus resulting in an uneven heterogeneous division of the labels of the devices. For example, device 1 has 100 digit 1, while device 2 may have 200 digit 1. Each device’s FNN is generated by the Matlab patternnet function. At each FL iteration, each device updates its local FL model using the SGD method with one epoch. The entire testing dataset is used to measure the identification accuracy.

Object Recognition in Images.

For 10-class object recognition, CIFAR-10 dataset (43, 44) is used to train FL models. The 50,000 training images are evenly allocated to 16 devices, and each device will have the same number of images of each object. Here, among these 16 devices, one of eight devices has 3,130 training data samples, and the number of training data samples of each object is 313. One of another eight devices has 3,120 training data samples, and the number of training data samples of each object is 312. The local FL model of each device is a CNN which is generated by the Matlab Layer and Option functions. The detailed architecture of the CNN is shown in SI Appendix, section 6. At each FL iteration, the CNN of each device is updated using the Adam method. One epoch with batch size 50 is adopt for each local FL model update. The entire testing dataset is used to measure the identification accuracy.

Finger Movement Detection.

For 15-class finger movement detection, each device trains its CNN using the electromyogram dataset 2 (45, 46). The electromyogram dataset 2 is collected from eight subjects that were asked to participate in 15-class finger movements, and hold each movement for a period of 20 s in each trial. Each figure movement is performed by a total of three trials. In our simulations, we only use 5-s data collected from each finger movement. Since the signal sampling frequency is 4,000 Hz, the number of signal samples collected by one sensor from each finger movement is 20,000. Since eight sensors are used to collect signals, the size of input vector of the local FL model will be 20,000 × 8. Hence, it is impractical to directly use all signal samples as one training sample to train the local FL model. In consequence, the windowing method in ref. 44 is used to process the original raw data. The size of each window is 500, and the increment is 50. Using the windowing method, the size of the input vector $\mathit{x}$ is $500\times 8$, and the size of the output vector $\mathit{y}$ is 15. Meanwhile, we assume that each device has 15-class finger movement data collected from one subject within one trial, and, hence, each device will have 391 training data samples for each finger movement and 5,865 data samples in total. The testing dataset consists of all 15-class finger movement data samples collected from eight subjects in one trial. The detailed architecture of the CNNs is shown in SI Appendix, section 7. Each device trains its CNN using Adam method. During each FL iteration, one epoch with batch size 75 is adopted for training each CNN.

Data Availability

All study data are included in the article and/or SI Appendix. All code required to reproduce the results reported in this paper is available online at GitHub (https://github.com/mzchen0/Communication-Efficient-Federated-Learning) and iHub (https://code.ihub.org.cn/projects/4394/repository/revisions/master/show/PNAS).