Minimax Estimation for Personalized Federated Learning: An Alternative between FedAvg and Local Training?

Abstract

A widely recognized difficulty in federated learning arises from the statistical heterogeneity among clients: local datasets often originate from distinct yet not entirely unrelated probability distributions, and personalization is, therefore, necessary to achieve optimal results from each individual’s perspective. In this paper, we show how the excess risks of personalized federated learning using a smooth, strongly convex loss depend on data heterogeneity from a minimax point of view, with a focus on the FedAvg algorithm (McMahan et al., 2017) and pure local training (i.e., clients solve empirical risk minimization problems on their local datasets without any communication). Our main result reveals an approximate alternative between these two baseline algorithms for federated learning: the former algorithm is minimax rate optimal over a collection of instances when data heterogeneity is small, whereas the latter is minimax rate optimal when data heterogeneity is large, and the threshold is sharp up to a constant.

As an implication, our results show that from a worst-case point of view, a dichotomous strategy that makes a choice between the two baseline algorithms is rate-optimal. Another implication is that the popular FedAvg following by local fine tuning strategy is also minimax optimal under additional regularity conditions. Our analysis relies on a new notion of algorithmic stability that takes into account the nature of federated learning.

1. Introduction

As one of the most important ingredients driving the success of machine learning, data are being generated and subsequently stored in an increasingly decentralized fashion in many real-world applications. For example, mobile devices will in a single day collect an unprecedented amount of data from users. These data commonly contain sensitive information such as web search histories, online shopping records, and health information, and thus are often not available to service providers (Poushter, 2016). This decentralized nature of (sensitive) data poses substantial challenges to many machine learning tasks.

To address this issue, McMahan et al. (2017) proposed a new learning paradigm, which they termed federated learning, for collaboratively training machine learning models on data that are locally possessed by multiple clients with the coordination of the central server (e.g., service provider), without having direct access to the local datasets. In its simplest form, federated learning considers a pool of $m$ clients, where the $i$-th client has a local dataset $S_i$ of size $n_i$, consisting of i.i.d. samples $\left\{z_j^{\left(i\right)}:j\in \left[n_i\right]\right\}$ (denote $\left[n\right]≔\{\mathrm{1,2},\dots ,n\}$) from some unknown distribution ${𝒟}_i$. Letting $\ell (\mathit{w},z)$ be a loss function, where $\mathit{w}$ denotes the model parameter, the optimal local model for the $i$-th client is given by

$${\mathit{w}}_{\star }^{\left(i\right)}\in \underset{\mathit{w}}{\text{argmin }}{\mathbb{E}}_{Z_i~{𝒟}_i}\ell \left(\mathit{w},Z_i\right).$$ (1)

From the client-wise perspective, any data-dependent estimator ${\widehat{\mathit{w}}}^{\left(i\right)}\left(\mathit{S}\right)$, with $\mathit{S}={\left\{S_i\right\}}_{i=1}^m$ denoting the collection of all samples, can be evaluated based on its individualized excess risk:

$${\text{IER}}_i≔{\mathbb{E}}_{Z_i~{𝒟}_i}\left[\ell \left({\widehat{\mathit{w}}}^{\left(i\right)},Z_i\right)-\ell \left({\mathit{w}}_{\star }^{\left(i\right)},Z_i\right)\right],$$

where the expectation is taken over a fresh sample $Z_i~{𝒟}_i$. At a high level, this learning paradigm of federated learning aims to obtain possibly different trained models for each client such that the individualized excess risks are low (see, e.g., Kairouz et al. 2019).

From a statistical viewpoint, perhaps the most crucial factor in determining the effectiveness of federated learning is data heterogeneity. When the data distribution ${𝒟}_i$ is (approximately) homogeneous across different clients, presumably a single global model would lead to small ${\text{IER}}_i$ for all $i$. In this regime, indeed, McMahan et al. (2017) proposed the federated averaging algorithm (FedAvg, see Algorithm 1), which can be regarded as an instance of local stochastic gradient descent (SGD) for solving (Mangasarian and Solodov, 1993; Stich, 2019)

$$\underset{w}{\text{min}}\frac{1}{N}\sum _{i\in \left[m\right]}{n}_i{L}_i\left(w,S_i\right),$$ (2)

where $L_i\left(\mathit{w},S_i\right)≔{\sum }_{j\in \left[n_i\right]}\ell \left(\mathit{w},z_j^{\left(i\right)}\right)/n_i$ is the empirical risk minimization (ERM) objective of the $i$-th client and $N=n_1+\cdots +n_m$ denotes the total number of training samples. Translating Algorithm 1 into words, FedAvg in effect learns a shared global model using gradients from each client and outputs a single model as an estimate of ${\mathit{w}}_{\star }^{\left(i\right)}$ for all clients. When the distributions $\left\{{𝒟}_i\right\}$ coincide with each other, FedAvg with a strongly convex loss achieves a weighted average excess risk of $𝒪(1/N)$, which is minimax optimal up to a constant factor (Shalev-Shwartz et al., 2009; Agarwal et al., 2012), see the formal statement in Theorem 6.

However, it is an entirely different story in the presence of data heterogeneity. FedAvg has been recognized to give inferior performance when there is a significant departure from complete homogeneity (see, e.g., Bonawitz et al. 2019). To better understand this point, consider the extreme case where the data distributions $\left\{{𝒟}_i\right\}$ are entirely unrelated. This roughly amounts to saying that the model parameters $\left\{{\mathit{w}}_{\star }^{\left(i\right)}\right\}$ can be arbitrarily different from each other. In such a “completely heterogeneous” scenario, the objective function (2) simply has no clear interpretation, and any single global model—for example, the output of FedAvg—would lead to unbounded risks for most, if not all, clients. As a matter of fact, it is not difficult to see that the optimal training strategy for federated learning in this regime is arguably PureLocalTraining, which lets each client separately run SGD to minimize its own local ERM objective

$$\underset{{\mathit{w}}^{\left(i\right)}}{\text{min }}{L}_i\left({\mathit{w}}^{\left(i\right)},S_i\right)$$ (3)

without any communication. Indeed, PureLocalTraining is minimax rate optimal in the completely heterogeneous regime, just as FedAvg in the completely homogeneous regime (see Theorem 4).

The level of data heterogeneity in practical federated learning problems is apparently neither complete homogeneity nor complete heterogeneity. Thus, the foregoing discussion raises a pressing question of what would happen if we are in the wide middle ground of the two extremes. This underlines the essence of personalized federated learning, which seeks to develop algorithms that perform well over a wide spectrum of data heterogeneity. Despite a venerable line of work on personalized federated learning (see, e.g., Kulkarni et al. 2020), the literature remains relatively silent on how the fundamental limits of personalized federated learning depend on data heterogeneity, as opposed to two extreme cases where both the minimax optimal rates and algorithms are known.

1.1. Main Contributions

The present paper takes a step toward understanding the statistical limits of personalized federated learning by establishing the minimax rates of convergence for both individualized excess risks and their weighted average with smooth strongly convex losses. We briefly summarize our main contributions below.

1. We prove that if the client-wise sample sizes are relatively balanced, then there exists a problem instance on which the ${\text{IER}}_i$’s of any algorithm are lower bounded by

   $$\left\{\begin{array}{ll}\Omega \left(1/N+R^2\right)& \text{if }{R}^2=𝒪(m/N)\\ \Omega (m/N)& \text{if }{R}^2=\Omega (m/N),\end{array}\right.$$ (4)

   where $R$ is the minimum quantity satisfying ${\text{min}}_{\mathit{w}\in 𝒲}{\sum }_{i\in \left[m\right]}{n}_i{∥{\mathit{w}}_{\star }^{\left(i\right)}-\mathit{w}∥}^2/N\le R^2$, i.e., it measures the maximum level of heterogeneity among clients (here $\Vert \cdot \Vert$ throughout the paper denotes the Euclidean distance). Meanwhile, we show that the ${\text{IER}}_i$’s of FedAvg are upper bounded by $𝒪\left(1/N+R^2\right)$, whereas the guarantee for PureLocalTraining is $𝒪(m/N)$, regardless of the specific value of $R$. Moreover, we also establish similar upper and lower bounds for a weighted average of the ${\text{IER}}_i$’s under a weaker condition.

2. A closer look at the above-mentioned bounds reveals a perhaps surprising phenomenon: for a given collection of problem instances with a specified maximum level of heterogeneity, exactly one of FedAvg or PureLocalTraining is minimax optimal.

3. The established minimax results suggest that the naïve dichotomous strategy of (1) running FedAvg when $R^2=𝒪(m/N)$, and (2) running PureLocalTraining when $R^2=\Omega (m/N)$, attains the lower bound (4). Moreover, for supervised problems, this dichotomous strategy can be implemented without knowing $R$ by (1) running both FedAvg and PureLocalTraining, (2) evaluating the test errors of the two algorithms in a distributed fashion, and (3) deploying the algorithm with a lower test error. We emphasize that the notion of optimality under our consideration overlooks constant factors. In practice, a better personalization result could be achieved by more sophisticated algorithms.

4. As a side product, we provide a novel analysis of FedProx, a popular algorithm for personalized federated learning that constrains the learned local models to be close via ${\ell }_2$ regularization (Li et al., 2018). In particular, we show that its ${\text{IER}}_i$’s are of order $𝒪\left(\frac{1}{N/m}\wedge \frac{R}{\sqrt{N/m}}+\frac{\sqrt{m}}{N}\right)$, and a weighted average of the ${\text{IER}}_i$’s satisfies a tighter $𝒪\left(\frac{1}{N/m}\wedge \frac{R}{\sqrt{N/m}}+\frac{1}{N}\right)$ bound, where $a\wedge b=\text{min}\{a,b\}$ for two real numbers $a$ and $b$.

5. On the technical side, our upper bound analysis is based on a generalized notion of algorithmic stability (Bousquet and Elisseeff, 2002), which we term federated stability and can be of independent interest. Briefly speaking, an algorithm $𝒜\left(\mathit{S}\right)=\left\{{\widehat{\mathit{w}}}^{\left(i\right)}\left(\mathit{S}\right)\right\}$ has federated stability $\left\{{\gamma }_i\right\}$ if for any $i\in \left[m\right]$, the loss function evaluated at ${\widehat{\mathit{w}}}^{\left(i\right)}\left(\mathit{S}\right)$ can only change by an additive term of $𝒪\left({\gamma }_i\right)$, if we perturb $S_i$ a little bit, while keeping the rest of datasets $\left\{S_{i^{\prime }}:i^{\prime }\ne i\right\}$ fixed. Similar ideas have appeared in Maurer (2005) and have been recently applied to multi-task learning (Wang et al., 2018). However, their notion of perturbation is based on the deletion of the whole client-wise dataset, whereas our notion of federated stability operates at the “record-level” and is more fine-grained. On the other hand, our construction of the lower bound is based on a generalization of Assound’s lemma (Assouad, 1983) (see also Yu 1997), which enables us to handle multiple heterogeneous datasets.

1.2. Related Work

Ever since the proposal of federated learning by McMahan et al. (2017), recent years have witnessed a rapidly growing line of work that is concerned with various aspects of FedAvg and its variants (see, e.g., Khaled et al. 2019; Haddadpour and Mahdavi 2019; Li et al. 2020b; Bayoumi et al. 2020; Malinovsky et al. 2020; Li and Richtárik 2020; Woodworth et al. 2020; Yuan and Ma 2020; Zheng et al. 2021).

In the context of personalized federated learning, there have been significant algorithmic developments in recent years. While the idea of using ${\ell }_2$ regularization to constrain the learned models to be similar has appeared in early works on multi-task learning (Evgeniou and Pontil, 2004), its applicability to personalized federated learning was only recently demonstrated by Li et al. (2018), where the FedProx algorithm was introduced. Similar regularization-based methods have been proposed and analyzed from the scope of convex optimization in Hanzely and Richtárik (2020); Dinh et al. (2020), and Hanzely et al. (2020). In particular, Hanzely et al. (2020) showed that an accelerated variant of FedProx is optimal in terms of communication complexity and the local oracle complexity. There is also a line of work using model-agnostic meta learning (Finn et al., 2017) to achieve personalization (Jiang et al., 2019; Fallah et al., 2020). Other strategies have been proposed (see, e.g., Arivazhagan et al. 2019; Li and Wang 2019; Mansour et al. 2020; Yu et al. 2020), and we refer readers to Kulkarni et al. (2020) for a comprehensive survey. We briefly remark here that all the papers mentioned above only consider the optimization properties of their proposed algorithms, while we focus on statistical properties of personalized federated learning.

Compared to the optimization understanding, our statistical understanding (in terms of sample complexity) of federated learning is still limited. Deng et al. (2020) proposed an algorithm for personalized federated learning with learning-theoretic guarantees. However, it is unclear how their bound scales with the heterogeneity among clients.

More generally, exploiting the information “shared among multiple learners” is a theme that constantly appears in other fields of machine learning such as multi-task learning (Caruana, 1997), meta learning (Baxter, 2000), and transfer learning (Pan and Yang, 2009), from which we borrow a lot of intuitions (see, e.g., Ben-David et al. 2006; Ben-David and Borbely 2008; Ben-David et al. 2010; Maurer et al. 2016; Cai and Wei 2019; Hanneke and Kpotufe 2019, 2020; Du et al. 2020; Tripuraneni et al. 2020a, b; Kalan et al. 2020; Shui et al. 2020; Li et al. 2020a; Zhang et al. 2020; Jose and Simeone 2021).

More related to our work, a series work by Denevi et al. (2018, 2019); Balcan et al. (2019), and Khodak et al. (2019) assumes the optimal local models lie in a small sub-parameter-space, and establishes “heterogeneity-aware” bounds on a weighted average of individualized excess risks. However, we would like to point out that they operate under the online learning setup, where the datasets are assumed to come in streams, and this is in sharp contrast to the federated learning setup, where the datasets are decentralized. Our notion of heterogeneity is also related to the hierarchical Bayesian model considered in Bai et al. (2020); Lucas et al. (2020); Konobeev et al. (2020), and Chen et al. (2020).

1.3. Paper Organization

The rest of this paper is organized as follows. In Section 2, we give an exposition of the problem setup and main assumptions. Section 3 presents our main results with proof sketches. We conclude this paper with a discussion of open problems in Section 4. For brevity, detailed proofs are deferred to the appendix.

2. Problem Setup

In this section, we detail some preliminaries to prepare the readers for our main results.

Notation.

We introduce the notation we are going to use throughout this paper. For two real numbers $a,b$, we let $a\vee b=\text{max}\{a,b\}$ and $a\wedge b=\text{min}\{a,b\}$. For two non-negative sequences $a_n,b_n$, we denote $a_n\lesssim b_n$ (resp. $a_n\gtrsim b_n$) if $a_n\le C{b}_n$ (resp. $a_n\ge C{b}_n$) for some constant $C>0$ when $n$ is sufficiently large. We use $a_n\asymp b_n$ to indicate that $a_n\gtrsim b_n,a_b\lesssim b_n$ hold simultaneously. We also use $a_n=𝒪\left(b_n\right)$, whose meaning is the same as $a_n\lesssim b_n$, and $a_n=\Omega \left(b_n\right)$, whose meaning is the same as $a_n\gtrsim b_n$. For two probability distributions ${𝒟}_1$ and ${𝒟}_2$, we use ${𝒟}_1\otimes {𝒟}_2$ to denote their joint distribution under independence. We use $𝒲$ to denote the parameter space and $𝒵$ to denote the sample space. Finally, we let ${𝒫}_{𝒲}\left(x\right)≔{\text{argmin }}_{y\in 𝒲}\Vert x-y\Vert$ denote the operator that projects $x$ onto $𝒲$ in Euclidean distance.

Evaluation Metrics.

The presentation of our main results relies on how to evaluate the performance of a federated learning algorithm. To this end, we consider the following two evaluation metrics.

Definition 1 (Individualized excess risk) Consider an algorithm $𝒜$ that outputs $\mathit{𝒜}\left(\mathit{S}\right)={\left\{{\widehat{\mathit{w}}}^{\left(i\right)}\left(\mathit{S}\right)\right\}}_{i=1}^m$. For the $i$-the client, its individualized excess risk (IER) is defined as

$${\text{IER}}_i\left(𝒜\right)≔{\mathbb{E}}_{Z_i~{𝒟}_i}\left[\ell \left({\widehat{\mathit{w}}}^{\left(i\right)}\left(\mathit{S}\right),Z_i\right)-\ell \left({\mathit{w}}_{\star }^{\left(i\right)},Z_i\right)\right],$$ (5)

where $Z_i~{𝒟}_i$ is a fresh data point independent of $\mathit{S}$.

Definition 2 (p-average excess risk) Consider an algorithm $𝒜$ that outputs $𝒜\left(\mathit{S}\right)=\left\{{\widehat{\mathit{w}}}^{\left(i\right)}\left(\mathit{S}\right)\right\}$. For a vector $\mathbf{p}=\left(p_1,\dots ,p_m\right)$ lying in the $m$-dimensional probability simplex (i.e., all $p_i$’s are non-negative and they sum to one), we define the $\mathbf{p}$-average excess risk (${\text{AER}}_{\mathbf{p}}$) of $𝒜$ to be

$${\text{AER}}_p\left(𝒜\right)≔\sum _{i\in \left[m\right]}{p}_i\cdot {\text{IER}}_i\left(𝒜\right).$$ (6)

In words, IER measures the performance of the algorithm from the client-wise perspective, whereas AER evaluates the performance of the algorithm from the system-wide perspective.

Intuitively speaking, the weight vector $\mathbf{p}$ in (6) can be regarded as the importance weight on each client and controls “how many resources are allocated to each client”. For example, setting $p_i=1/m$ enforces “fair allocation”, so that each client is treated uniformly, regardless of sample sizes. As another example, setting $p_i=n_i/N$ (recall that $N={\sum }_{i\in \left[m\right]}{n}_i$ is the total sample size) means that the central server pays more attention to clients with larger sample sizes, which, to a certain extend, incentivize the clients to contribute more data.

Notably, while a uniform upper bound on all ${\text{IER}}_i$’s can be carried over to the same bound on ${\text{AER}}_{\mathbf{p}}$, a bound on the ${\text{AER}}_{\mathbf{p}}$ alone in general does not imply a tight bound on each ${\text{IER}}_i$, other than the trivial bound ${\text{IER}}_i\le {\text{AER}}_{\mathbf{p}}/p_i$. Such a subtlety is a distinguishing feature of personalized federated learning in the following sense: under homogeneity, it suffices to estimate a single shared global model, and thus ${\text{AER}}_{\mathbf{p}}$ and all of ${\text{IER}}_i\text{s}$ are mathematically equivalent.

Regularity Conditions.

In this paper, we restrict ourselves to bounded, smooth, and strongly convex loss functions. Such assumptions are common in the federated learning literature (see, e.g., Li et al. 2020b; Hanzely et al. 2020) and cover many unsupervised learning problems such as mean estimation in exponential families and supervised learning problems such as generalized linear models.

Assumption A (Regularity conditions) Suppose the following conditions hold:

1. Compact and convex domain. The parameter space $𝒲$ is a compact convex subset of ${\mathbb{R}}^d$ with diameter $D≔{\text{sup}}_{\mathit{w},{\mathit{w}}^{\prime }\in 𝒟}∥\mathit{w}-{\mathit{w}}^{\prime }∥<\infty$;

2. Smoothness and strong convexity. For any $i\in \left[m\right]$, the loss function $\ell (\cdot ,z)$ is $\beta$-smooth for almost every $z$ in the support of ${𝒟}_i$, and the $i$-th ERM objective $L_i(\cdot ,S)$ is almost surely $\mu$-strongly convex on the convex domain $𝒲\subseteq {\mathbb{R}}^d$. We also assume that there exists a universal constant $\Vert \ell {\Vert }_{\infty }$ such that $0\le \ell (\cdot ,z)\le \Vert \ell {\Vert }_{\infty }$ for almost every $z$ in the support of ${𝒟}_i$;

3. Bounded gradient variance at optimum. There exists a positive constant $\sigma$ such that for any $i\in \left[m\right]$, we have ${\mathbb{E}}_{Z_i~{𝒟}_i}{∥\nabla \ell \left({\mathit{w}}_{\star }^{\left(i\right)},Z_i\right)∥}^2\le {\sigma }^2$.

Heterogeneity Conditions.

To quantify the level of heterogeneity among clients, we start by introducing the notion of an average global model. Assuming a strongly convex loss, the optimal local models (1) are uniquely defined. Thus, we can define the average global model as

$$w_p^{\left(\text{global}\right)}=\sum _{i\in \left[m\right]}{p}_i{w}_{\star }^{\left(i\right)}.$$ (7)

We remark that the average global model defined in (7) should not be interpreted as the “optimal global model”. Rather, it is more suitable to think of ${\mathit{w}}_{\mathbf{p}}^{\text{(global) }}$ as a point in the parameter space, from which every local model is close to. Indeed, one can readily check that the average global model is the minimizer of ${\sum }_{i\in \left[m\right]}{p}_i{∥{\mathit{w}}_{\star }^{\left(i\right)}-\mathit{w}∥}^2$ over $\mathit{w}\in {\mathbb{R}}^d$.

We are now ready to quantify the level of client-wise heterogeneity as follows.

Assumption B (Level of heterogeneity)There exists a positive constant$R$such that

1. either ${\sum }_{i\in \left[m\right]}{p}_i{∥{\mathit{w}}_{\star }^{\left(i\right)}-{\mathit{w}}_{\mathbf{p}}^{\left(\text{global}\right)}∥}^2\le R^2,$

2. or ${∥{\mathit{w}}_{\star }^{\left(i\right)}-{\mathit{w}}_{\mathbf{p}}^{\left(\text{global}\right)}∥}^2\le R^2\forall i\in \left[m\right]$.

Our study of the ${\text{AER}}_{\mathbf{p}}$ and ${\text{IER}}_i\text{s}$ will be based on Part (a) and (b) of Assumption B, respectively. Intuitively, the quantity $R$ encodes one’s belief on “how heterogeneous” the clients can be.

3. Main Results

3.1. Analyses of Two Baseline Algorithms

In this subsection, we characterize the performance of PureLocalTraining and FedAvg under the heterogeneity conditions imposed by Assumption B.

3.1.1. Warm Up: Uniform Stability and Analysis of PureLocalTraining

The analysis of PureLocalTraining is based on the classical notion of uniform stability, proposed by Bousquet and Elisseeff (2002).

Definition 3 (Uniform stability) Consider an algorithm $𝒜$ that takes a single dataset $S={\left\{z_j\right\}}_{j=1}^n$ of size $n$ as input and outputs a single model: $𝒜\left(S\right)=\stackrel{\mathit{ˆ}}{\mathit{w}}\left(S\right)$. We say $𝒜$ is $\gamma$-uniformly stable if for any dataset $S$, any $j\in \left[n\right]$, and any $z_j^{\prime }\in 𝒵$, we have

$${∥\ell \left(\stackrel{\mathit{ˆ}}{\mathit{w}}\right(S),\cdot )-\ell \left(\stackrel{\mathit{ˆ}}{\mathit{w}}\left(S^{\setminus j}\right),\cdot \right)∥}_{\infty }\le \gamma ,$$

where $S^{\setminus j}$ is the dataset formed by replacing $z_j$ with $z_j^{\prime }$:

$$S^{\setminus j}=\left\{z_1,\dots ,z_{j-1},z_j^{\prime },z_j,\dots ,z_n\right\}.$$

The main implication of uniformly stable algorithms is that “stable algorithms do not overfit”: if $𝒜$ is $\gamma$-uniformly stable, then its generalization error is upper bounded by a constant multiple of $\gamma$. Thus, one can dissect the analysis of $𝒜$ into two separate parts: (1) bounding its optimization error; (2) bounding its stability term.

Under our working assumptions, SGD with properly chosen step sizes is guaranteed to converge to the global minimum of (3) (see, e.g., Rakhlin et al. (2011)). Note that the bounds for the approximate minimizers only involve an extra additive term representing the optimization error, and this term will be negligible if we run SGD until convergence since our focus is sample complexity. Thus, we conduct the analysis for the global minimizer of (3). The performance of PureLocalTraining is given by the following theorem.

Theorem 4 (Performance of PureLocalTraining) Let Assumption A(b) hold and assume $n_i\ge 4\beta /\mu \forall i\in \left[m\right]$. Then the algorithm ${𝒜}_{\text{PLT }}$ which outputs the minimizer of (3) satisfies

$${\mathbb{E}}_{\mathit{S}}\left[{\text{IER}}_i\left({𝒜}_{\text{PLT}}\right)\right]\lesssim \frac{\beta \Vert \ell {\Vert }_{\infty }}{\mu n_i}$$

for all $i=1,\dots ,m$.

Proof The proof is a direct consequence of standard results on uniform stability of strongly convex ERM (see, e.g., Section 5 of Shalev-Shwartz et al. (2009) and Section 13 of Shalev-Shwartz and Ben-David (2014)), which assert that under the current assumptions, the minimizer of (3) is $𝒪\left(\frac{\beta \Vert \ell {\Vert }_{\infty }}{\mu n_i}\right)$-uniformly stable. We omit the details. ■

By definition, for any weight vector$\mathbf{ }\mathbf{p},$ ${\text{AER}}_{\mathbf{p}}$ of PureLocalTraining also admits the same upper bound as (4).

3.1.2. Federated Stability and Analysis of FedAvg

We consider the following weighted version of (2):

$$\underset{w\in 𝒲}{\text{min}}\sum _{i\in \left[m\right]}{p}_i{L}_i\left(w,S_i\right).$$ (8)

The FedAvg algorithm (Algorithm 1) also seamlessly generalizes. The above optimization formulation is in fact covered by the general theory of Li et al. (2020b), where they showed that FedAvg is guaranteed to converge to the global optimum under a suitable hyperparameter choice, even in the presence of heterogeneity (but the convergence is slower). Thus, in the following discussion, we again consider the global minimizer of (8).

It turns out that a tight analysis of FedAvg requires a more fine-grained notion of uniform stability, which we present below.

Definition 5 (Federated stability) An algorithm $𝒜$ that outputs $𝒜\left(\mathit{S}\right)=\left\{{\widehat{\mathit{w}}}^{\left(i\right)}\left(\mathit{S}\right)\right\}$ has federated stability ${\left\{{\gamma }_i\right\}}_{i=1}^m$ if for every $\mathit{S}~{⨂}_i{𝒟}_i^{\otimes n_i}$ and for any $i\in \left[m\right],j_i\in \left[n_i\right],z_{i,j_i}^{\prime }\in 𝒵$, we have

$${∥\ell \left({\widehat{\mathit{w}}}^{\left(i\right)}\left(\mathit{S}\right),\cdot \right)-\ell \left({\widehat{\mathit{w}}}^{\left(i\right)}\left({\mathit{S}}^{\setminus \left(i,j_i\right)}\right),\cdot \right)∥}_{\infty }\le {\gamma }_i.$$

Above, ${\mathit{S}}^{\setminus \left(i,j_i\right)}$ is the dataset formed by replacing $z_{j_i}^{\left(i\right)}$ in the $i$-the dataset with $z_{i,j_i}^{\prime }$:

$${\mathit{S}}^{\setminus \left(i,j_i\right)}=\left\{S_1,\dots ,S_{i-1},S_i^{\setminus j_i},S_{i+1},\dots ,S_m\right\},$$

$$S_i^{\setminus j_i}=\left\{z_1^{\left(i\right)},\dots ,z_{j_i-1}^{\left(i\right)},z_{i,j_i}^{\prime },z_{j_i+1}^{\left(i\right)},\dots ,z_{n_i}^{\left(i\right)}\right\}.$$

Compared to the conventional uniform stability in Definition 3, federated stability provides a finer control by allowing distinct stability measures $\left\{{\gamma }_i\right\}$ for different clients. Moreover, the classical statement that “stable algorithms do not overfit” still holds, in the sense that the average (resp. individualized) generalization error can be upper bounded by $𝒪\left({\sum }_{i\in \left[m\right]}{n}_i{\gamma }_i/N\right)$ (resp. $𝒪\left({\gamma }_i\right)$), plus a term scaling with the level of heterogeneity $R$. And this again enables us to separate the analysis of $𝒜$ into two parts (namely bounding the optimization error and bounding the stability), as is the case with the conventional uniform stability.

The notion of federated stability has other implications when restricted to the FedProx algorithm, and we refer the readers to Section 3.4 for details.

We are now ready to state the theorem that characterizes the performance of FedAvg.

Theorem 6 (Performance of FedAvg) Let Assumption A(b, c) hold and assume $n_i\ge 4\beta p_i/\mu \forall i\in \left[m\right]$. Suppose the FedAvg algorithm ${𝒜}_{\text{FA}}$ outputs the minimizer of (8). Then under Assumption B(a), we have

$${\mathbb{E}}_S\left[{\text{AER}}_p\left({𝒜}_{\text{FA}}\right)\right]\lesssim \frac{\beta \Vert \ell {\Vert }_{\infty }}{\mu }\sum _{i\in \left[m\right]}\frac{p_i^2}{n_i}+\beta R^2,$$ (9)

and under Assumption B(b), we have

$${\mathbb{E}}_{\mathit{S}}\left[{\text{IER}}_i\left({𝒜}_{\text{FA}}\right)\right]\lesssim \frac{\beta {\sigma }^2}{{\mu }^2}\sum _{i^{\prime }\in \left[m\right]}\frac{p_{i^{\prime }}^2}{n_{i^{\prime }}}+\frac{{\beta }^3}{{\mu }^2}{R}^2.$$ (10)

Proof The proof of (9) is, roughly speaking, based on the fact that the global minimizer of (8) has federated stability ${\gamma }_i\lesssim \frac{\beta \Vert \ell {\Vert }_{\infty }{p}_i}{\mu n_i}$, and thus the first term in the right-hand side of (9) corresponds to the average federated stability ${\sum }_{i\in \left[m\right]}{p}_i{\gamma }_i$. The second term $\beta R^2$ in the right-hand side of (9) reflects the presence of heterogeneity. For Equation (10), we were not able to obtain a federated stability based proof, and our current proof is based on an adaptation of the arguments in Theorem 7 of Foster et al. (2019), which explains why the dependence on $(\sigma ,\beta ,\mu )$ are different (and slightly worse) compared to Equation (9). In particular, the bound (10) has inverse quadratic dependence on $\mu$, wheres the bound (9) only has $1/\mu$ dependence. The $1/\mu$ dependence comes from the fact that the federated stability term has such dependence, and the $1/{\mu }^2$ dependence comes from the fact that the ${\ell }_2$ estimation error has such dependence. We refer the readers to Appendix C.1 for details. ■

Note that both bounds in the above theorem are minimized by choosing $p_i=n_i/N$. With this choice of $\mathbf{p}$, the two bounds read

$${\mathbb{E}}_{\mathit{S}}\left[{\text{AER}}_{\mathbf{p}}\left({𝒜}_{\text{FA}}\right)\right]\lesssim \frac{\beta \Vert \ell {\Vert }_{\infty }}{\mu N}+\beta R^2,{\mathbb{E}}_{\mathit{S}}\left[{\text{IER}}_i\left({𝒜}_{\text{FA}}\right)\right]\lesssim \frac{\beta {\sigma }^2}{{\mu }^{2}N}+\frac{{\beta }^3{R}^2}{{\mu }^2}.$$ (11)

This makes sense, since this choice of weight corresponds to the ERM objective under complete homogeneity. This observation also suggests that ensuring “fair resource allocation” (i.e., setting $p_i=1/m$) can lead to statistical inefficiency, especially when the sample sizes are imbalanced.

We conclude this subsection by noting that though the compactness assumption (Assumption A(a)) is not needed in Theorem 6, it is usually needed in the analysis of the optimization error of FedAvg and PureLocalTraining(see, e.g., Rakhlin et al. (2011); Li et al. (2020b)).

3.2. Lower Bounds

In this subsection, we present our construction of lower bounds, which characterize the information-theoretic limit of personalized federated learning. Throughout this section, we restrict out attention to the case where $p_i=n_i/N$ for any $i\in \left[m\right]$.

Our construction starts by considering a special class of problem instances: logistic regression. In logistic regression, given the collection of regression coefficients $\left\{{\mathit{w}}_{\star }^{\left(i\right)}\right\}\subseteq 𝒲$ where $𝒲$ has a diameter $D$, the data distributions ${𝒟}_i$’s are supported on ${\mathbb{R}}^d\times \{\pm 1\}$ and specified by a two-step procedure as follows:

1. Generate a feature vector $\mathit{x}$, whose coordinates are i.i.d. copies from some distribution ${\mathbb{P}}_X$ on $\mathbb{R}$, which is assumed to have mean zero and is almost surely bounded by some absolute constant $c_X$;

2. Generate the binary label $y\in \{\pm 1\}$, which is a biased Rademacher random variable with head probability ${\left(1+\text{exp}\left\{-{\mathit{x}}^{\top }{\mathit{w}}_{\star }^{\left(i\right)}\right\}\right)}^{-1}$.

The loss function is naturally chosen to be the negative log-likelihood function, which takes the following form:

$$\ell (\mathit{w},z)=\ell (\mathit{w},\mathit{x},y)=\text{log}\left(1+e^{-y{\mathit{x}}^{\top }\mathit{w}}\right).$$

The following lemma says that Assumption A holds for the aforementioned logistic regression models.

Lemma 7 (Logistic regressions are valid problem instances) The logistic regression problem described above is a class of problem instances that satisfies Assumption A with $\Vert \ell {\Vert }_{\infty }=c_{X}D\sqrt{d}$ and ${\sigma }^2=\beta =c_X^{2}d/4$. Moreover, if $m\lesssim (N/m{)}^c$ for some $c\ge 0$ and $N/m\ge Cd$ for some $C>1$, then there exists some event $⋿$ which only depends on the features $\left\{{\mathit{x}}_j^{\left(i\right)}:i\in \left[m\right],j\in \left[n_i\right]\right\}$ and happens with probability at least $1-e^{-𝒪\left(\sqrt{N/m}\right)}$, such that on this event, the strongly convex constant in Assumption A satisfies

$$\mu \asymp {\mu }_0={\left(\text{exp}\left\{c_{X}D\sqrt{d}/2\right\}+\text{exp}\left\{-c_{X}D\sqrt{d}\right\}\right)}^{-2}.$$ (12)

Proof The compactness of the domain and the boundedness of the loss function hold by construction. To verify the rest parts of Assumption A, with some algebra one finds that

$${\nabla }^2\ell \left(\mathit{w},\mathit{x},y\right)=\frac{\mathit{x}{\mathit{x}}^{\top }\text{exp}\left\{y{\mathit{x}}^{\top }\mathit{w}\right\}}{{\left(1+\text{exp}\left\{y{\mathit{x}}^{\top }\mathit{w}\right\}\right)}^2}⪯\frac{1}{4}\mathit{x}{\mathit{x}}^{\top },$$ (13)

where $⪯$ is the Loewner order and the inequality holds because $x/(1+x{)}^2=1/\left(x^{-1/2}+\right.{\left.x^{1/2}\right)}^2\le 1/4$ for $x>0$. Since the population gradient has mean zero at optimum, the gradient variance at optimum can be upper bounded by the trace of the expected Hessian matrix, which, by the above display, is further upper bounded by $c_X^{2}d/4$. Thus, we can take ${\sigma }^2=c_X^{2}d/4$ in Part (c). Another message of the above display is that we can set the smoothness constant in Part (b) to be $\beta =c_X^{2}d/4$.

The only subtlety that remains is to ensure each local loss function is $\mu$-strongly convex. Note that since $x/(1+x{)}^2$ is decreasing from (0, 1) and is increasing from $(1,\infty )$, the right-hand side of (13) dominates ${\mu }_0\mathit{x}{\mathit{x}}^{\top }$ in Loewner order, where ${\mu }_0$ is the right-hand side of (12). Thus, the local population losses ${\mathbb{E}}_{(\mathit{x},y)~{𝒟}_i}[\ell (\cdot ,\mathit{x},y\left)\right]$ are all ${\mu }_0$-strongly convex.

Now, note that

$${\nabla }^2{L}_i\left({\mathit{w}}^{\left(i\right)},S_i\right)=\frac{1}{n_i}\sum _{j\in \left[n_i\right]}\frac{{\mathit{x}}_j^{\left(i\right)}{\left({\mathit{x}}_j^{\left(i\right)}\right)}^{\top }\text{exp}\left\{y_j^{\left(i\right)}⟨{\mathit{x}}_j^{\left(i\right)},{\mathit{w}}^{\left(i\right)}⟩\right\}}{{\left(1+\text{exp}\left\{y_j^{\left(i\right)}⟨{\mathit{x}}_j^{\left(i\right)},{\mathit{w}}^{\left(i\right)}⟩\right\}\right)}^2}⪰{\mu }_0\cdot \frac{1}{n_i}\sum _{j\in \left[n_i\right]}{\mathit{x}}_j^{\left(i\right)}{\left({\mathit{x}}_j^{\left(i\right)}\right)}^{\top }.$$

Invoking Theorem 5.39 of Vershynin (2010) along with a union bound over all clients, we conclude that for any $i\in \left[m\right]$, the minimum eigenvalue of ${\sum }_{j\in \left[n_i\right]}{\mathit{x}}_j^{\left(i\right)}{\left({\mathit{x}}_j^{\left(i\right)}\right)}^{\top }$ is lower bounded by a constant multiple of $n_i-p\gtrsim n_i$ (this is the definition of the event $⋿$) with probability at least $1-m{e}^{-𝒪\left(n_i\right)}\ge 1-e^{-𝒪\left(\sqrt{N/m}\right)}$, and the proof is concluded. ■

Note that in the proof of the above lemma, we have established the ${\mu }_0\asymp \mu$-strong convexity of the client-wise population losses. Hence, lower bounding the excess risks reduces to lower bounding the ${\ell }_2$ estimation errors ${∥{\widehat{\mathit{w}}}^{\left(i\right)}-{\mathit{w}}_{\star }^{\left(i\right)}∥}^2$ of the estimators ${\widehat{\mathit{w}}}^{\left(i\right)}$ for ${\mathit{w}}_{\star }^{\left(i\right)}$. Such a reduction allows us to use powerful tools from information theory.

To this end, we introduce two parameter spaces, corresponding to Part (a) and (b) of Assumption B. Recalling ${\mathit{w}}_{\mathbf{p}}^{\left(\text{global}\right)}={\sum }_{i\in \left[m\right]}{p}_i{\mathit{w}}_{\star }^{\left(i\right)}$, we define

$${𝒫}_1≔\left\{{\left\{{\mathit{w}}_{\star }^{\left(i\right)}\right\}}_{i=1}^m\subseteq 𝒲:\sum _{i\in \left[m\right]}{p}_i{∥{\mathit{w}}_{\star }^{\left(i\right)}-{\mathit{w}}_{\mathbf{p}}^{\left(\text{global }\right)}∥}^2\le R^2\right\},$$

$${𝒫}_2≔\left\{{\left\{{\mathit{w}}_{\star }^{\left(i\right)}\right\}}_{i=1}^m\subseteq 𝒲:{∥{\mathit{w}}_{\star }^{\left(i\right)}-{\mathit{w}}_{\mathbf{p}}^{\left(\text{global }\right)}∥}^2\le R^2\forall i\in \left[m\right]\right\}.$$

Note that ${𝒫}_1$ and ${𝒫}_2$ index all possible values of $\left\{{\mathit{w}}_{\star }^{\left(i\right)}\right\}$ that can arise in the logistic regression models under Assumption B (a) and (b), respectively.

With the notations introduced so far, we are ready to state the main result of this subsection.

Theorem 8 (Minimax lower bounds for estimation errors) Consider the logistic regression model described above. Suppose $n_i\asymp n_{i^{\prime }}$ for any $i\ne i^{\prime }\in \left[m\right]$ and assume $p_i=n_i/N$ for any $i\in \left[m\right]$. Then we have

$$\underset{\left\{{\widehat{\mathit{w}}}^{\left(i\right)}\right\}}{\text{inf}}\underset{\left\{{\mathit{w}}_{\star }^{\left(i\right)}\right\}\in {𝒫}_1}{\text{sup}}\frac{1}{N}\sum _{i\in \left[m\right]}{n}_i{\mathbb{E}}_{\mathit{S}}{∥{\widehat{\mathit{w}}}^{\left(i\right)}-{\mathit{w}}_{\star }^{\left(i\right)}∥}^2\gtrsim \frac{d}{N/m}\wedge R^2+\frac{d}{N},$$ (14)

$$\underset{{\widehat{\mathit{w}}}^{\left(i\right)}}{\text{inf}}\underset{\left\{{\mathit{w}}_{\star }^{\left(i\right)}\right\}\in {𝒫}_2}{\text{sup}}{\mathbb{E}}_{\mathit{S}}{∥{\widehat{\mathit{w}}}^{\left(i\right)}-{\mathit{w}}_{\star }^{\left(i\right)}∥}^2\gtrsim \frac{d}{n_i}\wedge R^2+\frac{d}{N}$$ (15)

for all $i\in \left[m\right]$, where the infimum is taken over all possible ${\widehat{\mathit{w}}}^{\left(i\right)}s$ that are measurable functions of the data $\mathit{S}$.

Proof See Appendix A. ■

Note that both lower bounds in Theorem 8 are a superposition of two terms, and they correspond to two distinct steps in the proof.

The first step in our proof is to argue that the lower bound under complete homogeneity is in fact a valid lower bound under our working assumptions, which gives the $\Omega (d/N)$ term. This is reasonable, since estimation under complete homogeneity is, in many senses, an “easier” problem. The proof of the $\Omega (d/N)$ term is based on the classical Assouad’s method (Assouad, 1983).

The second step is to use a generalized version of Assouad’s method that allows us to deal with multiple heterogeneous datasets. In particular, we need to carefully choose the prior distributions over the parameter space based on the level of heterogeneity, which ultimately leads to the $\Omega \left(\frac{d}{N/m}\wedge R^2\right)$ term. Recall that in the vanilla version of Assouad’s method where there is only one parameter, say ${\mathit{w}}_{\star }$, one can lower bounds the minimax risk by the Bayes risk, and the prior distribution is usually chosen to be ${\mathit{w}}_{\star }=\delta \mathit{v}$, where $\mathit{v}$ follows a uniform distribution over all $d$-dimensional binary vectors and $\delta$ is chosen so that the resulting hypothesis testing problem has large type-I plus type-II error. In our case where there are $m$ parameters $\left\{{\mathit{w}}_{\star }^{\left(i\right)}\right\}$, we need to consider a different prior of the following form:

$${\mathit{w}}_{\star }^{\left(i\right)}={\delta }_i{\mathit{v}}^{\left(i\right)},$$

where ${\mathit{v}}^{\left(i\right)}$ are i.i.d. samples from the uniform distribution over all $d$-dimensional binary vectors, and ${\delta }_i$’s are scalers that need to be carefully chosen to make the resulting hypothesis testing problem hard.

The following result is an immediate corollary of Theorem 8.

Corollary 9 (Minimax lower bounds for excess errors) Assume there exist constants $C,C^{\prime }>0,c\ge 0$ such that $n_i\ge C\beta \forall i\in \left[m\right]$ and $m\le C^{\prime }(N/m{)}^c$. Moreover, assume $n_i\asymp n_{i^{\prime }}$ for any $i\ne i^{\prime }\in \left[m\right]$ and $p_i=n_i/N$ for any $i\in \left[m\right]$. Then there exists an absolute constant $c^{\prime }$ such that the following two statements hold:

1. There exists a problem instance such that Assumptions A and B(a) are satisfied with probability at least $1-e^{-c^{\prime }\sqrt{N/m}}$. Call this high probability event $⋿$. On this problem instance, any randomized algorithm $𝒜$ must suffer

$${\mathbb{E}}_{𝒜,\mathit{S}}\left[{\text{AER}}_{\mathbf{p}}\left(𝒜\right)\cdot {\mathbb{1}}_{\text{E}}\right]\gtrsim \mu \cdot \left(\frac{\beta }{N/m}\wedge R^2+\frac{\beta }{N}\right);$$ (16)

2. For any $i\in \left[m\right]$, there exists a problem instance such that Assumptions A and B(b) are satisfied with probability at least $1-e^{-c^{\prime }\sqrt{N/m}}$. Call this high probability event ${\text{E}}_i$. On this problem instance, any randomized algorithm $𝒜$ must suffer

$${\mathbb{E}}_{𝒜,\mathit{S}}\left[{\text{IER}}_i\left(𝒜\right)\cdot {\mathbb{1}}_{{\text{E}}_i}\right]\gtrsim \mu \cdot \left(\frac{\beta }{n_i}\wedge R^2+\frac{\beta }{N}\right).$$ (17)

In the two displays above, the expectation is taken over the randomness in both the algorithm $𝒜$ and the sample $\mathit{S}$.

Proof Along with Lemma 7 and Theorem 8, this corollary follows by the fact that the smoothness constant $\beta$ is of the same order as $d$ and the population losses are all ${\mu }_0\asymp \mu$-strongly convex. ■

3.3. Implications of the Main Results

The upper bounds in Section 3.1 and the lower bounds in Section 3.2 together reveal several intriguing phenomena regarding personalized FL, which we detail in this subsection.

Focusing on the dependence on the sample sizes and assuming the client-wise samples sizes are balanced (i.e., $n_i\asymp N/m$), the heterogeneity measure $R$ enters the lower and upper bounds in a dichotomous fashion:

• If $R^2\lesssim m/N$, then both lower bounds become $\Omega \left(R^2+1/N\right)$, and this lower bound can be attained by FedAvg up to factors that do not depend on the sample sizes;

• If $R^2\gtrsim m/N$, then both lower bounds become $\Omega (m/N)$. They agree with the minimax rate as if we were under complete heterogeneity and can be achieved by PureLocalTraining.

Now, let us consider the following naïve dichotomous strategy: if output $R^2\le \frac{\Vert \ell {\Vert }_{\infty }}{\mu }. m/N$, then output $𝒜={𝒜}_{\text{FA}}$; otherwise, output $𝒜={𝒜}_{\text{PLT}}$. That is, we switch between the two baseline algorithms at the threshold of $R^2\asymp m/N$. Then under the assumptions in Theorems 4 and 6, one can readily check that this dichotomous strategy satisfies the following AER guarantee:

$${\mathbb{E}}_{\mathit{S}}\left[{\text{AER}}_{\mathbf{p}}\left(𝒜\right)\right]\lesssim \beta \left(\frac{\Vert \ell {\Vert }_{\infty }}{\mu N/m}\wedge R^2\right)+\frac{\beta \Vert \ell {\Vert }_{\infty }}{\mu }\sum _{i\in \left[m\right]}\frac{p_i^2}{n_i}.$$ (18)

If in addition, $n_i\asymp n_{i^{\prime }}$ for any $i\ne i^{\prime }\in \left[m\right]$, then it also satisfies the following IER guarantee:

$${\mathbb{E}}_{\mathit{S}}\left[{\text{IER}}_i\left(𝒜\right)\right]\lesssim \frac{{\beta }^3}{{\mu }^2}\left(\frac{\Vert \ell {\Vert }_{\infty }}{\mu n_i}\wedge R^2\right)+\frac{\beta {\sigma }^2}{{\mu }^2}\sum _{i^{\prime }\in \left[m\right]}\frac{p_{i^{\prime }}^2}{n_{i^{\prime }}}.$$ (19)

When $p_i=n_i/N$, the two displays above simplify to

$${\mathbb{E}}_{\mathit{S}}\left[\text{AER}\right(𝒜\left)\right]\lesssim \beta \left(\frac{\Vert \ell {\Vert }_{\infty }}{\mu N/m}\wedge R^2\right)+\frac{\beta \Vert \ell {\Vert }_{\infty }}{\mu N},{\mathbb{E}}_{\mathit{S}}\left[{\text{IER}}_i\left(𝒜\right)\right]\lesssim \frac{{\beta }^3}{{\mu }^2}\left(\frac{\Vert \ell {\Vert }_{\infty }}{\mu n_i}\wedge R^2\right)+\frac{\beta {\sigma }^2}{{\mu }^{2}N},$$

which matches the lower bound in Corollary 9 up to constant factors, provided $\left(\beta ,\mu ,\Vert \ell {\Vert }_{\infty },\sigma \right)$ are all of constant order. In other words, switching between the two algorithms at the threshold of $R^2\asymp m/N$ gives an oracle algorithm that is minimax rate optimal.

Thus, we have shown an interesting property for personalized FL on the choice of the two baseline algorithms. In particular, consider a collection of problem instances indexed by $(R,\beta ,\mu ,\Vert \ell \Vert ,\sigma )$ using Assumptions A and B and assume $\left(\beta ,\mu ,\Vert \ell {\Vert }_{\infty },\sigma \right)$ are all of constant order. Now, for a fixed value of $R$, exactly one of these two algorithms is minimax optimal, where the optimality is defined over the specified collection of problem instances and with respect to both AER and IER. Moreover, the oracle dichotomous strategy that switches between the two baseline algorithms at the threshold of $R^2\asymp m/N$ is minimax optimal.

More implications of the theoretical results are described below.

Optimality of a dichotomous strategy.

From the practical side, for supervised learning problems, such a dichotomous strategy can be implemented without prior knowledge of $R$ if test errors can be evaluated in a distributed fashion. Indeed, we can first run both FedAvg and PureLocalTraining separately, evaluate their test errors (in a distributed fashion), and deploy the one with a lower test error. Due to the upper and lower bounds proved in Sections 3.1 and 3.2, such a strategy is guaranteed to be minimax rate optimal. As a caveat, however, one should refrain from interpreting our results as saying either of the two baseline algorithms is sufficient for practical problems. From a practical viewpoint, constants that are omitted in the minimax analysis are crucial. Even for supervised problems, a better personalization result could be achieved by more sophisticated algorithms in practice. Nevertheless, our results suggest that the two baseline algorithms can at least serve as a good starting point in the search for efficient personalized algorithms.

For unsupervised problems where the quality of a model is hard to evaluate, implementing the dichotomous strategy requires estimating an upper bound $R$ of the level of heterogeneity. This is an important open problem, which we leave for future work.

Optimality of FedAvg followed by local fine tuning.

Another popular baseline algorithm for personalized FL is to first run FedAvg until convergence, and then let each client run PureLocalTraining to fine tune the model. In strongly convex problems, global optima can be reached by gradient descent regardless the initialization with a suitable choice of the learning rate (see, e.g., Theorem 2.1.15 of Nesterov 2018). Thus, if each client run PureLocalTraining for long enough, the global optima for its local loss function will finally be reached. This fact tells that along the whole fine tuning trajectory, there is a point at which the model gives the worst-case optimal AER and IER, and for a fixed level of heterogeneity, this point is either at the very beginning (which is FedAvg), or at the very end (which is PureLocalTraining). Although this conclusion is almost trivial from a technical point of view given our minimax results, it provides a reassuring theoretical property (of being minimax optimal) for a popular method used by practitioners.

Illustrating the minimaxity in a simulated example.

We conduct a simulation on federated logistic regression to corroborate our theoretical results and the optimality of the FedAvg following by local fine tuning strategy. In the simulation, we set $m=5,n_i=100,\forall i\in \left[m\right]$ and we vary $R$ from 0 to 20 (see Appendix D for details). In the left panel of Figure 1, we plot the test accuracy (averaged over 100 rounds of simulations) of those three methods against the value of $R$. One can see that the accuracy of the fine tuning strategy roughly follows the maximum of the accuracies achieved by FedAvg and PureLocalTraining, confirming our theoretical prediction that the fine tuning strategy can indeed perform as well as the best between FedAvg and PureLocalTraining.

Figure 1:

Average classification accuracy of FedAvg, PureLocalTraining and FedAvg followed by fine tuning (left panel) as well as FedProx with different choice of $\lambda$ (right panel).

Beyond the current heterogeneity assumption.

Our minimax results are established under Assumption B, which states that all optimal local models are close to a certain “centroid” (i.e., the average global model defined in (7)). If we draw a graph of clients and connect two clients if their optimal local models are similar, then the current heterogeneity assumption gives rise to a complete graph (or a star-shaped graph if we introduce another node to represent the average global model). While such an idealized graph structure enables a clean theoretical analysis, the real world proximity patterns among clients are clearly far more sophisticated. In fact, counterexamples exist under which the minimax results does not hold in a “global” sense.

Suppose all $m$ clients exhibit a clustering structure as follows. We have $\sqrt{m}$ clients whose optimal global models serve as cluster centroids, and those centroids are very far apart. In the neighborhood of each centroid, there are $\sqrt{m}$ clients whose optimal local models are $R_c$ away from the centroid in the sense of Assumption B. Additionally, assume each client has equal sample sizes, so that $n_i=n$ for some $n$. Under this setting, the “global” heterogeneity parameter $R$ in Assumption B is very large, so our theory would suggest choosing PureLocalTraining, which gives a rate of $𝒪(1/n)$. However, this rate is clearly suboptimal. If one can successfully cluster the $m$ clients into $\sqrt{m}$ clusters (which is hopeful as the centroids are assumed to be far apart), then one can apply our theory to each cluster (i.e., run the dichotomous strategy for each cluster) and conclude that the rate for each cluster is $𝒪\left(\frac{1}{n\sqrt{m}}+\frac{1}{n}\wedge R_c^2\right)\ll \frac{1}{n}$ if $m$ diverges to infinity and $R_c\ll 1/\sqrt{n}$. The foregoing discussion reveals in such a clustered setting, our theoretical results only make sense at the cluster level, but not at the global level.

The behaviors of the minimax rate for more general client proximity graphs can be even more complicated, which we leave for future work.

3.4. More Implications of Federated Stability and Analysis of FedProx

In this subsection, we are concerned with the performance guarantees for FedProx. As with our earlier analysis of FedAvg, we consider a $\mathbf{p}$-weighted version of FedProx, whose optimization formulation is given below:

$$\underset{\begin{array}{c}{\mathit{w}}^{\left(\text{global}\right)}\in 𝒲\\ {\left\{{\mathit{w}}^{\left(i\right)}\right\}}_{i=1}^m\subseteq 𝒲\end{array}}{\text{min}}\sum _{i\in \left[m\right]}{p}_i\left(L_i\left({\mathit{w}}^{\left(i\right)},S_i\right)+\frac{\lambda }{2}{∥{\mathit{w}}^{\left(\text{global}\right)}-{\mathit{w}}^{\left(i\right)}∥}^2\right),$$ (20)

where we recall that $L_i\left(\mathit{w},S_i\right)≔{\sum }_{j\in \left[n_i\right]}\ell \left(\mathit{w},z_j^{\left(i\right)}\right)/n_i$ is the ERM objective for the $i$-th client. In this subsection, we let $\left({\tilde{\mathit{w}}}^{\text{(global)}},\left\{{\tilde{\mathit{w}}}^{\left(i\right)}\right\}\right)$ be the global minimizer of the above problem. Compared to (8), which imposes a “hard” constraint ${\mathit{w}}^{\left(i\right)}={\mathit{w}}^{\left(\text{global}\right)}$, and compared to (3), where there is no constraint at all, the above formulation imposes a “soft” constraint that the norm of $\left({\mathit{w}}^{\text{(global)}}-{\mathit{w}}^{\left(i\right)}\right)$ should be small, with a hyperparameter $\lambda$ controlling the strength of this constraint.

The rationale behind the optimization formulation (20) of FedProx is clear: by setting $\lambda =0$, the optimization formulation of PureLocalTraining (3) is recovered, and as $\lambda \to \infty$, the optimization formulation of FedAvg (8) is recovered. The hope is that by varying $\lambda \in (0,\infty )$, one can interpolate between the two extremes.

Applying the idea of local SGD to (20), one obtains the FedProx algorithm¹, which we detail in Algorithm 2. We separate the whole algorithm into two stages as they has distinct interpretations: in Stage I, the central server aims to learn a good global model with the help of local clients, whereas in Stage II, each local client takes advantage of the global model to personalize. Alternatively, one can also interpret FedProx as an instance of the general framework of model-agnostic meta learning (Finn et al., 2017), where Stage I learns a good initialization, and Stage II trains the local models starting from this initialization.

In contrast to our analyses for FedAvg and PureLocalTraining in Section 3.1, where we largely focused on global minimizers, the analysis for FedProx will be carried out for the approximate minimizer output by Algorithm 2. The reason for this is rooted in the tradeoff between the optimization error and the generalization error. Note that given the results derived in Section 3.1, the analysis for the global minimizer $\left({\tilde{\mathit{w}}}^{\left(\text{global}\right)},\left\{{\tilde{\mathit{w}}}^{\left(i\right)}\right\}\right)$ becomes trivial: by setting $\lambda =0$, we reduce the task to analyzing PureLocalTraining; by sending $\lambda \to \infty$, we reduce the task to analyzing FedAvg. Based on Theorems 4 and 6, one immediately concludes that there exists a choice of $\lambda$, such that the AER and IER of $\left\{{\tilde{\mathit{w}}}^{\left(i\right)}\right\}$ satisfy the bounds in (18) and (19), respectively. However, the foregoing discussion is purely restricted to generalization error. When we set $\lambda =0$ or send $\lambda \to \infty$, it is not known a priori whether FedProx algorithm will converge to the global minima. Worse still, the optimization error may depends on $\lambda$ in a particular way so that it becomes unbounded when $\lambda$ approaches zero or infinity. To the best of our knowledge, prior work only proved the optimization convergence of FedProx for the global model with a fixed value of $\lambda$, namely the convergence of ${\mathit{w}}_T^{\text{(global)}}$ to ${\tilde{\mathit{w}}}^{\text{(global)}}$ as the number of global communication rounds $T$ tends to infinity (Li et al., 2018; Dinh et al., 2020). To have a theoretical understanding of the performance of FedProx, it is crucial to (1) establish the optimization convergence for both global and local models; (2) bound the generalization error; and (3) balance the optimization error and the generalization error, both of which are functions of $\lambda$. In the following, we execute the those steps with the aid of federated stability.

Implications of federated stability for FedProx.

We have briefly mentioned the main implications of federated stability in Section 3.1.2: for an algorithm $𝒜=\left\{{\widehat{\mathit{w}}}^{\left(i\right)}\right\}$ with federated stability $\left\{{\gamma }_i\right\}$, its average generalization error (resp. individualized generalization error) can be upper bounded by $𝒪\left({\sum }_{i\in \left[m\right]}{p}_i{\gamma }_i\right)$ (resp. $𝒪\left({\gamma }_i\right)$), plus a term scaling with the level of heterogeneity $R$. We make such a statement precise here. Let us first define the optimization error of a generic algorithm $𝒜=\left({\widehat{\mathit{w}}}^{\left(\text{global}\right)},\left\{{\widehat{\mathit{w}}}^{\left(i\right)}\right\}\right)$ (which tries to solve (20)) as

$${ℰ}_{\text{OPT}}≔\sum _{i\in \left[m\right]}{p}_i\left(L_i\left({\widehat{\mathit{w}}}^{\left(i\right)},S_i\right)+\frac{\lambda }{2}{∥{\widehat{\mathit{w}}}^{\left(\text{global }\right)}-{\widehat{\mathit{w}}}^{\left(i\right)}∥}^2\right)-\sum _{i\in \left[m\right]}{p}_i\left(L_i\left({\tilde{\mathit{w}}}^{\left(i\right)},S_i\right)+\frac{\lambda }{2}{∥{\tilde{\mathit{w}}}^{\left(\text{global }\right)}-{\tilde{\mathit{w}}}^{\left(i\right)}∥}^2\right).$$

The main implications of federated stability, when applied to the specifics of FedProx, can then be summarized in the following proposition.

Proposition 10 (Implications of federated stability restricted to FedProx) Consider an algorithm $𝒜=\left({\widehat{\mathit{w}}}^{\text{(global)}},\left\{{\widehat{\mathit{w}}}^{\left(i\right)}\right\}\right)$ with federated uniform stability ${\left\{{\gamma }_i\right\}}_1^m$. Then we have

$${\mathbb{E}}_{𝒜,\mathit{S}}\left[{\text{AER}}_{\mathbf{p}}\left(𝒜\right)\right] \le {\mathbb{E}}_{𝒜,\mathit{S}}\left[{ℰ}_{\text{OPT}}\right]+2\sum _{i\in \left[m\right]}{p}_i{\mathbb{E}}_{𝒜,\mathit{S}}\left[{\gamma }_i\right]+\frac{\lambda }{2}\sum _{i\in \left[m\right]}{p}_i{∥{\mathit{w}}_{\text{avg}}^{\left(\text{global}\right)}-{\mathit{w}}_{\star }^{\left(i\right)}∥}^2,$$ (21)

$${\mathbb{E}}_{𝒜,\mathit{S}}\left[{\text{IER}}_i\left(𝒜\right)\right]\le \frac{{\mathbb{E}}_{𝒜,\mathit{S}}\left[{ℰ}_{\text{OPT}}\right]}{p_i}+2{\mathbb{E}}_{𝒜,\mathit{S}}\left[{\gamma }_i\right]+\frac{\lambda }{2}{\mathbb{E}}_{𝒜,\mathit{S}}{∥{\widehat{\mathit{w}}}^{\left(\text{global}\right)}-{\mathit{w}}_{\star }^{\left(i\right)}∥}^2\forall i\in \left[m\right].$$ (22)

Proof The proof of (21) is based on the following basic inequality for the AER:

$$\sum _{i\in \left[m\right]}{p}_i\left(L_i\left({\tilde{\mathit{w}}}^{\left(i\right)},S_i\right)+\frac{\lambda }{2}{∥{\tilde{\mathit{w}}}^{\left(\text{global}\right)}-{\tilde{\mathit{w}}}^{\left(i\right)}∥}^2\right)\le \sum _{i\in \left[m\right]}{p}_i\left(L_i\left({\mathit{w}}_{\star }^{\left(i\right)},S_i\right)+\frac{\lambda }{2}{∥{\mathit{w}}_{\text{avg}}^{\left(\text{global}\right)}-{\mathit{w}}_{\star }^{\left(i\right)}∥}^2\right),$$ (23)

whereas the proof of (22) is based on the following basic inequality for the IER: for any $s\in \left[m\right]$, we have

$$\sum _{i\in \left[m\right]}{p}_i\left(L_i\left({\tilde{\mathit{w}}}^{\left(i\right)},S_i\right)+\frac{\lambda }{2}{∥{\tilde{\mathit{w}}}^{\left(\text{global}\right)}-{\tilde{\mathit{w}}}^{\left(i\right)}∥}^2\right)\le p_s{L}_s\left({\mathit{w}}_{\star }^{\left(s\right)},S_s\right)+\frac{\lambda }{2}{∥{\widehat{\mathit{w}}}^{\left(\text{global}\right)}-{\mathit{w}}_{\star }^{\left(s\right)}∥}^2+\sum _{i\ne s}{p}_i\left(L_i\left({\widehat{\mathit{w}}}^{\left(i\right)},S_i\right)+\frac{\lambda }{2}{∥{\widehat{\mathit{w}}}^{\left(\text{global}\right)}-{\widehat{\mathit{w}}}^{\left(i\right)}∥}^2\right).$$ (24)

We refer the readers to Appendix C.2 for details. ■

Note that both bounds in Proposition 10 involve a term that scales linearly with both $\lambda$ and the heterogeneity measure. In general, we expect the stability measures to scale inversely with $\lambda$, and thus opening the possibility of carefully choosing $\lambda$ to balance the stability term and the heterogeneity term.

Let us observe that the heterogeneity term of (22) is slightly different than that of (21), in that it involves the estimated global model ${\widehat{\mathit{w}}}^{\text{(global)}}$. This suggests that achieving the IER guarantees might be intrinsically more difficult than achieving the AER guarantees.

In view of Proposition 10, we are left to bound the optimization error and the federated stability of FedProx. As discussed above, achieving the AER and IER guarantees requires somewhat different assumptions, as the latter involves characterizing the performance of the global model. So we split our discussion into two parts below.

Bounding the average excess error.

The following theorem characterize the performance of FedProx in terms of the AER.

Theorem 11 (AER guarantees for FedProx) Let Assumptions A and B(a) hold, and assume $n_i\ge 4\beta /\mu$ for all $i\in \left[m\right]$. Choose the weight vector $\mathit{p}$ such that

$$\frac{p_{\text{max}}\sum _{i\in \left[m\right]}{p}_i/n_i}{\sum _{i\in \left[m\right]}{p}_i^2/n_i}\le C_{\mathbf{p}}$$ (25)

for some constant $C_{\mathbf{p}}$, where $p_{\text{max}}={\text{max}}_i{p}_i$. Consider the FedProx algorithm, ${𝒜}_{\text{FP}}$, with the following hyperparameter configuration:

1. In the joint training stage (i.e., $0\le t\le T-1$), set

   $$\begin{gathered} {\eta }_{t,k}^{\left(i\right)}=\frac{1}{(\mu +\lambda )(k+1)},{\eta }_t^{\left(\text{global}\right)}=\frac{2(\mu +\lambda )}{\lambda \mu (t+1)},K_t+1\ge C_1\left({\lambda }^2\vee 1\right)t, \\ T\ge C_2\lambda (\lambda \vee 1)m\Vert \mathbf{p}{\Vert }^2\cdot \left({\left[\sum _{i\in \left[m\right]}{p}_i/n_i\right]}^{-1}\vee \left[\lambda (\lambda \vee 1)n_{\text{max}}^2\right]\right); \end{gathered}$$ (26)
2. In the final training stage (i.e., $t=T$), set

   $$\begin{gathered} {\eta }_{T,k}^{\left(i\right)}=\frac{1}{(\mu +\lambda )(k+1)}, \\ {K}_T\ge C_3(\lambda +1{)}^2\cdot \left({\left[\sum _{i\in \left[m\right]}\frac{p_i}{n_i}\right]}^{-1}\vee \left[{\lambda }^2\underset{i\in \left[m\right]}{\text{max}}{\left(p_i{n}_i\right)}^2\right]\right), \end{gathered}$$ (27)

where $C_1,C_2,C_3$ are constants depending only on $\left(\mu ,\beta ,\Vert \ell {\Vert }_{\infty },D\right)$. Then, there exists a choice of $\lambda$ such that

$${\mathbb{E}}_{{𝒜}_{\text{FP}},\mathit{S}}\left[{\text{AER}}_{\mathbf{p}}\left({𝒜}_{\text{FP}}\right)\right]\lesssim \left(\frac{\mu }{1\wedge C_{\mathbf{p}}}+\frac{\left(1\vee C_{\mathbf{p}}\right)\beta \Vert \ell {\Vert }_{\infty }}{\mu }\right)\left[\left(R\sqrt{\sum _{i\in \left[m\right]}\frac{p_i}{n_i}}\right)\wedge \left(\sum _{i\in \left[m\right]}\frac{p_i}{n_i}\right)+\sum _{i\in \left[m\right]}\frac{p_i^2}{n_i}\right].$$ (28)

Proof See Appendix C.3. ■

A few remarks are in order. First, (25) essentially says that the weight $\mathbf{p}$ cannot be too imbalanced, and too much imbalance in $\mathbf{p}$ can hurt the performance in view of the multiplicative factor of $C_{\mathbf{p}}$ in our bound (28). If we set $p_i=1/m$, then $C_{\mathbf{p}}$ is naturally of constant order; whereas if we set $p_i=n_i/N$, we have $C_{\mathbf{p}}\asymp m{n}_{\text{max}}/N$, where $n_{\text{max}}={\text{max}}_i{n}_i$, which calls for relative balance of the sample sizes.

We then briefly comment on the hyperparameter choice in the above theorem. The step sizes are of the form 1/(strongly convex constant × iteration counter), and such a choice is common in strongly convex stochastic optimization problems (see, e.g., Rakhlin et al. 2011; Shamir and Zhang 2013). Such a choice, along with the smoothness of the problem, is also the key for us to by-pass the need of doing any time-averaging operation, as is done in, for example, Dinh et al. (2020).

In Theorem 11, the choice of the communication rounds $T$ and the final local training round $K_T$ both scale polynomially with $\lambda$, which means that the optimization convergence of FedProx is slower when the data are less heterogeneous. This phenomenon happens more generally. For example, in Hanzely and Richtárik (2020), they proposed a variant of SGD that optimizes (20) with $p_i=1/m$ in $𝒪\left(\frac{L+\lambda }{\mu }\text{log}1/\epsilon \right)$-many iterations, where $L$ is the Lipschitz constant of the loss function and $\epsilon$ is the desired accuracy level.

The constants $C_1,C_2,C_3$ in the statement of Theorem 11 can be explicitly traced in our proof. We remark that the dependence on problem-specific constants $\left(\mu ,\beta ,\Vert \ell {\Vert }_{\infty },D\right)$ in our hyperparameter choice and on $\lambda$ may not be tight. A tight analysis of the optimization error is interesting, but less relevant for our purpose of understanding the sample complexity. So we defer such an analysis to future work².

Bounding individualized excess errors.

The following theorem gives the IER guarantees for FedProx.

Theorem 12 (IER guarantees for FedProx) Let Assumptions A and B(b) hold. Moreover, assume that $n_i\asymp n_{i^{\prime }}$ for any $i\ne i^{\prime }\in \left[m\right]$ and $n_i\ge 4\beta /\mu \forall i\in \left[m\right]$. Let the weight vector be chosen as $p_i\asymp 1/m\forall i\in \left[m\right]$. Consider the FedProx algorithm, ${𝒜}_{\text{FP}}$, with the following hyperparameter configuration:

1. In the joint training stage (i.e., $0\le t\le T-1$), set ${\eta }_{t,k}^{\left(i\right)},{\eta }_t^{\text{(global)}},K_t$ as in (26), and set

   $$T\ge C_2^{\prime }\lambda (\lambda \vee 1)\underset{i\in \left[m\right]}{\text{max }}{n}_i\cdot \left(p_i^{-1}\vee \left[\lambda (\lambda \vee 1)n_i\right]\right);$$
2. In the final training stage (i.e., $t=T$), set ${\eta }_{T,k}^{\left(i\right)}$ as in (27), and set

   $$K_T\ge C_3^{\prime }(\lambda +1{)}^2\underset{i\in \left[m\right]}{\text{max }}{n}_i\left(p_i^{-1}\vee {\lambda }^2{p}_i^2{n}_i\right),$$

where $C_2^{\prime },C_3^{\prime }$ are constants only depending on $\left(\mu ,\beta ,\Vert \ell {\Vert }_{\infty },D\right)$. Then, there exists a choice of $\lambda$ such that for any $i\in \left[m\right]$, we have

$${\mathbb{E}}_{{𝒜}_{\text{FP}},\mathit{S}}\left[{\text{IER}}_i\left({𝒜}_{\text{FP}}\right)\right]\lesssim \left[\left(\mu +{\mu }^{-1}\right)\left(\beta \Vert \ell {\Vert }_{\infty }+\frac{{\sigma }^2{\beta }^2+{\beta }^2+{\sigma }^2}{{\mu }^2}\right)+\mu D^2\right]\cdot \left(\frac{R}{\sqrt{n_i}}\wedge \frac{1}{n_i}+\frac{\sqrt{m}}{N}\right).$$ (29)

Proof See Appendix C.4. ■

Compared to Theorem 11, the above theorem imposes extra assumptions that the sample sizes are relative balanced and that $p_i\asymp 1/m$, both of which are due to the fact that we need to additionally take care of the estimation error of the global model. The hyperparameter choice slightly differs from that in Theorem 11 for the same reason. In practice, when one is to use FedProx to optimize highly non-convex functions like the loss function of deep neural networks, instead of sticking to the choices made in Theorems 11 and 12, the hyperparameters are usually tuned by trial-and-error for best test performance.

Comparison with the lower bounds.

In order to comment about the optimality/suboptimality of FedProx, let us restrict to the case when $p_i=n_i/N$. In this case, the bound in Theorem 11 becomes

$${\mathbb{E}}_{{𝒜}_{\text{FP}},\mathit{S}}\left[{\text{AER}}_{\mathbf{p}}\left({𝒜}_{\text{FP}}\right)\right]\lesssim \left(\mu +\frac{\beta \Vert \ell {\Vert }_{\infty }}{\mu }\right)\cdot \left(\frac{1}{N/m}\wedge \frac{R}{\sqrt{N/m}}+\frac{1}{N}\right).$$ (30)

Recall the lower bound in (16). Focusing on the dependence of sample sizes and heterogeneity measure, we have the following three cases. If $R^2\gtrsim m/N$, then (30) becomes $𝒪(m/N)$, which matches the lower bound. Meanwhile, if $1/mN\lesssim R^2\lesssim m/N$, then (30) becomes $𝒪(m/N)$, whereas the lower bound reads $\Omega \left(R^2+1/N\right)$, and thus (30) is suboptimal unless $R^2\asymp m/N$. Moreover, if $R^2\lesssim 1/mN$, then (30) becomes $𝒪(1/N)$, and is minimax optimal again.

A similar trilogy holds for IER of FedProx. Comparing the upper bound in (29) and the lower bound in (17), we still have three cases as follows. If $R^2\gtrsim m/N$, then (29) is $𝒪\left(1/n_i\right)$, which agrees with the lower bound. Meanwhile, if $1/N\lesssim R^2\lesssim m/N$, then (29) is $𝒪\left(R/\sqrt{n_i}\right)$, and is suboptimal compared to the $\Omega \left(R^2+1/N\right)$ lower bound unless $R^2\asymp m/N$. Moreover, if $R^2\lesssim 1/N$, then (29) is $𝒪(\sqrt{m}/N)$, and is off by a factor of order $\sqrt{m}$ compared to the $\Omega (1/N)$ lower bound.

While the bounds in Theorems 11 and 12 in general do not attain the lower bounds in Corollary 9, they are still non-trivial in the sense that they scale with the heterogeneity measure $R$. While there are some recent works establishing the AER guarantees for an objective similar to (20) under the online learning setup (see, e.g., Denevi et al. 2019; Balcan et al. 2019; Khodak et al. 2019), to the best of our knowledge, Theorems 11 and 12 are the first to establish both the AER and IER guarantees for (20) under the federated learning setup.

Curious readers may wonder if the suboptimality of the theoretical guarantees for FedProx (with non-zero $\lambda$) is a characteristic of this algorithm or if it is due to the artifact of our technical proof. To answer this question, we conduct a simulation where we apply FedProx with different $\lambda \text{s}$ on datasets generated by federated logistic regression (see Appendix D for details). The accuracies versus different values of $R$ is shown in the right panel of Figure 1. As expected by our theory, the performance of FedProx with $\lambda =0$ mimics that of PureLocalTraining, whereas the performance with $\lambda =4$ resembles that of FedAvg. Interestingly, FedProx with $\lambda =0.44$ bears a similar performance with the FedAvg followed by fine tuning strategy, which we know is minimax optimal. This observation supports the conjecture that optimally tuned FedProx is indeed minimax optimal, and the suboptimality of bounds from Theorems 11 and 12 are likely to be a consequence of the artifact of our theoretical analysis.

4. Discussion

This paper studies the statistical properties of personalized federated learning. Focusing on strongly-convex, smooth, and bounded empirical risk minimization problems, we have uncovered an intriguing phenomenon that given a specific level of heterogeneity, exactly one of FedAvg or PureLocalTraining is minimax optimal. In the course of proving this result, we obtained a novel analysis of FedProx and introduced a new notion of algorithmic stability termed federated stability, which is possibly of independent interest for analyzing generalization properties in the context of federated learning.

We close this paper by mentioning several open problems.

• Dependence on problem-specific parameters. This paper focuses on the dependence on the sample sizes, and in our bounds, the dependence on problem-specific parameters (e.g., the smoothness and strong convexity constants) may not be optimal. This can be problematic if those parameters are not of constant order, and it will be interesting to give a refined analysis that gives optimal dependence on those parameters.

• A refined analysis of FedProx. The upper bounds we develop for FedProx, as we have mentioned, do not match our minimax lower bounds. According to a simulated example, we suspect that this is an artifact of our analysis and a refined analysis of FedProx would be a welcome advance.

• Estimation of the level of heterogeneity and development of adaptive algorithms. For unsupervised problems where evaluation of a model is difficult, implementation of the oracle dichotomous strategy described in Section 3.3 would require estimating the level of heterogeneity $R$. Even for supervised problems, estimation of $R$ would be interesting, as it allows one to decide which algorithm to choose without model training. More generally, developing adaptive algorithms that attains the lower bound without prior information of $R$ is an important open problem.

• Beyond the current heterogeneity assumption. As discussed in Section 3.3, our theoretical results may not hold globally when one moves from Assumption B to more general heterogeneity assumptions. Establishing the minimax rates and designing provably optimal algorithms under those assumptions are of both theoretical and practical interest.

• Beyond convexity. Our analysis is heavily contingent upon the strong convexity of the loss function, which, to the best of our knowledge, is not easily generalizable to the non-convex case. Meanwhile, our notion of heterogeneity, which is based on the distance of optimal local models to the convex combination of them, may not be natural for non-convex problems. It is of interest, albeit difficult, to have a theoretical investigation of personalized federated learning for non-convex problems.

A. Proof of Theorem 8: Lower Bounds

We start by presenting a lower bound when all ${\mathit{w}}_{\star }^{\left(i\right)}$′s are the same.

Lemma 13 (Lower bound under homogeneity) Consider the logistic regression model with ${\mathit{w}}_{\star }^{\left(i\right)}={\mathit{w}}_{\mathbf{p}}^{\left(\text{global}\right)}$ for any $i\in \left[m\right]$. Then

$$\underset{{\widehat{\mathit{w}}}^{\text{(global) }}}{\text{inf}}\underset{{\mathit{w}}_{\mathbf{p}}^{\left(\text{global }\right)}}{\text{sup}}{\mathbb{E}}_{\mathit{S}}{∥{\widehat{\mathit{w}}}^{\left(\text{global }\right)}-{\mathit{w}}_{\mathbf{p}}^{\left(\text{global }\right)}∥}^2\gtrsim \frac{d}{N}.$$

Proof This is a classical result. See, e.g., Example 8.4 of Duchi (2019). ■

Proof [Proof of (14)] We first give a lower bound based on the observation that the homogeneous case is in fact included in the parameter space ${𝒫}_1$. More explicitly, let us define ${𝒫}_0=\left\{\left\{{\mathit{w}}_{\star }^{\left(i\right)}\right\}\in {𝒫}_1:{\mathit{w}}_{\star }^{\left(i\right)}={\mathit{w}}_{\mathbf{p}}^{\text{(global)}}\forall i\in \left[m\right]\right\}$. By Lemma 13, we have

$$\begin{gathered} \underset{\left\{{\widehat{\mathit{w}}}^{\left(i\right)}\right\}}{\text{inf}}\underset{\left\{{\mathit{w}}_{\star }^{\left(i\right)}\right\}\in {𝒫}_1}{\text{sup}}\sum _{i\in \left[m\right]}{p}_i{\mathbb{E}}_{\mathit{S}}{∥{\widehat{\mathit{w}}}^{\left(i\right)}-{\mathit{w}}_{\star }^{\left(i\right)}∥}^2\ge \underset{\left\{{\widehat{\mathit{w}}}^{\left(i\right)}\right\}}{\text{inf}}\underset{\left\{{\mathit{w}}_{\star }^{\left(i\right)}\right\}\in {𝒫}_0}{\text{sup}}\sum _{i\in \left[m\right]}{p}_i{\mathbb{E}}_{\mathit{S}}{∥{\widehat{\mathit{w}}}^{\left(i\right)}-{\mathit{w}}_{\star }^{\left(i\right)}∥}^2 \\ =\underset{{\widehat{\mathit{w}}}^{\left(\text{global}\right)}}{\text{inf}}\underset{{\mathit{w}}_{\mathbf{p}}^{\left(\text{global}\right)}}{\text{sup}}{\mathbb{E}}_{\mathit{S}}{∥{\widehat{\mathit{w}}}^{\left(\text{global}\right)}-{\mathit{w}}_{\mathbf{p}}^{\left(\text{global}\right)}∥}^2 \\ \gtrsim \frac{d}{N}. \end{gathered}$$ (31)

We now use a variant of Assouad’s method (Assouad, 1983) that allows us to tackle multiple datasets. Consider the following data generating process: nature generates $\mathit{V}=\left\{{\mathit{v}}^{\left(i\right)}:i\in \left[m\right]\right\}$ i.i.d. from the uniform distribution on $𝒱=\{\pm 1{\}}^d$ and sets ${\mathit{w}}_{\star }^{\left(i\right)}={\delta }_i{\mathit{v}}^{\left(i\right)}$ for some ${\delta }_i$ such that the following constraint is satisfied:

$$\sum _{i\in \left[m\right]}{p}_i{∥{\mathit{w}}_{\star }^{\left(i\right)}-{\mathit{w}}_{\mathbf{p}}^{\left(\text{global}\right)}∥}^2=\sum _{i\in \left[m\right]}{p}_i{∥{\delta }_i{\mathit{v}}^{\left(i\right)}-\sum _{s\in \left[m\right]}{p}_s{\delta }_s{\mathit{v}}^{\left(s\right)}∥}^2\le R^2.$$ (32)

We will specify the choice of ${\delta }_i$’s later. Denoting ${\mathbb{E}}_{\mathit{X}}$ as the marginal expectation operator with respect to all the features $\left\{{\mathit{x}}_j^{\left(i\right)}\right\}$ and ${\mathbb{E}}_{\mathit{Y}|\mathit{X}}$ as the conditional expectation operator with respect to $\left\{y_j^{\left(i\right)}\right\}|\left\{{\mathit{x}}_j^{\left(i\right)}\right\}$, we can lower bound the minimax risk by the Bayes risk as follows:

$$\begin{gathered} \underset{\left\{{\widehat{w}}^{(i)}\right\}}{\text{inf}}\underset{\left\{w_{\star }^{(i)}\right\}\in 𝒫}{\text{sup}}\sum _{i\in [m]}{p}_i{\mathbb{E}}_S{\Vert {\widehat{w}}^{(i)}-w_{\star }^{(i)}\Vert }^2 \\ \ge \underset{\left\{{\widehat{w}}^{(i)}\right\}}{\text{inf}}{\mathbb{E}}_{\left\{v^{(i)}\right\}}\sum _{i\in [m]}{p}_i{\mathbb{E}}_S{\Vert {\widehat{w}}^{(i)}-{\delta }_i{v}^{(i)}\Vert }^2 \\ =\underset{\left\{{\widehat{v}}^{(i)}\right\}\subseteq 𝒱}{\text{inf}}\sum _{i\in [m]}{p}_i{\mathbb{E}}_{V,S}{\Vert {\delta }_i{\widehat{v}}^{(i)}-{\delta }_i{v}^{(i)}\Vert }^2 \\ \ge {\mathbb{E}}_X\sum _{i\in [m]}{p}_i{\delta }_i^2\underset{{\widehat{v}}^{(i)}\in 𝒱}{\text{inf}}{\mathbb{E}}_{V,Y|X}{\Vert {\widehat{v}}^{(i)}-v^{(i)}\Vert }^2 \\ \ge {\mathbb{E}}_X\sum _{i\in [m]}{p}_i{\delta }_i^2\sum _{k\in [d]}\underset{{\widehat{v}}_k^{(i)}\in \{\pm 1\}}{\text{inf}}{\mathbb{E}}_{V,Y|X}{\left({\widehat{v}}_k^{(i)}-v_k^{(i)}\right)}^2 \\ \gtrsim {\mathbb{E}}_X\sum _{i\in [m]}{p}_i{\delta }_i^2\sum _{k\in [d]}\underset{{\widehat{v}}_k^{(i)}\in \{\pm 1\}}{\text{inf}}{\mathbb{P}}_{V,Y|X}\left({\widehat{v}}_k^{(i)}\ne v_k^{(i)}\right) \\ =\frac{1}{2}{\mathbb{E}}_X\sum _{i\in [m]}{p}_i{\delta }_i^2\sum _{k\in [d]}\underset{{\widehat{v}}_k^{(i)}\in \{\pm 1\}}{\text{inf}}\left({\mathbb{P}}_{i,+k}\left({\widehat{v}}_k^{(i)}=-1\right)+{\mathbb{P}}_{i,-k}\left({\widehat{v}}_k^{(i)}=+1\right)\right) \end{gathered}$$

where in the last line, we have let ${\mathbb{P}}_{i,\pm k}(\cdot )={\mathbb{P}}_{\mathit{V},\mathit{Y}|\mathit{X}}\left(\cdot |{\mathit{v}}_k^{\left(i\right)}=\pm 1\right)$ to denote the probability measure with respect to the randomness in $(\mathit{V},\mathit{S})$ conditional on the features $\left\{{\mathit{x}}_j^{\left(i\right)}\right\}$ as well as the realization of ${\mathit{v}}_k^{\left(i\right)}=\pm 1$. More explicitly, we can write

$$\begin{gathered} {\mathbb{P}}_{i,\pm k}=\left(\underset{s\ne i}{⨂}{\mathbb{P}}_{{\mathit{v}}^{\left(s\right)}}\otimes {\mathbb{P}}_{{\left\{y^{\left(s\right)}\right\}}_{j=1}^{n_s}\mid {\mathit{v}}^{\left(s\right)},{\left\{{\mathit{x}}_j^{\left(s\right)}\right\}}_{j=1}^{n_s}}\right)\otimes \left({\mathbb{P}}_{{\mathit{v}}^{\left(i\right)}\mid {\mathit{v}}_k^{\left(i\right)}=\pm 1}\otimes {\mathbb{P}}_{{\left\{y_j^{\left(i\right)}\right\}}_{j=1}^{n_i}\mid {\mathit{v}}^{\left(i\right)},{\mathit{v}}_k^{\left(i\right)}=\pm 1,{\left\{{\mathit{x}}_j^{\left(i\right)}\right\}}_{j=1}^{n_i}}\right) \\ =\frac{1}{2^{(m-1)d+d-1}}\sum _{\mathit{V}\setminus \left\{{\mathit{v}}_k^{\left(i\right)}\right\}}{\mathbb{P}}_{\mathit{V},i,\pm k}, \end{gathered}$$

where the $\otimes$ symbol stands for taking the product of two measures and ${\mathbb{P}}_{\mathit{V},i,\pm k}$ corresponds to the law of all the labels $\mathit{Y}$ conditional on a specific realization of $\left\{\mathit{V}:{\mathit{v}}_k^{\left(i\right)}=\pm 1\right\}$ and the features $\mathit{X}$. With the current notations and letting $\Vert \mathbb{P}-\mathbb{Q}{\Vert }_{\text{TV}}$ be the total variation distance between two probability measures $\mathbb{P}$ and $\mathbb{Q}$, we can invoke Neyman-Pearson lemma to get

$$\begin{gathered} \underset{\left\{{\widehat{\mathit{w}}}^{\left(i\right)}\right\}}{\text{inf}}\underset{\left\{{\mathit{w}}_{\star }^{\left(i\right)}\right\}\in 𝒫}{\text{sup}}\sum _{i\in \left[m\right]}{p}_i{\mathbb{E}}_{\mathit{S}}{∥{\widehat{\mathit{w}}}^{\left(i\right)}-{\mathit{w}}_{\star }^{\left(i\right)}∥}^2 \gtrsim {\mathbb{E}}_{\mathit{X}}\sum _{i\in \left[m\right]}{p}_i{\delta }_i^2\sum _{k\in \left[d\right]}\left(1-{∥{\mathbb{P}}_{i,+k}-{\mathbb{P}}_{i,-k}∥}_{\text{TV}}\right) \\ =d\sum _{i\in \left[m\right]}{p}_i{\delta }_i^2-{\mathbb{E}}_{\mathit{X}}\sum _{i\in \left[m\right]}{p}_i{\delta }_i^2\sum _{k\in \left[d\right]}{∥{\mathbb{P}}_{i,+k}-{\mathbb{P}}_{i,-k}∥}_{\text{TV}}. \end{gathered}$$ (33)

We then proceed by

$$\begin{gathered} \sum _{i\in \left[m\right]}{p}_i{\delta }_i^2\sum _{k\in \left[d\right]}{∥{\mathbb{P}}_{i,+k}-{\mathbb{P}}_{i,-k}∥}_{\text{TV}} \\ \le \sum _{i\in \left[m\right]}{p}_i{\delta }_i^2\sqrt{d}{\left(\sum _{k\in \left[d\right]}{∥{\mathbb{P}}_{i,+k}-{\mathbb{P}}_{i,-k}∥}_{\text{TV}}^2\right)}^{1/2} \\ =\sum _{i\in \left[m\right]}{p}_i{\delta }_i^2\sqrt{d}{\left(\sum _{k\in \left[d\right]}{∥\frac{1}{2^{(m-1)d+d-1}}\sum _{\mathit{V}\setminus \left\{{\mathit{v}}_k^{\left(i\right)}\right\}}{\mathbb{P}}_{\mathit{V},i,+k}-{\mathbb{P}}_{\mathit{V},i,-k}∥}_{\text{TV}}^2\right)}^{1/2} \\ =\sum _{i\in \left[m\right]}{p}_i{\delta }_i^2\sqrt{d}{\left(\sum _{k\in \left[d\right]}\frac{1}{2^{(m-1)d+d-1}}\sum _{\mathit{V}\setminus \left\{{\mathit{v}}_k^{\left(i\right)}\right\}}{∥{\mathbb{P}}_{\mathit{V},i,+k}-{\mathbb{P}}_{\mathit{V},i,-k}∥}_{\text{TV}}^2\right)}^{1/2}, \end{gathered}$$

where the last inequality is by convexity of the total variation distance. Note that ${\mathbb{P}}_{\mathit{V},i,\pm k}$ is the product of biased Rademacher random variables: if we let $\text{Rad}\left(p\right)$ be the $\pm 1$-valued random variable with positive probability $p$, we can write

$${\mathbb{P}}_{\mathit{V},i,\pm k}=\underset{s\in \left[m\right]}{⨂}\underset{j\in \left[n_s\right]}{⨂}\text{Rad}\left(\frac{1}{1+\text{exp}\left\{-{\delta }_s⟨{\mathit{v}}^{\left(s\right)},{\mathit{x}}_j^{\left(s\right)}⟩\right\}}\right),{\mathit{v}}_k^{\left(i\right)}=\pm 1.$$

Thus, by Pinsker’s inequality, we have

$$\begin{gathered} {∥{\mathbb{P}}_{\mathit{V},i,+k}-{\mathbb{P}}_{\mathit{V},i,-k}∥}_{\text{TV}}^2 \\ \le \frac{1}{2}{D}_{\text{JS}}\left({\mathbb{P}}_{\mathit{V},i,+k}\Vert {\mathbb{P}}_{\mathit{V},i,-k}\right) \\ =\frac{1}{2}\sum _{s\ne i}\sum _{j\in \left[n_s\right]}0+\frac{1}{2}\sum _{j\in \left[n_i\right]}{D}_{\text{JS}}\left[\text{Rad}\left(\frac{1}{1+\text{exp}\left\{-{\delta }_i\left.⟨{\mathit{v}}^{\left(i\right)},{\mathit{x}}_j^{\left(i\right)}\right\}\right.}\right)\Vert \text{Rad}\left(\frac{1}{1+\text{exp}\left\{-{\delta }_i\left.⟨{\tilde{\mathit{v}}}^{\left(i\right)},{\mathit{x}}_j^{\left(i\right)}\right\}\right.}\right)\right], \end{gathered}$$

where $D_{\text{JS}}(\mathbb{P}\Vert \mathbb{Q})=\frac{D_{\text{KL}}(\mathbb{P}\Vert \mathbb{Q})+D_{\text{KL}}(\mathbb{Q}\Vert \mathbb{P})}{2}$ is the Jensen-Shannon divergence between $\mathbb{P}$ and $\mathbb{Q}$, and ${\mathit{v}}^{\left(s\right)},{\tilde{\mathit{v}}}^{\left(s\right)}$ are two $𝒱$-valued vectors that only differs in the $k$-th coordinate. By a standard calculation, one finds that

$$\begin{gathered} D_{\text{JS}}\left[\text{Rad}\left(\frac{1}{1+\text{exp}\left\{-{\delta }_i\left.⟨{\mathit{v}}^{\left(i\right)},{\mathit{x}}_j^{\left(i\right)}\right\}\right.}\right)\Vert \text{Rad}\left(\frac{1}{1+\text{exp}\left\{-{\delta }_i\left.⟨{\tilde{\mathit{v}}}^{\left(i\right)},{\mathit{x}}_j^{\left(i\right)}\right\}\right.}\right)\right]\le {\delta }_i^2{\left({\mathit{v}}_k^{\left(i\right)}-{\tilde{\mathit{v}}}_k^{\left(i\right)}\right)}^2{\left({\mathit{x}}_{j,k}^{\left(i\right)}\right)}^2 \\ =4{\delta }_i^2{\left({\mathit{x}}_{j,k}^{\left(i\right)}\right)}^2. \end{gathered}$$

This gives

$${∥{\mathbb{P}}_{\mathit{V},i,+k}-{\mathbb{P}}_{\mathit{V},i,-k}∥}_{\text{TV}}^2\le 2{\delta }_i^2\sum _{j\in \left[n_i\right]}{\left({\mathit{x}}_{j,k}^{\left(i\right)}\right)}^2\le 2{\delta }_i^2{c}_X^2{n}_i.$$

and hence

$$\sum _{i\in \left[m\right]}{p}_i{\delta }_i^2\sum _{k\in \left[d\right]}{∥{\mathbb{P}}_{i,+k}-{\mathbb{P}}_{i,-k}∥}_{\text{TV}}\le \sqrt{2}{c}_X\sum _{i\in \left[m\right]}{p}_i{\delta }_i^{3}d{n}_i^{1/2}.$$

Plugging the above display to (33) gives

$$\underset{\left\{{\widehat{\mathit{w}}}^{\left(i\right)}\right\}}{\text{inf}}\underset{\left\{{\mathit{w}}_{\star }^{\left(i\right)}\right\}\in 𝒫}{\text{sup}}\sum _{i\in \left[m\right]}{p}_i{\mathbb{E}}_{\mathit{S}}{∥{\widehat{\mathit{w}}}^{\left(i\right)}-{\mathit{w}}_{\star }^{\left(i\right)}∥}^2\gtrsim d\left(\sum _{i\in \left[m\right]}{p}_i{\delta }_i^2-\sqrt{2}{c}_X\sum _{i\in \left[m\right]}{p}_i{\delta }_i^3\sqrt{n_i}\right).$$ (34)

To this end, all that is left is to choose ${\delta }_i$ approriately so that (1) the above display is as tight as possible; (2) (32) is satisfied. We consider the following two cases:

1. Assume $R^2\ge d{\sum }_{i\in \left[m\right]}{p}_i/n_i=dm/N$. Note that we can re-write the requirement (32) to be

   $$d\sum _{i\in \left[m\right]}{p}_i{\delta }_i^2-{∥\sum _{i\in \left[m\right]}{p}_i{\delta }_i{\mathit{v}}^{\left(i\right)}∥}^2\le R^2.$$
   Under the current assumption, this requirement will be satisfied if we choose ${\delta }_i=c/\sqrt{n_i}$ for any $c\le 1$. Under such a choice, the right-hand side of (34) becomes $\frac{c^{2}dm}{N}\left(c-\sqrt{2}{c}_X\right)$. Thus, by setting $c=2\sqrt{2}{c}_X$, we get the following lower bound:

   $$\underset{\left\{{\widehat{\mathit{w}}}^{\left(i\right)}\right\}}{\text{inf}}\underset{\left\{{\mathit{w}}_{\star }^{\left(i\right)}\right\}\in 𝒫}{\text{sup}}\sum _{i\in \left[m\right]}{p}_i{\mathbb{E}}_{\mathit{S}}{∥{\widehat{\mathit{w}}}^{\left(i\right)}-{\mathit{w}}_{\star }^{\left(i\right)}∥}^2\gtrsim \frac{d}{N/m}.$$
2. Assume $R^2\le d{\sum }_{i\in \left[m\right]}{p}_i/n_i=dm/N$. Note that if we set ${\delta }_i\equiv \delta =cR/\sqrt{d}$ where $c\le 1$, (32) reads

   $$c^2{R}^2-{∥\sum _{i\in \left[m\right]}{p}_i{\delta }_i{\mathit{v}}^{\left(i\right)}∥}^2\le R^2,$$

   which trivially holds. Now, the right-hand side of (34) becomes

   $$c^2{R}^2\left(1-\sqrt{2}c{c}_X\sum _{i\in \left[m\right]}{p}_{i}R\sqrt{n_i}/\sqrt{d}\right).$$
   Since $p_i=n_i/N$ and $n_i\asymp N/m$, our assumption on $R$ gives

   $$\sqrt{2}c{c}_X\sum _{i\in \left[m\right]}{p}_{i}R\sqrt{n_i}/\sqrt{d}\lesssim \sum _i\frac{n_i}{N}\cdot \sqrt{\frac{m{n}_i}{N}}=1.$$
   This means that we can choose $c$ to be a small constant such that the following lower bound holds:

   $$\underset{\left\{{\widehat{\mathit{w}}}^{\left(i\right)}\right\}}{\text{inf}}\underset{\left\{{\mathit{w}}_{\star }^{\left(i\right)}\right\}\in 𝒫}{\text{sup}}\sum _{i\in \left[m\right]}{p}_i{\mathbb{E}}_{\mathit{S}}{∥{\widehat{\mathit{w}}}^{\left(i\right)}-{\mathit{w}}_{\star }^{\left(i\right)}∥}^2\gtrsim R^2.$$

Summarizing the above two cases, we arrive at

$$\underset{\left\{{\widehat{\mathit{w}}}^{\left(i\right)}\right\}}{\text{inf}}\underset{\left\{{\mathit{w}}_{\star }^{\left(i\right)}\right\}\in 𝒫}{\text{sup}}\sum _{i\in \left[m\right]}{p}_i{\mathbb{E}}_{\mathit{S}}{∥{\widehat{\mathit{w}}}^{\left(i\right)}-{\mathit{w}}_{\star }^{\left(i\right)}∥}^2\gtrsim \frac{d}{N/m}\wedge R^2.$$

Combining the above bound with (31), we get

$$\underset{\left\{{\widehat{\mathit{w}}}^{\left(i\right)}\right\}}{\text{inf}}\underset{\left\{{\mathit{w}}_{\star }^{\left(i\right)}\right\}\in 𝒫}{\text{sup}}\sum _{i\in \left[m\right]}{p}_i{\mathbb{E}}_{\mathit{S}}{∥{\widehat{\mathit{w}}}^{\left(i\right)}-{\mathit{w}}_{\star }^{\left(i\right)}∥}^2\gtrsim \frac{d}{N/m}\wedge R^2+\frac{d}{N},$$

which is the desired result. ■

Proof [Proof of (15)] The proof is similar to the proof of (16), and we only provide a sketch here. Without loss of generality we consider the first client. By the same arguments as in the proof of (16), the left-hand side of (15) is lower bounded by a constant multiple of $d/N$. Now, by considering the same prior distribution on $𝒫$ as in the proof of (16), we get

$$\underset{{\stackrel{-}{\mathit{w}}}^{\left(1\right)}}{\text{inf}}\underset{\left\{{\mathit{w}}^{\left(i\right)}\right\}\in 𝒫}{\text{sup}}{\mathbb{E}}_{\mathit{S}}{∥{\stackrel{-}{\mathit{w}}}^{\left(1\right)}-{\mathit{w}}_{\star }^{\left(1\right)}∥}^2\gtrsim d{\delta }_1^2\left(1-{\delta }_1\sqrt{n_1}\right),$$

where the ${\delta }_i$ ‘s should obey the following inequality:

$${∥{\delta }_i{\mathit{v}}^{\left(i\right)}-\sum _{s\in \left[m\right]}{p}_s{\delta }_s{\mathit{v}}^{\left(s\right)}∥}^2\le R^2.$$

Choosing ${\delta }_i\asymp 1/\sqrt{n_i}$ when $R\ge dm/N$ and ${\delta }_i\asymp R/\sqrt{d}$ otherwise, we arrive at

$$\underset{{\stackrel{-}{\mathit{w}}}^{\left(1\right)}}{\text{inf}}\underset{\left\{{\mathit{w}}^{\left(i\right)}\right\}\in 𝒫}{\text{sup}}{\mathbb{E}}_{\mathit{S}}{∥{\stackrel{-}{\mathit{w}}}^{\left(1\right)}-{\mathit{w}}_{\star }^{\left(1\right)}∥}^2\gtrsim \frac{d}{n_1}\wedge R^2,$$

and the proof is concluded. ■

B. Optimization Convergence of FedProx

This section concerns the optimization convergence of FedProx. We first introduce some notations. Let ${\mathit{w}}_{t,k}^{\left(i\right)}$ be the output of $k$-th step of Algorithm 2 when the initial local model is given by ${\mathit{w}}_t^{\left(i\right)}\equiv {\mathit{w}}_{t,0}^{\left(i\right)}\equiv {\mathit{w}}_{t-1,K}^{\left(i\right)}$, let $I_{t,k}^{\left(t\right)}$ be the corresponding minibatch taken, and denote the initial global model by ${\mathit{w}}_t^{\text{(global)}}$. Let ${ℱ}_{t,k}$ be the sigma algebra generated by the randomness by Algorithm 2 up to ${\mathit{w}}_{t,k}^{\left(i\right)}$, namely the randomness in $\left\{{𝒞}_{\tau },\left\{I_{\tau ,l}^{\left(i\right)}:i\in {𝒞}_{\tau },0\le {\left.\left.l\le K_{\tau }-1\right\}\right\}}_{\tau =0}^{t-1},{𝒞}_t\right.\right.$, and $\left\{I_{t,l}^{\left(i\right)}:i\in {𝒞}_t,0\le l\le k-1\right\}$. For notational convenience we let ${𝒞}_T=\left[m\right]$ (i.e., all clients are involved in local training in Stage II of Algorithm 2). Then the sequence $\left\{{\mathit{w}}_{t,k}^{\left(i\right)}\right\}$ is adapted to the following filtration:

$${ℱ}_{\mathrm{0,0}}\subseteq {ℱ}_{\mathrm{0,1}}\subseteq \cdots \subseteq {ℱ}_{0,K}\subseteq {ℱ}_{\mathrm{1,0}}\subseteq {ℱ}_{\mathrm{1,1}}\subseteq \cdots \subseteq {ℱ}_{1,K}\subseteq \cdots \subseteq {ℱ}_{T,K}.$$

We write the optimization problem (20) as

$$\underset{{\mathit{w}}^{\left(\text{global}\right)}\in 𝒲}{\text{min}}\sum _{i\in \left[m\right]}{p}_i{F}_i\left({\mathit{w}}^{\left(\text{global}\right)},S_i\right),$$ (35)

where

$$F_i\left({\mathit{w}}^{\left(\text{global}\right)},S_i\right)=\underset{{\mathit{w}}^{\left(i\right)}\in 𝒲}{\text{min}}\left\{L_i\left({\mathit{w}}^{\left(i\right)},S_i\right)+\frac{\lambda }{2}{∥{\mathit{w}}^{\left(\text{global}\right)}-{\mathit{w}}^{\left(i\right)}∥}^2\right\}.$$ (36)

To simplify notations, we introduce the proximal opertor

$${\text{Prox}}_{L_i/\lambda }\left({\mathit{w}}^{\left(\text{global}\right)}\right)={\text{Prox}}_{L_i/\lambda }\left({\mathit{w}}^{\left(\text{global}\right)},S_i\right)={\text{argmin }}_{{\mathit{w}}^{\left(i\right)}\in 𝒲}\left\{L_i\left({\mathit{w}}^{\left(i\right)},S_i\right)+\frac{\lambda }{2}{∥{\mathit{w}}^{\left(\text{global}\right)}-{\mathit{w}}^{\left(i\right)}∥}^2\right\}.$$ (37)

The high-level idea of this proof is to regard $\lambda {\sum }_{i\in {𝒞}_t}\left({\mathit{w}}_t^{\left(\text{global}\right)}-{\mathit{w}}_{t+1}^{\left(i\right)}\right)/B^{\left(\text{global}\right)}$ as a biased stochastic gradient of $\frac{1}{n}{\sum }_{i\in \left[m\right]}{F}_i\left({\mathit{w}}_t^{\left(\text{global}\right)},S_i\right)$. This idea has appeared in various places (see, e.g., the proof of Proposition 5 in Denevi et al. (2019) and the proof of Theorem 1 in Dinh et al. (2020)). However, the implementation of this idea in our case is more complicated than the above mentioned works in that (1) we are not in an online learning setup (compared to Denevi et al. (2019)); (2) we don’t need to assume all clients are training at every round (compared to Dinh et al. (2020)); and (3) we use local SGD for the inner loop (instead of assuming the inner loop can be solved with arbitrary precision as assumed in Dinh et al. (2020)), so the gradient norm depends on $\lambda$, and could in principle be arbitrarily large, which causes extra complications.

Lemma 14 (Convergence of the inner loop) Let Assumption A(a, b) holds. Choose ${\eta }_{t,k}^{\left(i\right)}=\frac{1}{(\mu +\lambda )(k+1)}$. Then for any $k\ge 0$, we have

$$\mathbb{E}\left[{∥{\mathit{w}}_{t,k}^{\left(i\right)}-{\text{Prox}}_{L_i/\lambda }\left({\mathit{w}}_t^{\left(\text{global }\right)}\right)∥}^2|{ℱ}_{t,0},i\in {𝒞}_t\right]\le \frac{8{\beta }^2{D}^2}{{\mu }^2\left(k+1\right)}.$$

Proof See Appendix B.1.■

Lemma 15 (Convergence of the outer loop)Let the assumptions inLemma 14hold. Choose${\eta }_t^{\text{(global)}}=\frac{2(\mu +\lambda )}{\lambda \mu (t+1)}$and assume

$$K_{\tau }+1\ge \frac{\left(4\tau +20\right){\lambda }^2{\beta }^2{D}^2}{{\mu }^2\left({\beta }^2{D}^2\wedge 2\lambda \Vert \ell {\Vert }_{\infty }\wedge {\lambda }^2{D}^2\right)}\forall 0\le \tau \le t-1.$$ (38)

Then for any $t\ge 0$, we have

$${\mathbb{E}}_{{𝒜}_{\text{FP}}}{∥{\mathit{w}}_t^{\left(\text{global}\right)}-{\tilde{\mathit{w}}}^{\left(\text{global}\right)}∥}^2\le \frac{12(\lambda +\mu {)}^{2}m\Vert \mathbf{p}{\Vert }^2\left({\beta }^2{D}^2\wedge 2\lambda \Vert \ell {\Vert }_{\infty }\wedge {\lambda }^2{D}^2\right)}{{\lambda }^2{\mu }^2\left(t+1\right)},$$ (39)

where the expectation is taken over the randomness in Algorithm 2.

Proof See Appendix B.2.■

Proposition 16 (Optimization error of ${𝒜}_{\mathbf{F}\mathbf{P}}$) Under the assumptions of Lemma 14 and 15, for any dataset $\mathit{S}~{⨂}_i{𝒟}_i^{\otimes n_i}$, we have

$${\mathbb{E}}_{{𝒜}_{\text{FP}}}\left[{ℰ}_{\text{OPT}}\right]\le \frac{4(\beta +\lambda ){\beta }^2{D}^2}{{\mu }^2\left(K_T+1\right)}+\frac{6(\lambda +\mu {)}^{2}m\Vert \mathbf{p}{\Vert }^2\left({\beta }^2{D}^2\wedge 2\lambda \Vert \ell {\Vert }_{\infty }\wedge {\lambda }^2{D}^2\right)}{\lambda {\mu }^2(t+1)}$$

Proof By definition we have

$$\begin{gathered} {\mathbb{E}}_{{𝒜}_{\text{FP}}}\left[{ℰ}_{\text{OPT}}\right]≔{\mathbb{E}}_{{𝒜}_{\text{FP}}}\left[\sum _{i\in \left[m\right]}{p}_i\left(L_i\left({\mathit{w}}_{T+1}^{\left(i\right)},S_i\right)+\frac{\lambda }{2}{∥{\mathit{w}}_T^{\left(\text{global }\right)}-{\mathit{w}}_{T+1}^{\left(i\right)}∥}^2\right)-\sum _{i\in \left[m\right]}{p}_i{F}_i\left({\tilde{\mathit{w}}}^{\left(\text{global }\right)},S_i\right)\right] \\ \stackrel{(\text{ a) }}{\le }\sum _{i\in \left[m\right]}\frac{p_i(\beta +\lambda )}{2}\cdot {\mathbb{E}}_{{𝒜}_{\text{FP}}}{∥{\mathit{w}}_{T+1}^{\left(i\right)}-{\text{Prox}}_{L_i/\lambda }\left({\mathit{w}}_T^{\left(\text{global }\right)}\right)∥}^2+{\mathbb{E}}_{{𝒜}_{\text{FP}}}\left[\sum _{i\in \left[m\right]}{p}_i{F}_i\left({\mathit{w}}_T^{\text{(global) }},S_i\right)-\sum _{i\in \left[m\right]}{p}_i{F}_i\left({\tilde{\mathit{w}}}^{\text{(global) }},S_i\right)\right] \\ \stackrel{(\text{ b) }}{\le }\frac{4(\beta +\lambda ){\beta }^2{D}^2}{{\mu }^2\left(K_T+1\right)}+\frac{\lambda }{2}{\mathbb{E}}_{{𝒜}_{\text{FP}}}{∥{\mathit{w}}_T^{\text{(global) }}-{\tilde{\mathit{w}}}^{\text{(global) }}∥}^2 \\ \stackrel{\text{ (c) }}{\le }\frac{4(\beta +\lambda ){\beta }^2{D}^2}{{\mu }^2\left(K_T+1\right)}+\frac{6(\lambda +\mu {)}^{2}m\Vert \mathbf{p}{\Vert }^2\left({\beta }^2{D}^2\wedge 2\lambda \Vert \ell {\Vert }_{\infty }\wedge {\lambda }^2{D}^2\right)}{\lambda {\mu }^2(t+1)}. \end{gathered}$$

where (a) is by smoothness of $L_i$, (b) is by Lemma 14 and $\lambda$-smoothness of ${\sum }_{i\in \left[m\right]}{p}_i{F}_i$ (which holds by Lemma 17), and (c) is by Lemma 15.■

B.1. Proof of Lemma 14: Convergence of the Inner Loop

The proof is an adaptation of the proof of Lemma 1 in Rakhlin et al. (2011). However, we need to deal with the extra complication that the hyperparameter $\lambda$ can in principle be arbitrarily large. We start by noting that

$$\begin{gathered} {∥{\mathit{w}}_{t,k+1}^{\left(i\right)}-{\text{Prox}}_{L_i/\lambda }\left({\mathit{w}}_t^{\left(\text{global}\right)}\right)∥}^2 \\ ={∥{𝒫}_{𝒲}\left[{\mathit{w}}_{t,k}^{\left(i\right)}-\frac{{\eta }_{t,k}^{\left(i\right)}}{B^{\left(i\right)}}\sum _{j\in I_{t,k}^{\left(i\right)}}\left(\nabla \ell \left({\mathit{w}}_{t,k}^{\left(i\right)},z_j^{\left(i\right)}\right)+\lambda \left({\mathit{w}}_{t,k}^{\left(i\right)}-{\mathit{w}}_t^{\left(\text{global}\right)}\right)\right)\right]-{\text{Prox}}_{L_i/\lambda }\left({\mathit{w}}_t^{\left(\text{global}\right)}\right)∥}^2 \\ \le {∥{\mathit{w}}_{t,k}^{\left(i\right)}-\frac{{\eta }_{t,k}^{\left(i\right)}}{B^{\left(i\right)}}\sum _{j\in I_{t,k}^{\left(i\right)}}\left(\nabla \ell \left({\mathit{w}}_{t,k}^{\left(i\right)},z_j^{\left(i\right)}\right)+\lambda \left({\mathit{w}}_{t,k}^{\left(i\right)}-{\mathit{w}}_t^{\left(\text{global}\right)}\right)\right)-{\text{Prox}}_{L_i/\lambda }\left({\mathit{w}}_t^{\left(\text{global}\right)}\right)∥}^2 \\ \left.=\Vert {\mathit{w}}_{t,k}^{\left(i\right)}-{\text{Prox}}_{L_i/\lambda }\left({\mathit{w}}_t^{\left(\text{global}\right)}\right)\right)∥{ }^2+∥\frac{{\eta }_{t,k}^{\left(i\right)}}{B^{\left(i\right)}}\sum _{j\in I_{t,k}^{\left(i\right)}}\left(\nabla \ell \left({\mathit{w}}_{t,k}^{\left(i\right)},z_j^{\left(i\right)}\right)+\lambda \left({\mathit{w}}_{t,k}^{\left(i\right)}-{\mathit{w}}_t^{\left(\text{global}\right)}\right)\right){\Vert }^2-2⟨{\mathit{w}}_{t,k}^{\left(i\right)}-{\text{Prox}}_{L_i/\lambda }\left({\mathit{w}}_t^{\left(\text{global}\right)}\right),\frac{{\eta }_{t,k}^{\left(i\right)}}{B^{\left(i\right)}}\sum _{j\in I_{t,k}^{\left(i\right)}}\left(\nabla \ell \left({\mathit{w}}_{t,k}^{\left(i\right)},z_j^{\left(i\right)}\right)+\lambda \left({\mathit{w}}_{t,k}^{\left(i\right)}-{\mathit{w}}_t^{\left(\text{global}\right)}\right)\right)⟩, \end{gathered}$$

where the inequality is because ${\text{Prox}}_{L_i/\lambda }\left({\mathit{w}}_t^{\left(\text{global}\right)}\right)\in 𝒲$ and ${𝒫}_{𝒲}$ is non-expansive. Now by strong convexity and unbiasedness of the stochastic gradients, we have

$$\begin{gathered} \mathbb{E}\left[\langle w_{t,k}^{(i)}-Pro{x}_{L_i/\lambda }\left(w_t^{(\text{global })}\right),\frac{1}{B^{(i)}}\sum _{j\in I_{t,k}^{(i)}}\left(\nabla \ell \left(w_{t,k}^{(i)},z_j^{(i)}\right)+\lambda \left(w_{t,k}^{(i)}-w_t^{(\text{global })}\right)\right)\rangle |{ℱ}_{t,k},i\in {𝒞}_t\right] \\ \ge \left(L_i\left(w_{t,k}^{(i)},S_i\right)+\frac{\lambda }{2}\Vert w_{t,k}^{(i)}-w^{(\text{global })}{\Vert }^2\right)-\left(L_i\left(Pro{x}_{L_i/\lambda }\left(w_t^{(\text{global })}\right),S_i\right)+\frac{\lambda }{2}\Vert Pro{x}_{L_i/\lambda }\left(w_t^{(\text{global })}\right)-w_t^{(\text{global })}{\Vert }^2\right)+\frac{1}{2}\left({\mu }_i+\frac{\lambda n}{m{n}_i}\right)\Vert w_{t,k}^{(i)}-Pro{x}_{L_i/\lambda }\left(w_t^{(\text{global })}\right){\Vert }^2 \\ \ge (\mu +\lambda )\Vert w_{t,k}^{(i)}-Pro{x}_{L_i/\lambda }\left(w_t^{\text{(global) }}\right){\Vert }^2. \end{gathered}$$

On the other hand, applying Lemma 19 gives

$$\begin{gathered} \mathbb{E}\left[\left.{∥\frac{{\eta }_{t,k}^{\left(i\right)}}{B^{\left(i\right)}}\sum _{j\in I_{t,k}^{\left(i\right)}}\left(\nabla \ell \left({\mathit{w}}_{t,k}^{\left(i\right)},z_j^{\left(i\right)}\right)+\lambda \left({\mathit{w}}_{t,k}^{\left(i\right)}-{\mathit{w}}_t^{\left(\text{global }\right)}\right)\right)∥}^2\right|{ℱ}_{t,k},i\in {𝒞}_t\right] \\ ={\left({\eta }_{t,k}^{\left(i\right)}\right)}^2\cdot \left[\frac{n_i/B^{\left(i\right)}-1}{n_i\left(n_i-1\right)}\sum _{j\in \left[n_i\right]}{∥\nabla \ell \left({\mathit{w}}_{t,k}^{\left(i\right)},z_j^{\left(i\right)}\right)-\stackrel{-}{\nabla \ell \left({\mathit{w}}_{t,k}^{\left(i\right)},z_{\bullet }^{\left(i\right)}\right)}∥}^2\right.\left.+{∥\frac{1}{n_i}\sum _{j\in \left[n_i\right]}\nabla \ell \left({\mathit{w}}_{t,k}^{\left(i\right)},z_j^{\left(i\right)}\right)+\lambda \left({\mathit{w}}_{t,k}^{\left(i\right)}-{\mathit{w}}_t^{\left(\text{global }\right)}\right)∥}^2\right] \\ \le 2{\left({\eta }_{t,k}^{\left(i\right)}\right)}^2{\beta }^2{D}^2\cdot \frac{n_i/B^{\left(i\right)}-1}{\left(n_i-1\right)}+{\left(\beta +\frac{\lambda n}{m{n}_i}\right)}^2{∥{\mathit{w}}_{t,k}^{\left(i\right)}-{\text{Prox}}_{L_i/\lambda }\left({\mathit{w}}_t^{\left(\text{global }\right)}\right)∥}^2, \end{gathered}$$

where in the second line we let $\stackrel{-}{\nabla \ell \left({\mathit{w}}_{t,k}^{\left(i\right)},z_{.}^{\left(i\right)}\right)}≔{\sum }_{j\in \left[n_i\right]}\nabla \ell \left({\mathit{w}}_{t,k}^{\left(i\right)},z_j^{\left(i\right)}\right)/n_i$, and in the last line is by the $\beta$-smoothness of $\ell (\cdot ,z)$. Thus, we get

$$\mathbb{E}\left[{∥{\mathit{w}}_{t,k+1}^{\left(i\right)}-{\text{Prox}}_{L_i/\lambda }\left({\mathit{w}}_t^{\left(\text{global}\right)}\right)∥}^2|{ℱ}_{t,k},i\in {𝒞}_t\right] \le \left[1-2{\eta }_{t,k}^{\left(i\right)}(\mu +\lambda )+{\left({\eta }_{t,k}^{\left(i\right)}\right)}^2(\beta +\lambda {)}^2\right]{∥{\mathit{w}}_{t,k}^{\left(i\right)}-{\text{Prox}}_{L_i/\lambda }\left({\mathit{w}}_t^{\left(\text{global}\right)}\right)∥}^2+2{\left({\eta }_{t,k}^{\left(i\right)}\right)}^2{\beta }^2{D}^2\cdot \frac{\raisebox{1ex}{$n_i$}\!\left/ \!\raisebox{-1ex}{$B^{\left(i\right)}$}\right.-1}{\left(n_i-1\right)}.$$ (40)

We then proceed by induction. Note that if $k+1\le \frac{8{\beta }^2}{{\mu }^2}$, then we have the following trivial bound:

$$\mathbb{E}\left[{∥{\mathit{w}}_{t,k}^{\left(i\right)}-{\text{Prox}}_{L_i/\lambda }\left({\mathit{w}}_t^{\left(\text{global}\right)}\right)∥}^2|{ℱ}_{t,0},i\in {𝒞}_t\right]\le D^2\le \frac{8{\beta }^2{D}^2}{{\mu }^2\left(k+1\right)},$$ (41)

where the first inequality is by ${\mathit{w}}_{t,k}^{\left(i\right)},{\text{Prox}}_{L_i/\lambda }\left({\mathit{w}}_t^{\left(\text{global}\right)}\right)\in 𝒲$ and the second inequality is by our assumption on $k$. Thus, it suffices to show

$$\mathbb{E}\left[{∥{\mathit{w}}_{t,k+1}^{\left(i\right)}-{\text{Prox}}_{L_i/\lambda }\left({\mathit{w}}_t^{\left(\text{global}\right)}\right)∥}^2|{ℱ}_{t,0},i\in {𝒞}_t\right]\le \frac{8{\beta }^2{D}^2}{{\mu }^2(k+2)}$$ (42)

based on the inductive hypothesis (41) and $k+1\ge 8{\beta }^2/{\mu }^2$. By the recursive relationship (40) and taking expectation, we have

$$\mathbb{E}\left[{∥{\mathit{w}}_{t,k+1}^{\left(i\right)}-{\text{Prox}}_{L_i/\lambda }\left({\mathit{w}}_t^{\left(\text{global }\right)}\right)∥}^2|{ℱ}_{t,0},i\in {𝒞}_t\right] \le \left[1-2{\eta }_{t,k}^{\left(i\right)}(\mu +\lambda )+{\left({\eta }_{t,k}^{\left(i\right)}\right)}^2(\beta +\lambda {)}^2\right]\frac{8{\beta }^2{D}^2}{k+1}+2{\left({\eta }_{t,k}^{\left(i\right)}\right)}^2{\beta }^2{D}^2\cdot \frac{n_i/B^{\left(i\right)}-1}{\left(n_i-1\right)}.$$

Hence (42) is satisfied if

$$8{\beta }^2{D}^2\cdot \left[\frac{1}{k+2}-\frac{1}{k+1}+\frac{2{\eta }_{t,k}^{\left(i\right)}}{k+1}(\mu +\lambda )-\frac{{\left({\eta }_{t,k}^{\left(i\right)}\right)}^2}{k+1}(\beta +\lambda {)}^2\right]\ge 2{\left({\eta }_{t,k}^{\left(i\right)}\right)}^2{\beta }^2{D}^2\cdot \frac{n_i/B^{\left(i\right)}-1}{\left(n_i-1\right)}.$$

By our choice of ${\eta }_{t,k}^{\left(i\right)}$, the above display is equivalent to

$$8{\beta }^2{D}^2\cdot \left[-\frac{1}{(k+1)(k+2)}+\frac{2}{(k+1{)}^2}-\frac{1}{(k+1{)}^3}{\left(\frac{\beta +\lambda }{\mu +\lambda }\right)}^2\right]\ge \frac{2{\beta }^2{D}^2}{(\mu +\lambda {)}^2(k+1{)}^2}\cdot \frac{n_i/B^{\left(i\right)}-1}{n_i-1}$$

which is further equivalent to

$$8{\beta }^2{D}^2\cdot \left[-\frac{k+1}{k+2}+2-\frac{1}{k+1}{\left(\frac{\beta +\lambda }{\mu +\lambda }\right)}^2\right]\ge \frac{2{\beta }^2{D}^2}{(\mu +\lambda {)}^2}\cdot \frac{n_i/B^{\left(i\right)}-1}{n_i-1}.$$

We now claim that

$$\frac{1}{k+1}{\left(\frac{\beta +\lambda }{\mu +\lambda }\right)}^2\le \frac{1}{2}.$$

Indeed, since $k+1\ge 8{\beta }^2/{\mu }^2$, (1) if $\lambda \le \beta$, then the left-hand side above is less than $\frac{4{\beta }^2}{{\mu }^2(k+1)}\le \frac{1}{2}$; and (2) if $\lambda \ge \beta$, the left-hand side above is less than $\frac{4}{k+1}\le \frac{{\mu }^2}{2{\beta }^2}\le \frac{1}{2}$. By the above claim, (42) would hold if

$$4{\beta }^2{D}^2\ge \frac{2{\beta }^2{D}^2}{(\mu +\lambda {)}^2}\cdot \frac{n_i/B^{\left(i\right)}-1}{n_i-1}.$$

We finish the proof by noting that the right-hand side above is bounded above by $\frac{2{\beta }^2{D}^2}{{\mu }^2}$.

B.2. Proof of Lemma 15: Convergence of the Outer Loop

By construction we have

$$\begin{gathered} {∥{\mathit{w}}_t^{\left(\text{global }\right)}-{\tilde{\mathit{w}}}^{\text{(global) }}∥}^2={∥\frac{\lambda m{\eta }_t^{\left(\text{global }\right)}}{B^{\left(\text{global }\right)}}\sum _{i\in {𝒞}_t}{p}_i\left({\mathit{w}}_t^{\left(\text{global }\right)}-{\mathit{w}}_{t+1}^{\left(i\right)}\right)∥}^2 \\ ={∥{\mathit{w}}_t^{\left(\text{global }\right)}-{\tilde{\mathit{w}}}^{\left(\text{global }\right)}∥}^2+{∥\frac{\lambda m{\eta }_t^{\left(\text{global }\right)}}{B^{\left(\text{global }\right)}}\sum _{i\in {𝒞}_t}{p}_i\left({\mathit{w}}_t^{\left(\text{global }\right)}-{\mathit{w}}_{t+1}^{\left(i\right)}\right)∥}^2-2⟨{\mathit{w}}_t^{\left(\text{global }\right)}-{\tilde{\mathit{w}}}^{\left(\text{global }\right)},\frac{\lambda m{\eta }_t^{\left(\text{global }\right)}}{B^{\left(\text{global }\right)}}\sum _{i\in {𝒞}_t}{p}_i\left({\mathit{w}}_t^{\left(\text{global }\right)}-{\mathit{w}}_{t+1}^{\left(i\right)}\right)⟩ \\ \le {∥{\mathit{w}}_t^{\left(\text{global }\right)}-{\tilde{\mathit{w}}}^{\text{(global) }}∥}^2-\underset{\text{I}}{\underbrace{2⟨{\mathit{w}}_t^{\left(\text{global }\right)}-{\tilde{\mathit{w}}}^{\text{(global) }},\frac{\lambda m{\eta }_t^{\text{(global) }}}{B^{\left(\text{global }\right)}}\sum _{i\in {𝒞}_t}{p}_i\left({\mathit{w}}_t^{\left(\text{global }\right)}-{\text{Prox}}_{L_i/\lambda }\left({\mathit{w}}_t^{\left(\text{global }\right)}\right)\right)⟩}}+\underset{\text{II }}{\underbrace{2{∥\frac{\lambda m{\eta }_t^{\left(\text{global }\right)}}{B^{\text{(global) }}}\sum _{i\in {𝒞}_t}{p}_i\left({\mathit{w}}_t^{\text{(global) }}-{\text{Prox}}_{L_i/\lambda }\left({\mathit{w}}_t^{\text{(global) }}\right)\right)∥}^2}}+\underset{\text{III }}{\underbrace{2{∥\frac{\lambda m{\eta }_t^{\left(\text{global }\right)}}{B^{\text{(global) }}}\sum _{i\in {𝒞}_t}{p}_i\left({\text{Prox}}_{L_i/\lambda }\left({\mathit{w}}_t^{\left(\text{global }\right)}\right)-{\mathit{w}}_{t+1}^{\left(i\right)}\right)∥}^2}}-\underset{\text{IV }}{\underbrace{2⟨{\mathit{w}}_t^{\left(\text{global }\right)}-{\tilde{\mathit{w}}}^{\left(\text{global }\right)},\frac{\lambda m{\eta }_t^{\left(\text{global }\right)}}{B^{\text{(global) }}}\sum _{i\in {𝒞}_t}{p}_i\left({\text{Prox}}_{L_i/\lambda }\left({\mathit{w}}_t^{\left(\text{global }\right)}\right)\right)-{\mathit{w}}_{t+1}^{\text{(i) }}⟩}}. \end{gathered}$$

We first consider Term I. Note that $\frac{\lambda m}{B^{\text{(global)}}}{\sum }_{i\in {𝒞}_t}{p}_i\left({\mathit{w}}_t^{\text{(global)}}-{\text{Prox}}_{L_i/\lambda }\left({\mathit{w}}_t^{\text{(global)}}\right)\right)$ is an unbiased stochastic gradient of ${\sum }_i{p}_i{F}_i$, which is ${\mu }_F=\lambda \mu /(\lambda +\mu )$-strongly convex. Thus, we have

$$\begin{gathered} \mathbb{E}\left[\text{I}|{ℱ}_{t-1,K_{t-1}}\right] =2{\eta }_t^{\left(\text{global }\right)}⟨{\mathit{w}}_t^{\left(\text{global }\right)}-{\tilde{\mathit{w}}}^{\left(\text{global }\right)},\sum _{i\in \left[m\right]}{p}_i\nabla F_i\left({\mathit{w}}_t^{\left(\text{global }\right)},S_i\right)⟩ \\ \ge 2{\eta }_t^{\left(\text{global }\right)}{\mu }_F{∥{\mathit{w}}_t^{\left(\text{global }\right)}-{\tilde{\mathit{w}}}^{\left(\text{global }\right)}∥}^2. \end{gathered}$$

Now for Term II, we have

$$\begin{gathered} \mathbb{E}\left[\text{ II }|{ℱ}_{t-1,K_{t-1}}\right] \\ \le 2{\left({\eta }_t^{\text{(global) }}\right)}^2\cdot \mathbb{E}\left[\left.{\left(\frac{1}{B^{\left(\text{global }\right)}}\sum _{i\in {𝒞}_t}m{p}_i\right)}^2\right|{ℱ}_{t-1,K_{t-1}}\right]\cdot \underset{i\in \left[m\right]}{\text{max}}{∥\nabla F_i\left({\mathit{w}}_t^{\left(\text{global }\right)},S_i\right)∥}^2 \\ \le 2{\left({\eta }_t^{\left(\text{global }\right)}\right)}^2\cdot \mathbb{E}\left[\left.{\left(\frac{1}{B^{\left(\text{global }\right)}}\sum _{i\in {𝒞}_t}m{p}_i\right)}^2\right|{ℱ}_{t-1,K_{t-1}}\right]\cdot \underset{i\in \left[m\right]}{\text{max}}\cdot \left({\beta }^2{D}^2\wedge 2\lambda \Vert \ell {\Vert }_{\infty }\wedge {\lambda }^2{D}^2\right) \\ \le 2{\left({\eta }_t^{\left(\text{global }\right)}\right)}^2\cdot \left(\frac{1}{m}\sum _{i\in \left[m\right]}{\left(m{p}_i-1\right)}^2+1\right)\cdot \left({\beta }^2{D}^2\wedge 2\lambda \Vert \ell {\Vert }_{\infty }\wedge {\lambda }^2{D}^2\right) \\ =2{\left({\eta }_t^{\left(\text{global }\right)}\right)}^{2}m\Vert \mathbf{p}{\Vert }^2\left({\beta }^2{D}^2\wedge 2\lambda \Vert \ell {\Vert }_{\infty }\wedge {\lambda }^2{D}^2\right), \end{gathered}$$

where the second line is by Lemma 18 and the third line is by Lemma 19. For Term III, we invoke Lemma 14 to get

$$\begin{gathered} \mathbb{E}\left[\text{ III }|{ℱ}_{t-1,K_{t-1}}\right] \le 2{\lambda }^2{\left({\eta }_t^{\left(\text{global }\right)}\right)}^2\cdot \frac{8{\beta }^2{D}^2}{{\mu }^2\left(K_t+1\right)}\cdot \mathbb{E}\left[\left.{\left(\frac{1}{B^{\text{(global) }}}\sum _{i\in {𝒞}_t}m{p}_i\right)}^2\right|{ℱ}_{t-1,K_{t-1}}\right] \\ \le \frac{16{\lambda }^2{\left({\eta }_t^{\left(\text{global }\right)}\right)}^2{\beta }^2{D}^{2}m\Vert \mathbf{p}{\Vert }^2}{{\mu }^2\left(K_t+1\right)}, \end{gathered}$$

where the last line is again by Lemma 19. For Term IV, we invoke Young’s inequality for products to get

$$\begin{gathered} \mathbb{E}\left[-\text{IV}|{ℱ}_{t-1,K_{t-1}}\right] \\ \le \left({\eta }_t^{\left(\text{global}\right)}{\mu }_F\right){∥{\mathit{w}}_t^{\left(\text{global}\right)}-{\tilde{\mathit{w}}}^{\left(\text{global}\right)}∥}^2+{\left({\eta }_t^{\left(\text{global}\right)}{\mu }_F\right)}^{-1}\cdot \mathbb{E}\left[\left.\frac{\text{III}}{2}\right|{ℱ}_{t-1,K_{t-1}}\right] \\ \le \left({\eta }_t^{\left(\text{global}\right)}{\mu }_F\right){∥{\mathit{w}}_t^{\left(\text{global}\right)}-{\tilde{\mathit{w}}}^{\left(\text{global}\right)}∥}^2+{\left({\eta }_t^{\left(\text{global}\right)}{\mu }_F\right)}^{-1}\cdot \frac{8{\lambda }^2{\left({\eta }_t^{\left(\text{global}\right)}\right)}^2{\beta }^2{D}^{2}m\Vert \mathbf{p}{\Vert }^2}{{\mu }^2\left(K_t+1\right)}. \end{gathered}$$

Summarizing the above bounds on the four terms, we arrive at

$$\begin{gathered} \mathbb{E}\left[{∥{\mathit{w}}_{t+1}^{\left(\text{global}\right)}-{\tilde{\mathit{w}}}^{\text{(global) }}∥}^2|{ℱ}_{t-1,K_{t-1}}\right] \\ \le \left(1-{\eta }_t^{\left(\text{global}\right)}{\mu }_F\right){∥{\mathit{w}}_t^{\left(\text{global}\right)}-{\tilde{\mathit{w}}}^{\left(\text{global}\right)}∥}^2+2\underset{\text{V}}{\underbrace{{\left({\eta }_t^{\left(\text{global}\right)}\right)}^{2}m\Vert \mathbf{p}{\Vert }^2\left({\beta }^2{D}^2\wedge 2\lambda \Vert \ell {\Vert }_{\infty }\wedge {\lambda }^2{D}^2\right)}}+\underset{\text{VI}}{\underbrace{\frac{{\lambda }^2{\left({\eta }_t^{\left(\text{global}\right)}\right)}^2{\beta }^2{D}^{2}m\Vert \mathbf{p}{\Vert }^2}{{\mu }^2\left(K_t+1\right)}\cdot \left(16+\frac{8}{{\delta }_t^{\left(\text{global}\right)}{\mu }_F}\right)}}. \end{gathered}$$

We claim that $\text{VI}\le \text{V}$. Indeed, with our choice of ${\eta }_t^{\left(\text{global}\right)}=\frac{2}{{\mu }_F(t+1)}$, with some algebra, one recognizes that this claim is equivalent to

$$\frac{20+4t}{{\mu }^2\left(K_t+1\right)}\le \left(\frac{1}{{\lambda }^2}\wedge \frac{2\Vert \ell {\Vert }_{\infty }}{\lambda {\beta }^2{D}^2}\wedge \frac{1}{{\beta }^2}\right),$$

which is exactly (38). Thus, we have

$$\begin{gathered} \mathbb{E}\left[{∥{\mathit{w}}_{t+1}^{\left(\text{global}\right)}-{\tilde{\mathit{w}}}^{\text{(global)}}∥}^2|{ℱ}_{t-1,K_{t-1}}\right] \\ \le \left(1-{\eta }_t^{\left(\text{global}\right)}{\mu }_F\right){∥{\mathit{w}}_t^{\left(\text{global}\right)}-{\tilde{\mathit{w}}}^{\text{(global)}}∥}^2+3\cdot \text{V} \\ =\left(1-\frac{2}{t+1}\right){∥{\mathit{w}}_t^{\left(\text{global}\right)}-{\tilde{\mathit{w}}}^{\left(\text{global}\right)}∥}^2+\frac{12m\Vert \mathbf{p}{\Vert }^2\left({\beta }^2{D}^2\wedge 2\lambda \Vert \ell {\Vert }_{\infty }\wedge {\lambda }^2{D}^2\right)}{{\mu }_F^2(t+1{)}^2}. \end{gathered}$$ (43)

We then proceed by induction. For the base case, we invoke the strong convexity of ${\sum }_i{p}_i{F}_i$ and Lemma 18 to get

$$\frac{{\mu }_F^2}{4}{∥{\mathit{w}}_0^{\left(\text{global }\right)}-{\tilde{\mathit{w}}}^{\text{(global) }}∥}^2\le {∥\sum _{i\in \left[m\right]}{p}_i\nabla F_i\left({\mathit{w}}_0^{\text{(global) }},S_i\right)∥}^2\le {\beta }^2{D}^2\wedge 2\lambda \Vert \ell {\Vert }_{\infty }\wedge {\lambda }^2{D}^2.$$

Along with the fact that $1={\left({\sum }_{i\in \left[m\right]}{p}_i\right)}^2\le m\Vert \mathbf{p}{\Vert }^2$, we conclude that (15) is true for $t=0$. Now assume (39) hold for any $0\le t\le \tau$. For $t=\tau +1$, using (43) and the inductive hypothesis, we have

$$\begin{gathered} {\mathbb{E}}_{{𝒜}_{\text{FP}}}{∥{\mathit{w}}_{\tau +1}^{\left(\text{global }\right)}-{\tilde{\mathit{w}}}^{\text{(global) }}∥}^2\le \left(1-\frac{2}{\tau +1}\right)\frac{12m\Vert \mathbf{p}{\Vert }^2\left({\beta }^2{D}^2\wedge 2\lambda \Vert \ell {\Vert }_{\infty }\wedge {\lambda }^2{D}^2\right)}{(\tau +1){\mu }_F^2}+\frac{12m\Vert \mathbf{p}{\Vert }^2\left({\beta }^2{D}^2\wedge 2\lambda \Vert \ell {\Vert }_{\infty }\wedge {\lambda }^2{D}^2\right)}{(\tau +1{)}^2{\mu }_F^2} \\ =\left(\frac{1}{\tau +1}-\frac{1}{(\tau +1{)}^2}\right)\cdot \frac{12m\Vert \mathbf{p}{\Vert }^2\left({\beta }^2{D}^2\wedge 2\lambda \Vert \ell {\Vert }_{\infty }\wedge {\lambda }^2{D}^2\right)}{{\mu }_F^2} \\ \le \frac{12m\Vert \mathbf{p}{\Vert }^2\left({\beta }^2{D}^2\wedge 2\lambda \Vert \ell {\Vert }_{\infty }\wedge {\lambda }^2{D}^2\right)}{(\tau +2){\mu }_F^2}, \end{gathered}$$

which is the desired result.

B.3. Auxiliary lemmas

Lemma 17 (Convexity and smoothness ${\mathit{F}}_{\mathit{i}}$) Under Assumption $A\left(b\right)$, each $F_i$ is $\lambda$-smooth and $\frac{\mu \lambda }{\mu +\lambda }$-strongly convex.

Proof The smoothness is a standard fact about the Moreau envelope. The strongly convex constant of $F_i$ follows from Theorem 2.2 of Lemaréchal and Sagastizábal (1997). ■

Lemma 18 (A priori gradient norm bound) Under Assumption A(a, b), for any $w\in 𝒲$ and $i\in \left[m\right]$, we have

$${∥\nabla F_i\left(w,S_i\right)∥}^2\le {\beta }^2{D}^2\wedge 2\lambda \Vert \ell {\Vert }_{\infty }\wedge {\lambda }^2{D}^2.$$

Proof Since $\nabla F_i\left(w,S_i\right)=\lambda \left(w-{\text{Prox}}_{L_i/\lambda }\left(w\right)\right)$, its norm is trivially bounded by $\lambda D$. Now, since ${\text{Prox}}_{L_i/\lambda }\left(w\right)$ achieves a lower objective value than $w$ for the objective function $L_i\left(\cdot ,S_i\right)+\frac{\lambda }{2}\Vert w-\cdot {\Vert }^2$, we have

$$\frac{\lambda }{2}\Vert w-{\text{Prox}}_{L_i/\lambda }\left(w\right){\Vert }^2\le L_i\left(w,S_i\right)-L_i\left({\text{Prox}}_{L_i/\lambda }\left(w\right),S_i\right)\le \Vert \ell {\Vert }_{\infty },$$

and hence ${∥\nabla F_i\left(w,S_i\right)∥}^2\le 2\lambda \Vert \ell {\Vert }_{\infty }$. Finally, by the first-order condition, we have

$$\nabla L_i\left({\text{Prox}}_{L_i/\lambda }\left(w\right),S_i\right)+\lambda \left({\text{Prox}}_{L_i/\lambda }\left(w\right)-w\right)=0.$$

Hence, we get $∥\nabla F_i\left(w,S_i\right)∥=∥\nabla L_i\left({\text{Prox}}_{L_i/\lambda }\left(w\right),S_i\right)∥\le \beta D$. ■

Lemma 19 (Variance of minibatch sampling) Let $ℬ\subseteq \left[n\right]$ be a randomly sampled batch with batch size $B$ and let ${\left\{x_i\right\}}_{i=1}^n\subseteq {\mathbb{R}}^d$ be an arbitrary set of vectors, then

$${\mathbb{E}}_{ℬ}{∥\frac{1}{B}\sum _{i\in ℬ}{x}_i∥}^2=\frac{n/B-1}{n(n-1)}\sum _{i\in \left[n\right]}{∥x_i-\bar{x}∥}^2+\Vert \bar{x}{\Vert }^2\le \frac{1}{n}\sum _{i\in \left[n\right]}{∥x_i-\bar{x}∥}^2+\Vert \bar{x}{\Vert }^2,$$

where $\bar{x}≔{\sum }_{i\in \left[n\right]}{x}_i/n$.

Proof Since ${\mathbb{E}}_{ℬ}{\sum }_{i\in ℬ}{x}_i/B=\bar{x}$, we have

$$\begin{gathered} {\mathbb{E}}_{ℬ}{∥\frac{1}{B}\sum _{i\in ℬ}{x}_i∥}^2={\mathbb{E}}_{ℬ}{∥\frac{1}{B}\sum _{i\in ℬ}{x}_i-\bar{x}∥}^2+\Vert \bar{x}{\Vert }^2 \\ =\frac{1}{B^2}\left(\sum _{i\in \left[n\right]}1\{i\in ℬ\}{∥x_i-\bar{x}∥}^2+2\sum _{i<j}1\{i,j\in ℬ\}⟨x_i-\bar{x},x_j-\bar{x}⟩\right)+\Vert \bar{x}{\Vert }^2 \\ =\frac{1}{B^2}\left(\frac{B}{n}\sum _{i\in \left[n\right]}{∥x_i-\bar{x}∥}^2+\frac{2B(B-1)}{n(n-1)}\sum _{i<j}⟨x_i-\bar{x},x_j-\bar{x}⟩\right)+\Vert \bar{x}{\Vert }^2, \end{gathered}$$

where the last line is by ${\mathbb{P}}_{ℬ}(i\in ℬ)=B/n$ and ${\mathbb{P}}_{ℬ}(i,j\in ℬ)=B(B-1)n^{-1}(n-1{)}^{-1}$ for any $i\ne j$. Now, since ${\sum }_{i\in \left[n\right]}{∥x_i-\bar{x}∥}^2+2{\sum }_{i<j}⟨x_i-\bar{x},x_j-\bar{x}⟩=0$, we arrive at

$$\begin{gathered} {\mathbb{E}}_{ℬ}{∥\frac{1}{B}\sum _{i\in ℬ}{x}_i∥}^2 \\ =\frac{1}{B^2}\left(\frac{B}{n}-\frac{B\left(B-1\right)}{n\left(n-1\right)}\right)\sum _i{∥x_i-\bar{x}∥}^2+\Vert x{\Vert }^2 =\frac{n/B-1}{n(n-1)}\sum _{i\in \left[n\right]}{∥x_i-\bar{x}∥}^2+\Vert \bar{x}{\Vert }^2, \end{gathered}$$

which is the desired result. ■

C. Proofs of Upper Bounds

C.1. Proof of Theorem 6

In this proof, we let ${\widehat{\mathit{w}}}^{\text{(global)}}$ be the global minimizer of (8) and we write ${\mathit{w}}_{\text{avg}}^{\text{(global,p)}}\equiv {\mathit{w}}_{\text{p}}^{\text{(global)}}$ when there is no ambiguity.

Proof [Proof of (9)] We have

$$\begin{gathered} 0=- \sum _{i\in \left[m\right]}{p}_i{L}_i\left({\widehat{\mathit{w}}}^{\left(\text{global }\right)}\left(\mathit{S}\right),S_i\right)+\sum _{i\in \left[m\right]}{p}_i{L}_i\left({\widehat{\mathit{w}}}^{\left(\text{global }\right)}\left(\mathit{S}\right),S_i\right) \\ \le - \sum _{i\in \left[m\right]}{p}_i{L}_i\left({\widehat{\mathit{w}}}^{\left(\text{global }\right)}\left(\mathit{S}\right),S_i\right)+\sum _{i\in \left[m\right]}{p}_i{L}_i\left({\mathit{w}}_{\star }^{\left(1\right)},S_i\right) \\ =- \sum _{i\in \left[m\right]}\frac{p_i}{n_i}\sum _{j\in \left[n_i\right]}\left(\ell \left({\widehat{\mathit{w}}}^{\left(\text{global }\right)}\left({\mathit{S}}^{\setminus (i,j)}\right),z_j^{\left(i\right)}\right)-\ell \left({\mathit{w}}_{\star }^{\left(1\right)},z_j^{\left(i\right)}\right)\right)+\sum _{i\in \left[m\right]}\frac{p_i}{n_i}\sum _{j\in \left[n_i\right]}\left(\ell \left({\widehat{\mathit{w}}}^{\left(\text{global }\right)}\left({\mathit{S}}^{\setminus (i,j)}\right),z_j^{\left(i\right)}\right)-\ell \left({\widehat{\mathit{w}}}^{\left(\text{global }\right)}\left(\mathit{S}\right),z_j^{\left(i\right)}\right)\right), \end{gathered}$$

where ${\mathit{S}}^{\setminus (i,j)}$ stands for the dataset formed by replacing $z_j^{\left(i\right)}$ by another $z_{i,j}^{\prime }~{𝒟}_i$, which is independent of everything else. Taking expectation in both sides, we get

$$\begin{gathered} 0\le -\sum _{i\in \left[m\right]}{p}_i\cdot {\mathbb{E}}_{\mathit{S},Z_i~{𝒟}_i}\left[\ell \left({\widehat{\mathit{w}}}^{\left(\text{global }\right)}\left(\mathit{S}\right),Z_i\right)-\ell \left({\mathit{w}}_{\star }^{\left(1\right)},Z_i\right)\right]+\sum _{i\in \left[m\right]}\frac{p_i}{n_i}\sum _{j\in \left[n_i\right]}{\mathbb{E}}_{\mathit{S},z_{i,j}^{\prime }}\left[\ell \left({\widehat{\mathit{w}}}^{\left(\text{global }\right)}\left({\mathit{S}}^{\setminus (i,j)}\right),z_j^{\left(i\right)}\right)-\ell \left({\widehat{\mathit{w}}}^{\left(\text{global }\right)}\left(\mathit{S}\right),z_j^{\left(i\right)}\right)\right] \\ =- \sum _{i\in \left[m\right]}{p}_i\cdot {\mathbb{E}}_{\mathit{S},Z_i~{𝒟}_i}\left[\ell \left({\widehat{\mathit{w}}}^{\left(\text{global }\right)}\left(\mathit{S}\right),Z_i\right)-\ell \left({\mathit{w}}_{\star }^{\left(i\right)},Z_i\right)\right]-\sum _{i\in \left[m\right]}{p}_i\cdot {\mathbb{E}}_{Z_i~{𝒟}_i}\left[\ell \left({\mathit{w}}_{\star }^{\left(i\right)},Z_i\right)-\ell \left({\mathit{w}}_{\star }^{\left(1\right)},Z_i\right)\right]+\sum _{i\in \left[m\right]}\frac{p_i}{n_i}\sum _{j\in \left[n_i\right]}{\mathbb{E}}_{\mathit{S},z_{i,j}^{\prime }}\left[\ell \left({\widehat{\mathit{w}}}^{\left(\text{global }\right)}\left({\mathit{S}}^{\setminus (i,j)}\right),z_j^{\left(i\right)}\right)-\ell \left({\widehat{\mathit{w}}}^{\left(\text{global }\right)}\left(\mathit{S}\right),z_j^{\left(i\right)}\right)\right]. \end{gathered}$$

Noting that ${\mathit{w}}_{\star }^{\left(i\right)}$ is the argmin of ${\mathbb{E}}_{Z_i~{𝒟}_i}\left[\ell \left(\cdot ,Z_i\right)\right]$ and invoking the $\beta$-smoothness assumption, we get

$$\begin{gathered} {\mathbb{E}}_{\mathit{S}}\left[{\text{AER}}_{\mathbf{p}}\left({\widehat{\mathit{w}}}^{\left(\text{global }\right)}\right)\right] \\ \le \beta \sum _{i\in \left[m\right]}{p}_i{∥{\mathit{w}}_{\star }^{\left(1\right)}-{\mathit{w}}_{\star }^{\left(i\right)}∥}^2+\sum _{i\in \left[m\right]}\frac{p_i}{n_i}\sum _{j\in \left[n_i\right]}{\mathbb{E}}_{\mathit{S},z_{i,j}^{\prime }}\left[\ell \left({\widehat{\mathit{w}}}^{\left(\text{global }\right)}\left({\mathit{S}}^{\setminus (i,j)}\right),z_j^{\left(i\right)}\right)-\ell \left({\widehat{\mathit{w}}}^{\left(\text{global }\right)}\left(\mathit{S}\right),z_j^{\left(i\right)}\right)\right] \\ \le 2\beta {∥{\mathit{w}}_{\star }^{\left(1\right)}-{\mathit{w}}_{\mathbf{p}}^{\left(\text{global }\right)}∥}^2+2\beta \sum _{i\in \left[m\right]}{p}_i∥{\mathit{w}}_{\star }^{\left(i\right)}-{\mathit{w}}_{\mathbf{p}}^{\left(\text{global }\right)}∥+\sum _{i\in \left[m\right]}\frac{p_i}{n_i}\sum _{j\in \left[n_i\right]}{\mathbb{E}}_{\mathit{S},z_{i,j}^{\prime }}\left[\ell \left({\widehat{\mathit{w}}}^{\left(\text{global }\right)}\left({\mathit{S}}^{\setminus (i,j)}\right),z_j^{\left(i\right)}\right)-\ell \left({\widehat{\mathit{w}}}^{\left(\text{global }\right)}\left(\mathit{S}\right),z_j^{\left(i\right)}\right)\right]. \end{gathered}$$

Taking a weighted average, we arrive at

$${\mathbb{E}}_{\mathit{S}}\left[{\text{AER}}_{\mathbf{p}}\left({\widehat{\mathit{w}}}^{\left(\text{global}\right)}\right)\right] \le 4\beta R^2+\sum _{i\in \left[m\right]}\frac{p_i}{n_i}\sum _{j\in \left[n_i\right]}{\mathbb{E}}_{\mathit{S},z_{i,j}^{\prime }}\left[\ell \left({\widehat{\mathit{w}}}^{\left(\text{global }\right)}\left({\mathit{S}}^{\setminus (i,j)}\right),z_j^{\left(i\right)}\right)-\ell \left({\widehat{\mathit{w}}}^{\left(\text{global }\right)}\left(\mathit{S}\right),z_j^{\left(i\right)}\right)\right]$$ (44)

To bound the second term in the right-hand side above, we bound the federated stability of ${\widehat{\mathit{w}}}^{\text{(global)}}$. Without loss of generality we consider the first client. By $\mu$-strongly convexity of $L_1$, for any $j_1\in \left[n_1\right]$ we have

$$\begin{gathered} \frac{\mu }{2}{∥{\widehat{\mathit{w}}}^{\text{(global) }}\left(\mathit{S}\right)-{\widehat{\mathit{w}}}^{\text{(global) }}\left({\mathit{S}}^{\setminus \left(1,j_1\right)}\right)∥}^2 \\ \le \sum _{i\in \left[m\right]}{p}_i\left(L_i\left({\widehat{\mathit{w}}}^{\text{(global) }}\left({\mathit{S}}^{\setminus \left(1,j_1\right)}\right),S_i\right)-L_i\left({\widehat{\mathit{w}}}^{\text{(global) }}\left(\mathit{S}\right),S_i\right)\right) \\ =\left(\sum _{i\ne 1}{p}_i{L}_i\left({\widehat{\mathit{w}}}^{\left(\text{global }\right)}\left({\mathit{S}}^{\setminus \left(1,j_1\right)}\right),S_i\right)+p_1{L}_1\left({\widehat{\mathit{w}}}^{\text{(global) }}\left({\mathit{S}}^{\setminus \left(1,j_1\right)}\right),S_1^{\setminus j_1}\right)\right)-\left(\sum _{i\ne 1}{p}_i{L}_i\left({\widehat{\mathit{w}}}^{\text{(global) }}\left(\mathit{S}\right),S_i\right)+p_1{L}_1\left({\widehat{\mathit{w}}}^{\text{(global) }}\left(\mathit{S}\right),S_1^{\setminus j_1}\right)\right)+p_1\left(L_1\left({\widehat{\mathit{w}}}^{\left(\text{global }\right)}\left({\mathit{S}}^{\setminus \left(1,j_1\right)}\right),S_1\right)-L_1\left({\widehat{\mathit{w}}}^{\text{(global) }}\left({\mathit{S}}^{\setminus \left(1,j_1\right)}\right),S_1^{\setminus j_1}\right)\right)+p_1\left(L_1\left({\widehat{\mathit{w}}}^{\text{(global) }}\left(\mathit{S}\right),S_1^{\setminus j_1}\right)-L_1\left({\widehat{\mathit{w}}}^{\text{(global) }}\left(\mathit{S}\right),S_1\right)\right) \\ \le p_1\left(L_1\left({\widehat{\mathit{w}}}^{\text{(global)}}\left({\mathit{S}}^{\setminus \left(1,j_1\right)}\right),S_1\right)-L_1\left({\widehat{\mathit{w}}}^{\text{(global)}}\left({\mathit{S}}^{\setminus \left(1,j_1\right)}\right),S_1^{\setminus j_1}\right)\right)+p_1\left(L_1\left({\widehat{\mathit{w}}}^{\text{(global)}}\left(\mathit{S}\right),S_1^{\setminus j_1}\right)-L_1\left({\widehat{\mathit{w}}}^{\text{(global)}}\left(\mathit{S}\right),S_1\right)\right) \\ =\frac{p_1}{n_1}\left(\ell \left({\widehat{\mathit{w}}}^{\text{(global)}}\left({\mathit{S}}^{\setminus \left(1,j_1\right)}\right),z_{j_1}^{\left(1\right)}\right)-\ell \left({\widehat{\mathit{w}}}^{\text{(global)}}\left(\mathit{S}\right),z_{j_1}^{\left(1\right)}\right)\right)+\frac{p_1}{n_1}\left(\ell \left({\widehat{\mathit{w}}}^{\text{(global)}}\left(\mathit{S}\right),z_{1,j_1}^{\prime }\right)-\ell \left({\widehat{\mathit{w}}}^{\text{(global)}}\left({\mathit{S}}^{\setminus \left(1,j_1\right)}\right),z_{1,j_1}^{\prime }\right)\right), \end{gathered}$$ (45)

where the second inequality is because ${\widehat{\mathit{w}}}^{\left(\text{global}\right)}\left({\mathit{S}}^{\setminus \left(1,j_1\right)}\right)$ minimizes $L_1\left(\cdot ,S_1^{\setminus j_1}\right)+{\sum }_{i\ne 1}{n}_i{L}_i\left(\cdot ,S_i\right)$. By an identical argument as in the proof of Lemma 26, we have

$$\begin{gathered} \ell \left({\widehat{\mathit{w}}}^{\text{(global)}}\left({\mathit{S}}^{\setminus \left(1,j_1\right)}\right),z_{j_1}^{\left(1\right)}\right)-\ell \left({\widehat{\mathit{w}}}^{\text{(global)}}\left(\mathit{S}\right),z_{j_1}^{\left(1\right)}\right) \\ \le \sqrt{2\beta \Vert \ell {\Vert }_{\infty }}\cdot ∥{\widehat{\mathit{w}}}^{\text{(global)}}\left(\mathit{S}\right)-{\widehat{\mathit{w}}}^{\text{(global)}}\left({\mathit{S}}^{\setminus \left(1,j_1\right)}\right)∥+\frac{\beta }{2}{∥{\widehat{\mathit{w}}}^{\text{(global)}}\left(\mathit{S}\right)-{\widehat{\mathit{w}}}^{\text{(global)}}\left({\mathit{S}}^{\setminus \left(1,j_1\right)}\right)∥}^2 \end{gathered}$$ (46)

The same bound also holds for $\ell \left({\widehat{\mathit{w}}}^{\text{(global)}}\left(\mathit{S}\right),z_{1,j_1}^{\prime }\right)-\ell \left({\widehat{\mathit{w}}}^{\left(\text{global}\right)}\left({\mathit{S}}^{\setminus \left(1,j_1\right)}\right),z_{1,j_1}^{\prime }\right)$. Plugging these two bounds to (45) and rearranging terms, we get

$$\left(\frac{\mu }{2}-\frac{\beta p_1}{n_1}\right)∥{\widehat{\mathit{w}}}^{\left(\text{global }\right)}\left(\mathit{S}\right)-{\widehat{\mathit{w}}}^{\left(\text{global }\right)}\left({\mathit{S}}^{\setminus \left(1,j_1\right)}\right)∥\le \frac{2\sqrt{2\beta \Vert \ell {\Vert }_{\infty }}\cdot p_1}{n_1}.$$

Since $n_1\ge 4\beta p_1/\mu$, we in fact have

$$\frac{\mu }{4}∥{\widehat{\mathit{w}}}^{\text{(global) }}\left(\mathit{S}\right)-{\widehat{\mathit{w}}}^{\text{(global) }}\left({\mathit{S}}^{\setminus \left(1,j_1\right)}\right)∥\le \frac{2\sqrt{2\beta \Vert \ell {\Vert }_{\infty }}\cdot p_1}{n_1}.$$

Plugging the above display back to (46), we arrive at

$$\ell \left({\widehat{\mathit{w}}}^{\text{(global) }}\left({\mathit{S}}^{\setminus \left(1,j_1\right)}\right),z_{j_1}^{\left(1\right)}\right)-\ell \left({\widehat{\mathit{w}}}^{\text{(global) }}\left(\mathit{S}\right),z_{j_1}^{\left(1\right)}\right)\le \frac{16\beta \Vert \ell {\Vert }_{\infty }{p}_1}{\mu n_1}\left(1+\frac{4\beta p_1}{\mu n_1}\right)\le \frac{32\beta \Vert \ell {\Vert }_{\infty }{p}_1}{\mu n_1},$$

where the last inequality is again by $n_1\ge 4\beta p_1/\mu$. The desired result follows by plugging the above inequality back to (44). ■

Proof [Proof of (10)] Without loss of generality we consider the first client. Since ${\mathit{w}}_{\star }^{\left(1\right)}$ is the minimizer of ${\mathbb{E}}_{Z_1~{𝒟}_1}\ell \left(\cdot ,Z_1\right)$, by $\beta$-smoothness we have

$$\begin{gathered} {\mathbb{E}}_{Z_1~{𝒟}_1}\left[\ell \left({\widehat{\mathit{w}}}^{\left(\text{global}\right)},Z_1\right)-\ell \left({\mathit{w}}_{\star }^{\left(1\right)},Z_1\right)\right] \lesssim \beta \cdot {\mathbb{E}}_{Z_1~{𝒟}_1}{∥{\widehat{\mathit{w}}}^{\left(\text{global}\right)}-{\mathit{w}}_{\star }^{\left(1\right)}∥}^2 \\ \lesssim \beta \cdot {\mathbb{E}}_{Z_1~{𝒟}_1}{∥{\widehat{\mathit{w}}}^{\left(\text{global}\right)}-{\mathit{w}}_{\mathbf{p}}^{\left(\text{global}\right)}∥}^2+\beta R^2, \end{gathered}$$ (47)

where the last inequality is by Part (b) of Assumption B. By optimality of ${\widehat{\mathit{w}}}^{\text{(global)}}$ and the strong convexity of $L_i\prime$s, we have

$$⟨\sum _{i\in \left[m\right]}{p}_i\nabla L_i\left({\mathit{w}}_{\mathbf{p}}^{\left(\text{global }\right)},S_i\right),{\widehat{\mathit{w}}}^{\left(\text{global }\right)}-{\mathit{w}}_{\mathbf{p}}^{\left(\text{global }\right)}⟩+\frac{\mu }{2}{∥{\widehat{\mathit{w}}}^{\left(\text{global }\right)}-{\mathit{w}}_{\mathbf{p}}^{\left(\text{global }\right)}∥}^2\le 0.$$

If ${\widehat{\mathit{w}}}^{\text{(global)}}-{\mathit{w}}_{\mathbf{p}}^{\text{(global)}}=0$ then we are done. Otherwise, the above display gives

$$\begin{gathered} ∥{\widehat{\mathit{w}}}^{\text{(global) }}-{\mathit{w}}_{\mathbf{p}}^{\left(\text{global }\right)}∥ \\ \le \frac{2}{\mu }∥\sum _{i\in \left[m\right]}{p}_i\nabla L_i\left({\mathit{w}}_{\mathbf{p}}^{\left(\text{global }\right)},S_i\right)∥ \\ \le \frac{2}{\mu }\left(∥\sum _{i\in \left[m\right]}{p}_i\nabla L_i\left({\mathit{w}}_{\star }^{\left(i\right)},S_i\right)∥+∥\sum _{i\in \left[m\right]}{p}_i\left(\nabla L_i\left({\mathit{w}}_{\mathbf{p}}^{\left(\text{global }\right)},S_i\right)-\nabla L_i\left({\mathit{w}}_{\star }^{\left(i\right)},S_i\right)\right)∥\right) \\ \le \frac{2}{\mu }\left(∥\sum _{i\in \left[m\right]}{p}_i\nabla L_i\left({\mathit{w}}_{\star }^{\left(i\right)},S_i\right)∥+\beta \sum _{i\in \left[m\right]}{p}_i∥{\mathit{w}}_{\mathbf{p}}^{\left(\text{global }\right)}-{\mathit{w}}_{\star }^{\left(i\right)}∥\right) \\ \le \frac{2}{\mu }\left(∥\sum _{i\in \left[m\right]}{p}_i\nabla L_i\left({\mathit{w}}_{\star }^{\left(i\right)},S_i\right)∥+\beta R\right). \end{gathered}$$

Thus, we get

$${∥{\widehat{\mathit{w}}}^{\left(\text{global }\right)}-{\mathit{w}}_{\mathbf{p}}^{\left(\text{global }\right)}∥}^2\le \frac{8}{{\mu }^2}\left({∥\sum _{i\in \left[m\right]}{p}_i\nabla L_i\left({\mathit{w}}_{\star }^{\left(i\right)},S_i\right)∥}^2+{\beta }^2{R}^2\right).$$

Taking expectation with respect to the sample $\mathit{S}$ at both sides, we have

$$\begin{gathered} {\mathbb{E}}_{\mathit{S}}{∥{\widehat{\mathit{w}}}^{\left(\text{global }\right)}-{\mathit{w}}_{\mathbf{p}}^{\left(\text{global }\right)}∥}^2 \lesssim \frac{1}{{\mu }^2}{\mathbb{E}}_{\mathit{S}}{∥\sum _{i\in \left[m\right]}{p}_i\left(\nabla L_i\left({\mathit{w}}_{\star }^{\left(i\right)},S_i\right)-{\mathbb{E}}_{\mathit{S}}\left[\nabla L_i\left({\mathit{w}}_{\star }^{\left(i\right)},S_i\right)\right]\right)∥}^2+\frac{{\beta }^2{R}^2}{{\mu }^2} \\ \le \frac{1}{{\mu }^2}\cdot \sum _{i\in \left[m\right]}\frac{p_i^2{\sigma }^2}{n_i}+\frac{{\beta }^2{R}^2}{{\mu }^2}. \end{gathered}$$

Plugging the above inequality to (47) gives the desired result. ■

C.2. Proof of Proposition 10

Proof [Proof of (21)] By the definitions of the AER and ${ℰ}_{\text{OPT}}$, we have

$${\text{AER}}_{\mathbf{p}}={ℰ}_{\text{OPT}} +\frac{\lambda }{2}\sum _{i\in \left[m\right]}{p}_i\left({∥{\tilde{\mathit{w}}}^{\left(\text{global }\right)}\left(\mathit{S}\right)-{\tilde{\mathit{w}}}^{\left(i\right)}\left(\mathit{S}\right)∥}^2-{∥{\widehat{\mathit{w}}}^{\left(\text{global }\right)}\left(\mathit{S}\right)-{\widehat{\mathit{w}}}^{\left(i\right)}\left(\mathit{S}\right)∥}^2\right) +\sum _{i\in \left[m\right]}{p}_i\left({\mathbb{E}}_{Z_i~{𝒟}_i}\left[\ell \left({\widehat{\mathit{w}}}^{\left(i\right)}\left(\mathit{S}\right),Z_i\right)\right]-L_i\left({\widehat{\mathit{w}}}^{\left(i\right)}\left(\mathit{S}\right),S_i\right)\right) +\sum _{i\in \left[m\right]}{p}_i\left(L_i\left({\tilde{\mathit{w}}}^{\left(i\right)}\left(\mathit{S}\right),S_i\right)-{\mathbb{E}}_{Z_i~{𝒟}_i}\left[\ell \left({\mathit{w}}_{\star }^{\left(i\right)},Z_i\right)\right]\right).$$

By the basic inequality (23), we can bound the AER by

$${\text{AER}}_{\text{p}}\le {ℰ}_{\text{OPT}}+\frac{\lambda }{2}\sum _{i\in \left[m\right]}{p}_i{∥{\mathit{w}}_{\mathbf{p}}^{\left(\text{global }\right)}-{\mathit{w}}_{\star }^{\left(i\right)}∥}^2+\sum _{i\in \left[m\right]}{p}_i\left({\mathbb{E}}_{Z_i~{𝒟}_i}\left[\ell \left({\widehat{\mathit{w}}}^{\left(i\right)}\left(\mathit{S}\right),Z_i\right)\right]-L_i\left({\widehat{\mathit{w}}}^{\left(i\right)}\left(\mathit{S}\right),S_i\right)\right)+\sum _{i\in \left[m\right]}{p}_i\left(L_i\left({\mathit{w}}_{\star }^{\left(i\right)},S_i\right)-{\mathbb{E}}_{Z_i~{𝒟}_i}\left[\ell \left({\mathit{w}}_{\star }^{\left(i\right)},Z_i\right)\right]\right).$$

Now, invoking federated stability, we can further bound the AER by

$${\text{AER}}_{\mathbf{p}}\le {ℰ}_{\text{OPT}} +\frac{\lambda }{2}\sum _{i\in \left[m\right]}{p}_i{∥{\mathit{w}}_{\mathbf{p}}^{\left(\text{global }\right)}-{\mathit{w}}_{\star }^{\left(i\right)}∥}^2+2\sum _{i\in \left[m\right]}{p}_i{\gamma }_i +\sum _{i\in \left[m\right]}{p}_i\cdot \frac{1}{n_i}\sum _{j\in \left[n_i\right]}{\mathbb{E}}_{z_{i,j}^{\prime }~{𝒟}_i}\left[{\mathbb{E}}_{Z_i~{𝒟}_i}\left[\ell \left({\widehat{\mathit{w}}}^{\left(i\right)}\left({\mathit{S}}^{\setminus (i,j)}\right),Z_i\right)\right]-\ell \left({\widehat{\mathit{w}}}^{\left(i\right)}\left({\mathit{S}}^{\setminus (i,j)}\right),z_j^{\left(i\right)}\right)\right] +\sum _{i\in \left[m\right]}{p}_i\left(L_i\left({\mathit{w}}_{\star }^{\left(i\right)},S_i\right)-{\mathbb{E}}_{Z_i~{𝒟}_i}\left[\ell \left({\mathit{w}}_{\star }^{\left(i\right)},Z_i\right)\right]\right),$$

where ${\mathit{S}}^{\setminus (i,j)}$ is the dataset formed by replacing $z_j^{\left(i\right)}$ with a new sample $z_{i,j}^{\prime }$, and here we are choosing $z_{i,j}^{\prime }$ to be an independent sample from ${𝒟}_i$. Note that the last two terms of the above display have mean zero under the randomness of the algorithm $𝒜$, the dataset $\mathit{S}$, and $\left\{z_{i,j}^{\prime }:i\in \left[m\right],j\in \left[n_i\right]\right\}$. Thus, the desired result follows by taking expectation in both sides. ■

Proof [Proof of (22)] Without loss of generality we consider the first client. By definitions of ${\text{IER}}_1$ and ${ℰ}_{\text{OPT}}$, we have

$$p_1\cdot {\text{IER}}_1={ℰ}_{\text{OPT}} +\sum _{i\in \left[m\right]}{p}_i\left(L_i\left({\tilde{\mathit{w}}}^{\left(i\right)}\left(\mathit{S}\right),S_i\right)+\frac{\lambda }{2}{∥{\tilde{\mathit{w}}}^{\left(\text{global }\right)}\left(\mathit{S}\right)-{\tilde{\mathit{w}}}^{\left(i\right)}\left(\mathit{S}\right)∥}^2\right) -\sum _{i\in \left[m\right]}{p}_i\left(L_i\left({\widehat{\mathit{w}}}^{\left(i\right)}\left(\mathit{S}\right),S_i\right)+\frac{\lambda }{2}{∥{\widehat{\mathit{w}}}^{\left(\text{global }\right)}\left(\mathit{S}\right)-{\widehat{\mathit{w}}}^{\left(i\right)}\left(\mathit{S}\right)∥}^2\right) +p_1{\mathbb{E}}_{Z_1~{𝒟}_1}\left[\ell \left({\widehat{\mathit{w}}}^{\left(1\right)}\left(\mathit{S}\right),Z_1\right)-\ell \left({\mathit{w}}_{\star }^{\left(1\right)},Z_1\right)\right].$$

Invoking the basic inequality (24), with some algebra, we arrive at

$$p_1\cdot {\text{IER}}_1 \le {ℰ}_{\text{OPT}}+\frac{p_1\lambda }{2}{∥{\widehat{\mathit{w}}}^{\left(\text{global }\right)}\left(\mathit{S}\right)-{\mathit{w}}_{\star }^{\left(1\right)}∥}^2+p_1\left({\mathbb{E}}_{Z_1~{𝒟}_1}\left[\ell \left({\widehat{\mathit{w}}}^{\left(1\right)}\left(\mathit{S}\right),Z_1\right)\right]-L_1\left({\widehat{\mathit{w}}}^{\left(1\right)}\left(\mathit{S}\right),S_1\right)\right)+p_1\left(L_1\left({\mathit{w}}_{\star }^{\left(1\right)},S_1\right)-{\mathbb{E}}_{Z_1~{𝒟}_1}\left[\ell \left({\mathit{w}}_{\star }^{\left(1\right)},Z_1\right)\right]\right).$$

Now, invoking federated stability for the first client, we can bound its IER by

$$p_1\cdot {\text{IER}}_1\le {ℰ}_{\text{OPT}} +\frac{p_1\lambda }{2}{∥{\widehat{\mathit{w}}}^{\left(\text{global }\right)}\left(\mathit{S}\right)-{\mathit{w}}_{\star }^{\left(1\right)}∥}^2+2p_1{\gamma }_1 +\frac{p_1}{n_1}\sum _{j\in \left[n_1\right]}{\mathbb{E}}_{z_{1,j}^{\prime }~{𝒟}_1}\left[{\mathbb{E}}_{Z_1~{𝒟}_1}\left[\ell \left({\widehat{\mathit{w}}}^{\left(1\right)}\left({\mathit{S}}^{\setminus \left(1,j\right)}\right),Z_1\right)\right]-\ell \left({\widehat{\mathit{w}}}^{\left(1\right)}\left({\mathit{S}}^{\setminus \left(1,j\right)}\right),z_j^{\left(1\right)}\right)\right]+p_1\left(L_1\left({\mathit{w}}_{\star }^{\left(1\right)},S_1\right)-{\mathbb{E}}_{Z_1~{𝒟}_1}\left[\ell \left({\mathit{w}}_{\star }^{\left(1\right)},Z_1\right)\right]\right),$$

where we recall that ${\mathit{S}}^{\setminus (1,j)}$ is the dataset formed by replacing $z_j^{\left(1\right)}$ with a new sample $z_{1,j}^{\prime }$, and here we are choosing $z_{1,j}^{\prime }$ to be an independent sample from ${𝒟}_1$. We finish the proof by taking the expectation with respect to $𝒜,\mathit{S},\left\{z_{1,j}^{\prime }:j\in \left[n_1\right]\right\}$ at both sides. ■

C.3. Proof of Theorem 11

In this proof, we let $𝒜=\left({\widehat{\mathit{w}}}^{\text{(global)}},\left\{{\widehat{\mathit{w}}}^{\left(i\right)}\right\}\right.$ be a generic algorithm that tries to minimize (20). For notiontional simplicity, we use $a_n{\lesssim }_{\beta }{b}_n$ (resp. $a_n{\gtrsim }_{\beta }{b}_n$) to denote that $a_n\le C_{\beta }{b}_n$ (resp. $a_n\ge C_{\beta }{b}_n$) for large $n$, where $C_{\beta }$ has explicit dependence on a parameter $\beta$.

Recall that $\left({\tilde{\mathit{w}}}^{\text{(global)}},\left\{{\tilde{\mathit{w}}}^{\left(i\right)}\right\}\right)$ is the global minimizer of (20), and recall the notations in (35)–(37). We start by bounding the federated stability of approximate minimizers of (20). We need the following definition.

Definition 20 (Approximate minimizers) We say an algorithm $𝒜=\left({\widehat{\mathit{w}}}^{\left(\text{global}\right)},{\left\{{\widehat{\mathit{w}}}^{\left(i\right)}\right\}}_1^m\right)$ produces an $\left({\epsilon }^{\left(\text{global}\right)},{\left\{{\epsilon }^{\left(i\right)}\right\}}_1^m\right)$-minimizer of the objective function (20) on the dataset $\mathit{S}$ if the following two conditions hold:

1. there exist a positive constant ${\epsilon }^{\text{(global)}}$ such that $∥{\widehat{\mathit{w}}}^{\text{(global)}}-{\tilde{\mathit{w}}}^{\text{(global)}}∥\le {\epsilon }^{\text{(global)}}$;

2. for any $i\in \left[m\right]$, there exist a positive constant ${\epsilon }^{\left(i\right)}$ such that $∥{\widehat{\mathit{w}}}^{\left(i\right)}-{\text{Prox}}_{L_i/\lambda }\left({\widehat{\mathit{w}}}^{\left(\text{global}\right)}\right)∥\le {\epsilon }^{\left(i\right)}$.

The stability bound is as follows.

Proposition 21 (Federated stability of approximate minimizers) Let Assumption A(b) holds, and consider an algorithm $𝒜=\left({\widehat{\mathit{w}}}^{\left(\text{global}\right)},{\left\{{\widehat{\mathit{w}}}^{\left(i\right)}\right\}}_1^m\right)$ that produces an $\left({\epsilon }^{\left(\text{global}\right)},{\left\{{\epsilon }^{\left(i\right)}\right\}}_1^m\right)$-minimizer of the objective function (20) on the dataset $\mathit{S}$. Assume in addition that

$$n_i\ge \frac{4\beta }{\mu },p_i\lambda \le \frac{\mu }{16}\forall i\in \left[m\right].$$ (48)

Then $𝒜$ has federated stability

$${\gamma }_i\le \frac{160\beta \Vert \ell {\Vert }_{\infty }}{n_i\left(\mu +\lambda \right)}+{\text{Err}}_i,$$

Where

$${\text{Err}}_i≔2\sqrt{2\beta \Vert \ell {\Vert }_{\infty }}\left[4{\epsilon }^{\text{(global) }}\left(\frac{\beta +\lambda }{\mu +\lambda }+\frac{3\lambda }{\mu }\right)+{\epsilon }^{\left(i\right)}\left(\sqrt{\frac{\beta +\lambda }{\mu +\lambda }}+\frac{16p_i\lambda }{\mu }\right)\right] +8{\beta }^2\left[16{\left({\epsilon }^{\text{(global) }}\right)}^2{\left(\frac{\beta +\lambda }{\mu +\lambda }+\frac{3\lambda }{\mu }\right)}^2+{\left({\epsilon }^{\left(i\right)}\right)}^2{\left(\sqrt{\frac{\beta +\lambda }{\mu +\lambda }}+\frac{16p_i\lambda }{\mu }\right)}^2\right]$$

is the error term due to not exactly minimizing the soft weight sharing objective (20).

Proof See Appendix C.3.1. ■

Taking the optimization error into account, we have the following result.

Proposition 22 (Federated stability of ${𝒜}_{\mathbf{F}\mathbf{P}}$) Let Assumption A(a, b) and (48) hold. Run ${𝒜}_{\text{FP}}$ with hyperparameters chosen as in Lemma 14 and 15. Then, as long as

$$T\ge C_1\cdot {\lambda }^2(\lambda \vee 1{)}^{2}m\Vert \mathbf{p}{\Vert }^2{n}_i^2,K_T\ge C_2\cdot {\lambda }^2(\lambda \vee 1{)}^2{p}_i^2{n}_i^2\forall i\in \left[m\right],$$ (49)

the algorithm ${𝒜}_{\text{FP}}$ have expected federated stability

$${\mathbb{E}}_{{𝒜}_{\text{FP}}}\left[{\gamma }_i\right]\le C\cdot \frac{\beta \Vert \ell {\Vert }_{\infty }}{n_i\left(\mu +\lambda \right)},$$

where $C_1,C_2$ are two constants only depending on $\left(\mu ,\beta ,\Vert \ell {\Vert }_{\infty },D\right)$, and $C$ is an absolute constant.

Proof By Proposition 21, it suffices to upper bound the error term ${\text{Err}}_i$ by a constant multiple of $\frac{\beta \Vert \ell {\Vert }_{\infty }}{n_i(\mu +\lambda )}$. Invoking Lemma 15, we have

$${\left({\mathbb{E}}_{{𝒜}_{\text{FP}}}\left[{\epsilon }^{\left(\text{global }\right)}\right]\right)}^2\le {\mathbb{E}}_{{𝒜}_{\text{FP}}}\left[{\left({\epsilon }^{\left(\text{global }\right)}\right)}^2\right]\le \frac{12(\lambda +\mu {)}^{2}m\Vert \mathbf{p}{\Vert }^2\left({\beta }^2{D}^2\wedge 2\lambda \Vert \ell {\Vert }_{\infty }\wedge {\lambda }^2{D}^2\right)}{{\lambda }^2{\mu }^2\left(T+1\right)}.$$

This gives

$${\mathbb{E}}_{{𝒜}_{\text{FP}}}\left[{\left({\epsilon }^{\left(\text{global}\right)}\right)}^2\right]{\lesssim }_{\left(\mu ,\beta ,\Vert \mu {\Vert }_{\infty },D\right)}\frac{(\lambda \vee 1{)}^{2}m\Vert \mathbf{p}{\Vert }^2\left(1\wedge \lambda \wedge {\lambda }^2\right)}{{\lambda }^2\left(T+1\right)}\lesssim \frac{m\Vert \mathbf{p}{\Vert }^2}{T+1},$$ (50)

where we recall that ${a_n\lesssim }_{\left(\mu ,\beta ,\Vert \mu {\Vert }_{\infty },D\right)}{b}_n$ means $\left|a_n\right|\le C{b}_n$ for a constant $C$ that only depending on $\left(\mu ,\beta ,\Vert \mu {\Vert }_{\infty },D\right)$, and the last inequality follows from $(\lambda \vee 1{)}^2\left(1\wedge \lambda \wedge {\lambda }^2\right)\le {\lambda }^2$ regardless $\lambda \ge 1$ or $\lambda \le 1$. Meanwhile, by Lemma 14, we have

$${\left.\left({\mathbb{E}}_{{𝒜}_{\text{FP}}}\left[{\epsilon }^{\left(i\right)}\right)\right]\right)}^2\le {\mathbb{E}}_{{𝒜}_{\text{FP}}}\left[{\left({\epsilon }^{\left(i\right)}\right)}^2\right]\le \frac{8{\beta }^2{D}^2}{{\mu }^2\left(K_T+1\right)}\lesssim \left(\beta ,D\right)\frac{1}{K_T+1}.$$

Recalling the definition of ${\text{Err}}_i$, we have

$$\begin{gathered} {\mathbb{E}}_{{𝒜}_{\text{FP}}}\left[{\text{Err}}_i\right] \\ {\lesssim }_{\left(\mu ,\beta ,\Vert \mu {\Vert }_{\infty },D\right)}\lambda {\mathbb{E}}_{{𝒜}_{\text{FP}}}\left[{\epsilon }^{\left(\text{global }\right)}\right]+\lambda p_i{\mathbb{E}}_{{𝒜}_{\text{FP}}}\left[{\epsilon }^{\left(i\right)}\right]+{\lambda }^2{\mathbb{E}}_{{𝒜}_{\text{FP}}}\left[{\left({\epsilon }^{\left(\text{global }\right)}\right)}^2\right]+p_i^2{\lambda }^2{\mathbb{E}}_{{𝒜}_{\text{FP}}}\left[{\left({\epsilon }^{\left(i\right)}\right)}^2\right] \\ {\lesssim }_{\left(\mu ,\beta ,\Vert \mu {\Vert }_{\infty },D\right)}\frac{\lambda \sqrt{m}\Vert \mathbf{p}\Vert }{\sqrt{T+1}}+\frac{\lambda p_i}{\sqrt{K_T+1}}+\frac{{\lambda }^{2}m\Vert \mathbf{p}{\Vert }^2}{T+1}+\frac{p_i^2{\lambda }^2}{K_T+1}. \end{gathered}$$

Thus, it suffices to require

$$\begin{gathered} \sqrt{T}\gtrsim { }_{\left(\mu ,\beta ,\Vert \mu {\Vert }_{\infty },D\right)}\lambda \sqrt{m}\Vert \mathbf{p}\Vert n_i(\mu +\lambda ),T\gtrsim { }_{\left(\mu ,\beta ,\Vert \mu {\Vert }_{\infty },D\right)}{\lambda }^{2}m\Vert \mathbf{p}{\Vert }^2{n}_i(\mu +\lambda ), \\ \sqrt{K_T}\gtrsim { }_{\left(\mu ,\beta ,\Vert \mu {\Vert }_{\infty },D\right)}\lambda p_i{n}_i(\mu +\lambda ),K_T\gtrsim { }_{\left(\mu ,\beta ,\Vert \mu {\Vert }_{\infty },D\right)}{p}_i^2{\lambda }^2{n}_i(\mu +\lambda ), \end{gathered}$$

which is equivalent to

$$T{\gtrsim }_{\left(\mu ,\beta ,\Vert \mu {\Vert }_{\infty },D\right)}\text{max}\left\{{\lambda }^{2}m\Vert \mathbf{p}{\Vert }^2{n}_i^2(\lambda \vee 1{)}^2,{\lambda }^{2}m\Vert \mathbf{p}{\Vert }^2{n}_i(\lambda \vee 1)\right\}={\lambda }^{2}m\Vert \mathbf{p}{\Vert }^2{n}_i^2(\lambda \vee 1{)}^2$$

$$K_T\gtrsim \left(\mu ,\beta ,\Vert \mu {\Vert }_{\infty },D\right)\text{max}\left\{{\lambda }^2{p}_i^2{n}_i^2(\mu \vee 1{)}^2,p_i^2{\lambda }^2{n}_i(\mu \vee 1)\right\}={\lambda }^2{p}_i^2{n}_i^2(\lambda \vee 1{)}^2,$$

which is exactly (49). ■

Combining the above proposition with Proposition 10, we get the following result.

Proposition 23 ($\mathit{\lambda }$-dependent bound on the AER) Let Assumption A (a, b) and (48) hold. Run ${𝒜}_{\text{FP}}$ with hyperparameters chosen as in Lemma 14 and 15. Then, as long as

$$\begin{gathered} T \ge C_1\cdot \lambda (\lambda \vee 1)m\Vert \mathbf{p}{\Vert }^2\cdot \left({\left[\sum _{s\in \left[m\right]}{p}_s/n_s\right]}^{-1}\vee \left[\lambda (\lambda \vee 1)n_i^2\right]\right), \\ {K}_T \ge C_2\cdot (\lambda +1{)}^2\cdot \left({\left[\sum _{s\in \left[m\right]}{p}_s/n_s\right]}^{-1}\vee \left[{\lambda }^2{p}_i^2{n}_i^2\right]\right), \end{gathered}$$ (51)

for any $i\in \left[m\right]$, the algorithm ${𝒜}_{\text{FP}}$ satisfies

$${\mathbb{E}}_{{𝒜}_{\text{FP}},\mathit{S}}\left[{\text{AER}}_{\mathbf{p}}\left({𝒜}_{\text{FP}}\right)\right]\le C\cdot \frac{\beta \Vert \ell {\Vert }_{\infty }}{\mu +\lambda }\sum _{i\in \left[m\right]}\frac{p_i}{n_i}+\frac{\lambda }{2}\sum _{i\in \left[m\right]}{p}_i{∥{\mathit{w}}_{\mathbf{p}}^{\left(\text{global }\right)}-{\mathit{w}}_{\star }^{\left(i\right)}∥}^2,$$

where $C_1,C_2$ are two constants only depending on $\left(\mu ,\beta ,\Vert \ell {\Vert }_{\infty },D\right)$, and $C$ is an absolute constant.

Proof In view of Propositions 10 and 22, it suffices to set $T,K_T$ such that (1) (49) is satisfied; and (2) ${\mathbb{E}}_{{𝒜}_{\text{FP}}}\left[{ℰ}_{\text{OPT}}\right]$ is upper bounded by a constant multiple of $\frac{\beta \Vert \ell {\Vert }_{\infty }}{\mu +\lambda }{\sum }_{i\in \left[m\right]}\frac{p_i}{n_i}$. To achive the second goal, note that by Proposition 16, the optimization error is bounded by

$${\mathbb{E}}_{{𝒜}_{\text{FP}}}\left[{ℰ}_{\text{OPT}}\right]{\lesssim }_{\left(\mu ,\beta ,\Vert \mu {\Vert }_{\infty },D\right)}\frac{\lambda \vee 1}{K_T+1}+\frac{\lambda m\Vert \mathbf{p}{\Vert }^2}{T+1}.$$ (52)

Thus, it suffices to require ${T\gtrsim }_{\left(\mu ,\beta ,\Vert \mu {\Vert }_{\infty },D\right)}\frac{\lambda (\lambda \vee 1)m\Vert \mathbf{p}{\Vert }^2}{{\sum }_{i\in \left[m\right]}{p}_i/n_i}$ and ${K_T\gtrsim }_{\left(\mu ,\beta ,\Vert \mu {\Vert }_{\infty },D\right)}\frac{(\lambda \vee 1{)}^2}{{\sum }_{i\in \left[m\right]}{p}_i/n_i}$. This requirement, combined with (49), is exactly (51). ■

With the above proposition at hand, we are ready to give our proof of Theorem 11.

Proof [Proof of Theorem 11] We first define the following three events:

$$\text{A}≔\left\{R\ge \sqrt{\sum _{i\in \left[m\right]}\frac{p_i}{n_i}}\right\},\text{B}≔\left\{\frac{\sum _{i\in \left[m\right]}{p}_i^2/n_i}{\sqrt{\sum _{i\in \left[m\right]}{p}_i/n_i}}\le R\le \sqrt{\sum _{i\in \left[m\right]}\frac{p_i}{n_i}}\right\},\text{C}≔\left\{R\le \frac{\sum _{i\in \left[m\right]}{p}_i^2/n_i}{\sqrt{\sum _{i\in \left[m\right]}{p}_i/n_i}}\right\}.$$

We then choose $\lambda$ to be

$$\lambda =\frac{\mu }{16R^2}\sum _{i\in \left[m\right]}\frac{p_i}{n_i}\cdot {\mathbb{1}}_{\text{A}}+\frac{\mu }{16C_{\mathbf{p}}R}\sqrt{\sum _{i\in \left[m\right]}\frac{p_i}{n_i}}\cdot {\mathbb{1}}_{\text{B}}+\frac{\mu }{16C_{\mathbf{p}}\sum _{i\in \left[m\right]}{p}_i^2/n_i}\sum _{i\in \left[m\right]}\frac{p_i}{n_i}\cdot {\mathbb{1}}_{\text{C}}.$$

We now consider the three events separately.

1. If A holds, then $p_i\lambda =\frac{p_i\mu }{16R^2}{\sum }_{i\in \left[m\right]}\frac{p_i}{{\mu }_i}\le \frac{p_i\mu }{16}\le \frac{\mu }{16}$. Thus we can invoke Proposition 23 to get

   $${\mathbb{E}}_{{𝒜}_{\text{FP}},\mathit{S}}\left[{\text{AR}}_{\mathbf{p}}\right]\le \left(\frac{C\beta \parallel \ell {\parallel }_{\infty }}{\mu }+\frac{\mu }{32}\right)\sum _{i\in \left[m\right]}\frac{p_i}{n_i}\lesssim \text{right-hand side of }\left(28\right).$$
2. If $\text{B}$ holds, then $p_i\lambda =\frac{p_i\mu }{16C_{\mathbf{p}}R}\sqrt{{\sum }_{i\in \left[m\right]}\frac{p_i}{n_i}}\le \frac{p_{\text{max}}\mu N}{16C_{\mathbf{p}}{\sum }_{i\in \left[m\right]}{p}_i^2/n_i}{\sum }_{i\in \left[m\right]}\frac{p_i}{n_i}\le \frac{\mu }{16}$, where the last inequality is by the definition of $C_{\mathbf{p}}$. Hence, by Proposition 23, we have

   $${\mathbb{E}}_{{𝒜}_{\text{FP}},\mathit{S}}\left[{\text{AER}}_{\mathbf{p}}\right]\le \left(\frac{16C{C}_{\mathbf{p}}\beta \parallel \ell {\parallel }_{\infty }}{\mu }+\frac{\mu }{32C_{\mathbf{p}}}\right)R\sqrt{\sum _{i\in \left[m\right]}\frac{p_i}{n_i}}\lesssim \text{right-hand side of }\left(28\right).$$
3. If C holds, then $p_i\lambda =\frac{p_i\mu }{16C_{\mathbf{p}}{\sum }_{i\in \left[m\right]}{p}_i^2/n_i}{\sum }_{i\in \left[m\right]}\frac{p_i}{n_i}\le \frac{\mu }{16}$, and thus Proposition 23 gives

   $${\mathbb{E}}_{{𝒜}_{\text{FP}},\mathit{S}}\left[{\text{AER}}_{\mathbf{p}}\right]\le \left(\frac{16C{C}_{\mathbf{p}}\beta \parallel \ell {\parallel }_{\infty }}{\mu }+\frac{\mu }{32C_{\mathbf{p}}}\right)\sum _{i\in \left[m\right]}\frac{p_i^2}{n_i}\lesssim \text{right-hand side of }\left(28\right).$$

The desired result follows by combining the above three cases together. ■

C.3.1. Proof of Proposition 21: Stability of Approximate Minimizers

We first present two lemmas, from which Proposition 21 will follow.

Lemma 24 (Federated stability of approximate minimizers, Part I) Let Assumption A(b) holds, and consider an algorithm $𝒜=\left({\widehat{\mathit{w}}}^{\text{(global)}},{\left\{{\widehat{\mathit{w}}}^{\left(i\right)}\right\}}_1^m\right)$ that satisfies the following conditions:

1. there exist positive constants ${\delta }^{\text{(global)}},{\zeta }^{\text{(global)}}$ such that

   $$\sum _{i\in \left[m\right]}{p}_i{F}_i\left({\widehat{\mathit{w}}}^{\left(\text{global}\right)},S_i\right)\le {\delta }^{\left(\text{global}\right)}+\sum _{i\in \left[m\right]}{p}_i{F}_i\left({\tilde{\mathit{w}}}^{\text{(global)}},S_i\right),$$ (53)

   $$∥\sum _{i\in \left[m\right]}{p}_i\nabla F_i\left({\widehat{\mathit{w}}}^{\text{(global)}},S_i\right)∥\le {\zeta }^{\text{(global)}}.$$ (54)
2. for any $i\in \left[m\right]$, there exist positive constants ${\left\{{\delta }^{\left(i\right)},{\zeta }^{\left(i\right)},{\epsilon }^{\left(i\right)}\right\}}_{i=1}^m$ such that

   $$L_i\left({\widehat{\mathit{w}}}^{\left(i\right)},S_i\right)+\frac{\lambda }{2}{∥{\widehat{\mathit{w}}}^{\left(\text{global}\right)}-{\widehat{\mathit{w}}}^{\left(i\right)}∥}^2\le {\delta }^{\left(i\right)}+F_i\left({\widehat{\mathit{w}}}^{\left(\text{global}\right)},S_i\right),$$ (55)

   $$∥\nabla L_i\left({\widehat{\mathit{w}}}^{\left(i\right)},S_i\right)+\lambda \left({\widehat{\mathit{w}}}^{\left(i\right)}-{\widehat{\mathit{w}}}^{\left(\text{global}\right)}\right)∥\le {\zeta }^{\left(i\right)},$$ (56)

   $$∥{\widehat{\mathit{w}}}^{\left(i\right)}-{\text{Prox}}_{L_i/\lambda }\left({\widehat{\mathit{w}}}^{\left(\text{global}\right)}\right)∥\le {\epsilon }^{\left(i\right)}.$$ (57)

Assume in addition that (48) holds. Then $𝒜$ has federated stability

$${\gamma }_i\le \frac{160\beta \Vert \ell {\Vert }_{\infty }}{n_i\left(\mu +\lambda \right)}+\sqrt{2\beta \Vert \ell {\Vert }_{\infty }}\cdot {ℰ}_{\lambda ,i}+\beta {ℰ}_{\lambda ,i}^2,$$ (58)

where

$${ℰ}_{\lambda ,i}≔\frac{8{\zeta }^{\left(i\right)}}{\mu +\lambda }+\sqrt{\frac{8{\delta }^{\left(i\right)}}{\mu +\lambda }}+8{\mu }^{-1}\left(2{\zeta }^{\left(\text{global}\right)}+4p_i\lambda {\epsilon }^{\left(i\right)}+\sqrt{\frac{2\mu \lambda {\delta }^{\left(\text{global}\right)}}{\mu +\lambda }}\right)$$ (59)

is the error term due to not exactly minimizing (20).

Lemma 25 (Federated stability of approximate minimizers, Part II) Let Assumption A(b) holds and consider an algorithm $𝒜=\left({\widehat{\mathit{w}}}^{\left(\text{global}\right)},{\left\{{\widehat{\mathit{w}}}^{\left(i\right)}\right\}}_1^m\right)$ that produces an $\left({\epsilon }^{\left(\text{global}\right)},{\left\{{\epsilon }^{\left(i\right)}\right\}}_1^m\right)$-minimizer in the sense of Definition 20. Then $𝒜$ also satisfies Equations (53)–(57) with

$${\delta }^{\left(\text{global }\right)}=\frac{\lambda }{2}{\epsilon }^{\left(\text{global }\right)},{\zeta }^{\left(\text{global }\right)}=\lambda {\epsilon }^{\left(\text{global }\right)},{\delta }^{\left(i\right)}=\frac{\beta +\lambda }{2}{\epsilon }^{\left(i\right)},{\zeta }^{\left(i\right)}=(\beta +\lambda ){\epsilon }^{\left(i\right)}.$$

Proof These correspondences are consequences of $\lambda$-smoothness of $F_i$ and $(\beta +\lambda )$-smoothness of $L_i\left(\cdot ,S_i\right)+\frac{\lambda }{2}{∥{\widehat{\mathit{w}}}^{\text{(global)}}-\cdot ∥}^2$. We omit the details. ■

With the above two lemmas at hand, the proof of Proposition 21 is purely computational:

Proof [Proof of Proposition 21 given Lemma 24 and 25] Invoking 25, the error term ${ℰ}_{\lambda ,i}$ defined in Equation 59 can be bounded above by

$$\begin{gathered} {ℰ}_{\lambda ,i} \le \frac{8(\beta +\lambda )}{\mu +\lambda }\cdot {\epsilon }^{\left(\text{global }\right)}+\sqrt{\frac{4(\beta +\lambda )}{\mu +\lambda }}\cdot {\epsilon }^{\left(i\right)}+\frac{8}{\mu }\left(2\lambda {\epsilon }^{\left(\text{global }\right)}+4p_i\lambda {\epsilon }^{\left(i\right)}+\sqrt{\frac{\mu {\lambda }^2}{\mu +\lambda }}\right) \\ =8{\epsilon }^{\left(\text{global }\right)}\left(\frac{\beta +\lambda }{\mu +\lambda }+\frac{2\lambda }{\mu }+\frac{\lambda }{\sqrt{\mu (\mu +\lambda )}}\right)+2{\epsilon }^{\left(i\right)}\left(\sqrt{\frac{\beta +\lambda }{\mu +\lambda }}+\frac{16p_i\lambda }{\mu }\right) \\ \le 8{\epsilon }^{\left(\text{global }\right)}\left(\frac{\beta +\lambda }{\mu +\lambda }+\frac{3\lambda }{\mu }\right)+2{\epsilon }^{\left(i\right)}\left(\sqrt{\frac{\beta +\lambda }{\mu +\lambda }}+\frac{16p_i\lambda }{\mu }\right). \end{gathered}$$

This gives

$${ℰ}_{\lambda ,i}^2\le 128{\left({\epsilon }^{\left(\text{global }\right)}\right)}^2{\left(\frac{\beta +\lambda }{\mu +\lambda }+\frac{3\lambda }{\mu }\right)}^2+8{\left({\epsilon }^{\left(i\right)}\right)}^2{\left(\sqrt{\frac{\beta +\lambda }{\mu +\lambda }}+\frac{16p_i\lambda }{\mu }\right)}^2.$$

Plugging the above two displays to (58) gives the desired result. ■

We now present our proof of Lemma 24. We start by stating and proving several useful lemmas.

Lemma 26 (From loss stability to parameter stability)LetAssumption A(b)holds. Then the algorithm$𝒜=\left({\widehat{\mathit{w}}}^{\text{(global)}},\left\{{\widehat{\mathit{w}}}^{\left(i\right)}\right\}\right)$has federated stability

$${\gamma }_i\le \sqrt{2\beta \Vert \ell {\Vert }_{\infty }}\cdot ∥{\widehat{\mathit{w}}}^{\left(i\right)}\left(\mathit{S}\right)-{\widehat{\mathit{w}}}^{\left(i\right)}\left({\mathit{S}}^{\setminus \left(i,j_i\right)}\right)∥+\frac{\beta }{2}{∥{\widehat{\mathit{w}}}^{\left(i\right)}\left(\mathit{S}\right)-{\widehat{\mathit{w}}}^{\left(i\right)}\left({\mathit{S}}^{\setminus \left(i,j_i\right)}\right)∥}^2.$$

Proof This lemma has implicitly appeared in the proofs of many stability-based generalization bounds (see, e.g., Section 13.3.2 of Shalev-Shwartz and Ben-David (2014)), and we provide a proof for completeness. By $\beta$-smoothness, for an arbitrary $z\in 𝒵$ we have

$$\begin{gathered} \ell \left({\widehat{\mathit{w}}}^{\left(i\right)}\left(\mathit{S}\right),z\right)-\ell \left({\widehat{\mathit{w}}}^{\left(i\right)}\left({\mathit{S}}^{\setminus \left(i,j_i\right)}\right),z\right) \\ \le ⟨\nabla \ell \left({\widehat{\mathit{w}}}^{\left(i\right)}\left({\mathit{S}}^{\setminus \left(i,j_i\right)}\right),z\right),{\widehat{\mathit{w}}}^{\left(i\right)}\left(\mathit{S}\right)-{\widehat{\mathit{w}}}^{\left(i\right)}\left({\mathit{S}}^{\setminus \left(i,j_i\right)}\right)⟩+\frac{\beta }{2}{∥{\widehat{\mathit{w}}}^{\left(i\right)}\left(\mathit{S}\right)-{\widehat{\mathit{w}}}^{\left(i\right)}\left({\mathit{S}}^{\setminus \left(i,j_i\right)}\right)∥}^2 \\ \le ∥\nabla \ell \left({\widehat{\mathit{w}}}^{\left(i\right)}\left({\mathit{S}}^{\setminus \left(i,j_i\right)}\right),z\right)∥\cdot ∥{\widehat{\mathit{w}}}^{\left(i\right)}\left(\mathit{S}\right)-{\widehat{\mathit{w}}}^{\left(i\right)}\left({\mathit{S}}^{\setminus \left(i,j_i\right)}\right)∥+\frac{\beta }{2}{∥{\widehat{\mathit{w}}}^{\left(i\right)}\left(\mathit{S}\right)-{\widehat{\mathit{w}}}^{\left(i\right)}\left({\mathit{S}}^{\setminus \left(i,j_i\right)}\right)∥}^2 \\ \le \sqrt{2\beta \left(\ell \left({\widehat{\mathit{w}}}^{\left(i\right)}\left({\mathit{S}}^{\setminus \left(i,j_i\right)}\right),z\right)-\underset{{\mathit{w}}^{\left(i\right)}\in 𝒲}{\text{min}}\ell \left({\mathit{w}}^{\left(i\right)},z\right)\right)}\cdot ∥{\widehat{\mathit{w}}}^{\left(i\right)}\left(\mathit{S}\right)-{\widehat{\mathit{w}}}^{\left(i\right)}\left({\mathit{S}}^{\setminus \left(i,j_i\right)}\right)∥+\frac{\beta }{2}{∥{\widehat{\mathit{w}}}^{\left(i\right)}\left(\mathit{S}\right)-{\widehat{\mathit{w}}}^{\left(i\right)}\left({\mathit{S}}^{\setminus \left(i,j_i\right)}\right)∥}^2 \\ \le \sqrt{2\beta \Vert \ell {\Vert }_{\infty }}\cdot ∥{\widehat{\mathit{w}}}^{\left(i\right)}\left(\mathit{S}\right)-{\widehat{\mathit{w}}}^{\left(i\right)}\left({\mathit{S}}^{\setminus \left(i,j_i\right)}\right)∥+\frac{\beta }{2}{∥{\widehat{\mathit{w}}}^{\left(i\right)}\left(\mathit{S}\right)-{\widehat{\mathit{w}}}^{\left(i\right)}\left({\mathit{S}}^{\setminus \left(i,j_i\right)}\right)∥}^2, \end{gathered}$$

where the last inequality follows from boundedness of $\ell$. By a nearly identical argument, the above upper bound also holds for $-\ell \left({\widehat{\mathit{w}}}^{\left(i\right)}\left(\mathit{S}\right),z\right)+\ell \left({\widehat{\mathit{w}}}^{\left(i\right)}\left({\mathit{S}}^{\setminus \left(i,j_i\right)}\right),z\right)$, and the desired result follows. ■

Lemma 27 (Local stability implies global stability) Assume Assumption A(b) holds and consider an algorithm $𝒜=\left({\widehat{\mathit{w}}}^{\text{(global)}},{\left\{{\widehat{\mathit{w}}}^{\left(i\right)}\right\}}_1^m\right)$ that satisfies Equations (53), (54) and (57). Then for any $i\in \left[m\right]$, $j_i\in \left[n_i\right]$, we have

$$∥{\widehat{\mathit{w}}}^{\text{(global)}}\left({\mathit{S}}^{\setminus \left(i,j_i\right)}\right)-{\widehat{\mathit{w}}}^{\text{(global)}}\left(\mathit{S}\right)∥ \le \frac{\lambda +\mu }{\lambda \mu }\left(2{\zeta }^{\left(\text{global}\right)}+\sqrt{\frac{2\lambda \mu {\delta }^{\left(\text{global}\right)}}{\lambda +\mu }}+4p_i\lambda {\epsilon }^{\left(i\right)}+2p_i\lambda ∥{\widehat{\mathit{w}}}^{\left(i\right)}\left({\mathit{S}}^{\setminus \left(i,j_1\right)}\right)-{\widehat{\mathit{w}}}^{\left(i\right)}\left(\mathit{S}\right)∥\right).$$ (60)

Proof Without loss of generality we consider the first client. Let ${\mu }_F$ be the strongly convex constant of ${\sum }_i{p}_i{F}_i$, which, by Lemma 17, is equal to ${\sum }_i{p}_i\cdot \frac{\mu \lambda }{\mu +\lambda }=\lambda \mu /(\lambda +\mu )$. Now, by strong convexity, we have

$$\begin{gathered} \frac{{\mu }_F}{2}{∥{\widehat{\mathit{w}}}^{\left(\text{global }\right)}\left(\mathit{S}\right)-{\widehat{\mathit{w}}}^{\text{(global) }}\left({\mathit{S}}^{\setminus \left(1,j_1\right)}\right)∥}^2 \\ \le \sum _{i\in \left[m\right]}{p}_i\left(F_i\left({\widehat{\mathit{w}}}^{\left(\text{global }\right)}\left({\mathit{S}}^{\setminus \left(1,j_1\right)}\right),S_i\right)-F_i\left({\widehat{\mathit{w}}}^{\text{(global) }}\left(\mathit{S}\right),S_i\right)\right)+⟨\sum _{i\in \left[m\right]}{p}_i\nabla F_i\left({\widehat{\mathit{w}}}^{\text{(global) }}\left(\mathit{S}\right),S_i\right),{\widehat{\mathit{w}}}^{\text{(global) }}\left({\mathit{S}}^{\setminus \left(1,j_1\right)}-{\widehat{\mathit{w}}}^{\text{(global) }}\left(\mathit{S}\right)\right)⟩ \\ \stackrel{\left(54\right)}{\le }\sum _{i\in \left[m\right]}{p}_i\left(F_i\left({\widehat{\mathit{w}}}^{\text{(global) }}\left({\mathit{S}}^{\setminus \left(1,j_1\right)}\right),S_i\right)-F_i\left({\widehat{\mathit{w}}}^{\text{(global) }}\left(\mathit{S}\right),S_i\right)\right)+{\zeta }^{\left(\text{global }\right)}∥{\widehat{\mathit{w}}}^{\left(\text{global }\right)}\left({\mathit{S}}^{\setminus \left(1,j_1\right)}\right)-{\widehat{\mathit{w}}}^{\left(\text{global }\right)}\left(\mathit{S}\right)∥ \\ =\left(p_1{F}_1\left({\widehat{\mathit{w}}}^{\text{(global) }}\left({\mathit{S}}^{\setminus \left(1,j_1\right)}\right),S_1^{\setminus j_1}\right)+\sum _{i\ne 1}{p}_i{F}_i\left({\widehat{\mathit{w}}}^{\text{(global) }}\left({\mathit{S}}^{\setminus \left(1,j_1\right)}\right),S_i\right)\right)-\left(p_1{F}_1\left({\widehat{\mathit{w}}}^{\text{(global) }}\left(\mathit{S},S_1^{\setminus j_1}\right)+\sum _{i\ne 1}{p}_i{F}_i\left({\widehat{\mathit{w}}}^{\text{(global) }}\left(\mathit{S},S_i\right)\right)\right.\right.+p_1\left(F_1\left({\widehat{\mathit{w}}}^{\text{(global) }}\left({\mathit{S}}^{\setminus \left(1,j_1\right)}\right),S_1\right)-F_1\left({\widehat{\mathit{w}}}^{\text{(global) }}\left({\mathit{S}}^{\setminus \left(1,j_1\right)}\right),S_1^{\setminus j_1}\right)\right.\left.+F_1\left({\widehat{\mathit{w}}}^{\text{(global) }}\left(\mathit{S}\right),S_1^{\setminus j_1}\right)-F_1\left({\widehat{\mathit{w}}}^{\text{(global) }}\left(\mathit{S}\right),S_1\right)\right)+{\zeta }^{\text{(global) }}∥{\widehat{\mathit{w}}}^{\text{(global) }}\left({\mathit{S}}^{\setminus \left(1,j_1\right)}\right)-{\widehat{\mathit{w}}}^{\text{(global) }}\left(\mathit{S}\right)∥ \\ \stackrel{\left(53\right)}{\le }{\delta }^{\left(\text{global }\right)}+{\zeta }^{\text{(global) })}∥{\widehat{\mathit{w}}}^{\text{(global) }}\left({\mathit{S}}^{\setminus \left(1,j_1\right)}\right)-{\widehat{\mathit{w}}}^{\text{(global) }}\left(\mathit{S}\right)∥+p_1\left(F_1\left({\widehat{\mathit{w}}}^{\text{(global) }}\left({\mathit{S}}^{\setminus \left(1,j_1\right)}\right),S_1\right)-F_1\left({\widehat{\mathit{w}}}^{\text{(global) }}\left(\mathit{S}\right),S_1\right)\right.\left.+F_1\left({\widehat{\mathit{w}}}^{\text{(global) }}\left(\mathit{S}\right),S_1^{\setminus j_1}\right)-F_1\left({\widehat{\mathit{w}}}^{\text{(global) })}\left({\mathit{S}}^{\setminus \left(1,j_1\right)}\right),S_1^{\setminus j_1}\right)\right). \end{gathered}$$

Since $F_1$ is $\lambda$-smooth by Lemma 17, we can proceed by

$$\begin{gathered} \frac{{\mu }_F}{2}{∥{\widehat{\mathit{w}}}^{\text{(global) }}\left(\mathit{S}\right)-{\widehat{\mathit{w}}}^{\text{(global) }}\left({\mathit{S}}^{\setminus \left(1,j_1\right)}\right)∥}^2 \\ \le {\delta }^{\text{(global) }}+{\zeta }^{\text{(global) }}∥{\widehat{\mathit{w}}}^{\text{(global) }}\left({\mathit{S}}^{\setminus \left(1,j_1\right)}\right)-{\widehat{\mathit{w}}}^{\text{(global) }}\left(\mathit{S}\right)∥+p_1\lambda {∥{\widehat{\mathit{w}}}^{\text{(global) }}\left({\mathit{S}}^{\setminus \left(1,j_1\right)}\right)-{\widehat{\mathit{w}}}^{\text{(global) }}\left(\mathit{S}\right)∥}^2+p_1⟨\nabla F_1\left({\widehat{\mathit{w}}}^{\text{(global) }}\left(\mathit{S}\right),S_1\right)-\nabla F_1\left({\widehat{\mathit{w}}}^{\text{(global) }}\left({\mathit{S}}^{\setminus \left(1,j_1\right)}\right),S_1^{\setminus j_1}\right),{\widehat{\mathit{w}}}^{\text{(global) }}\left({\mathit{S}}^{\setminus \left(1,j_1\right)}\right)-{\widehat{\mathit{w}}}^{\text{(global) }}\left(\mathit{S}\right)⟩. \end{gathered}$$

Since $\nabla F_1\left({\mathit{w}}^{\left(\text{global}\right)},S_1\right)=\lambda \left({\mathit{w}}^{\text{(global)}}-{\text{Prox}}_{L_1/\lambda }\left({\mathit{w}}^{\text{(global)}},S_1\right)\right)$, with some algebra, the right-hand side above is in fact equal to

$$\begin{gathered} {\delta }^{\text{(global) }}+{\zeta }^{\text{(global) }}∥{\widehat{\mathit{w}}}^{\text{(global) }}\left({\mathit{S}}^{\setminus \left(1,j_1\right)}\right)-{\widehat{\mathit{w}}}^{\text{(global) }}\left(\mathit{S}\right)∥+p_1\lambda ⟨{\widehat{\mathit{w}}}^{\left(1\right)}\left(\mathit{S}\right)-{\text{Prox}}_{L_1/\lambda }\left({\widehat{\mathit{w}}}^{\left(\text{global }\right)}\left(\mathit{S}\right),S_1\right),{\widehat{\mathit{w}}}^{\left(\text{global }\right)}\left({\mathit{S}}^{\setminus \left(1,j_1\right)}\right)-{\widehat{\mathit{w}}}^{\left(\text{global }\right)}\left(\mathit{S}\right)⟩+p_1\lambda ⟨{\text{Prox}}_{L_1/\lambda }\left({\widehat{\mathit{w}}}^{\left(\text{global }\right)}\left({\mathit{S}}^{\setminus \left(1,j_1\right)}\right),S_1^{\setminus j_1}\right)-{\widehat{\mathit{w}}}^{\left(1\right)}\left({\mathit{S}}^{\setminus \left(1,j_1\right)},{\widehat{\mathit{w}}}^{\left(\text{global }\right)}\left({\mathit{S}}^{\setminus \left(1,j_1\right)}\right)-{\widehat{\mathit{w}}}^{\left(\text{global }\right)}\left(\mathit{S}\right)\right.⟩+p_1\lambda ⟨{\widehat{\mathit{w}}}^{\left(1\right)}\left({\mathit{S}}^{\setminus \left(1,j_1\right)}\right)-{\widehat{\mathit{w}}}^{\left(1\right)}\left(\mathit{S}\right),{\widehat{\mathit{w}}}^{\text{(global) }}\left({\mathit{S}}^{\setminus \left(1,j_1\right)}\right)-{\widehat{\mathit{w}}}^{\text{(global) }}\left(\mathit{S}\right)⟩ \\ \stackrel{\text{ (57) }}{\le }{\delta }^{\text{(global) }}+\left({\zeta }^{\text{(global) }}+2p_1\lambda \right)∥{\widehat{\mathit{w}}}^{\text{(global) }}\left({\mathit{S}}^{\setminus \left(1,j_1\right)}\right)-{\widehat{\mathit{w}}}^{\text{(global) }}\left(\mathit{S}\right)∥+p_1\lambda ∥{\widehat{\mathit{w}}}^{\left(i\right)}\left({\mathit{S}}^{\setminus \left(1,j_1\right)}\right)-{\widehat{\mathit{w}}}^{\left(i\right)}\left(\mathit{S}\right)∥∥{\widehat{\mathit{w}}}^{\left(\text{global }\right)}\left({\mathit{S}}^{\setminus \left(1,j_1\right)}\right)-{\widehat{\mathit{w}}}^{\text{(global) }}\left(\mathit{S}\right)∥. \end{gathered}$$

The above bound gives a quadratic inequality: if we let ${\mathbf{s}}_G≔∥{\widehat{\mathit{w}}}^{\text{(global)}}\left({\mathit{S}}^{\setminus \left(1,j_1\right)}\right)-{\widehat{\mathit{w}}}^{\left(\text{global}\right)}\left(\mathit{S}\right)∥$ and ${\mathbf{s}}_1≔∥{\widehat{\mathit{w}}}^{\left(i\right)}\left({\mathit{S}}^{\setminus \left(1,j_1\right)}\right)-{\widehat{\mathit{w}}}^{\left(i\right)}\left(\mathit{S}\right)∥$, then the above bound can be written as

$$\frac{{\mu }_F}{2}\cdot {\text{s}}_G^2-\left({\zeta }^{\left(\text{global }\right)}+2p_1\lambda {\epsilon }^{\left(1\right)}+p_1\lambda {\text{s}}_1\right)\cdot {\text{s}}_G-{\delta }^{\left(\text{global }\right)}\le 0.$$

Solving this inequality gives

$$\begin{gathered} {\text{s}}_G \le \frac{1}{{\mu }_F}\cdot \left[{\zeta }^{\left(\text{global}\right)}+2p_1\lambda {\epsilon }^{\left(1\right)}+p_1\lambda {\text{s}}_1+\sqrt{{\left({\zeta }^{\text{(global)}}+2p_1\lambda {\epsilon }^{\left(1\right)}+p_1\lambda {\text{s}}_1\right)}^2+2{\mu }_F{\delta }^{\text{(global)}}}\right] \\ \le \frac{1}{{\mu }_F}\left(2{\zeta }^{\text{(global)}}+4p_1\lambda {\epsilon }^{\left(1\right)}+2p_1\lambda {\text{s}}_1+\sqrt{2{\mu }_F{\delta }^{\left(\text{global}\right)}}\right), \end{gathered}$$

which is exactly (60). ■

Lemma 28 (Parameter stability) Under the same assumptions as Proposition 21, for any $i\in \left[m\right]$, $j_i\in \left[n_i\right]$, we have

$$∥{\widehat{\mathit{w}}}^{\left(i\right)}\left({\mathit{S}}^{\setminus \left(i,j_i\right)}\right)-{\widehat{\mathit{w}}}^{\left(i\right)}\left(\mathit{S}\right)∥\le \frac{16\sqrt{2\beta \Vert \ell {\Vert }_{\infty }}}{n_i\left(\mu +\lambda \right)}+{ℰ}_{\lambda ,i}.$$

Proof Without loss of generality we consider the first client. Since $L_1\left(\cdot ,S_1\right)+\frac{\lambda }{2}\Vert {\widehat{\mathit{w}}}^{\text{(global)}}\left(\mathit{S}\right)-\cdot {\Vert }^2$ is $(\mu +\lambda )$-strongly convex, we have

$$\begin{gathered} \frac{1}{2}(\mu +\lambda ){∥{\widehat{\mathit{w}}}^{\left(1\right)}\left(\mathit{S}\right)-{\widehat{\mathit{w}}}^{\left(1\right)}\left({\mathit{S}}^{\setminus \left(1,j_1\right)}\right)∥}^2 \\ \le \left(L_1\left({\widehat{\mathit{w}}}^{\left(1\right)}\left({\mathit{S}}^{\setminus \left(1,j_1\right)}\right),S_1\right)+\frac{\lambda }{2}{∥{\widehat{\mathit{w}}}^{\left(\text{global }\right)}\left(\mathit{S}\right)-{\widehat{\mathit{w}}}^{\left(1\right)}\left({\mathit{S}}^{\setminus \left(1,j_1\right)}\right)∥}^2\right)-\left(L_1\left({\widehat{\mathit{w}}}^{\left(1\right)}\left(\mathit{S}\right),S_1\right)+\frac{\lambda }{2}{∥{\widehat{\mathit{w}}}^{\left(\text{global }\right)}\left(\mathit{S}\right)-{\widehat{\mathit{w}}}^{\left(1\right)}\left(\mathit{S}\right)∥}^2\right)+⟨\nabla L_1\left({\widehat{\mathit{w}}}^{\left(1\right)}\left(\mathit{S}\right),S_1\right)+\lambda \left({\widehat{\mathit{w}}}^{\left(1\right)}\left(\mathit{S}\right)-{\widehat{\mathit{w}}}^{\left(\text{global }\right)}\left(\mathit{S}\right)\right),{\widehat{\mathit{w}}}^{\left(1\right)}\left({\mathit{S}}^{\setminus \left(1,j_1\right)}\right)-{\widehat{\mathit{w}}}^{\left(1\right)}\left(\mathit{S}\right)⟩ \\ \stackrel{\left(56\right)}{\le }\left(L_1\left({\widehat{\mathit{w}}}^{\left(1\right)}\left({\mathit{S}}^{\setminus \left(1,j_1\right)}\right),S_1^{\setminus j_1}\right)+\frac{\lambda }{2}{∥{\widehat{\mathit{w}}}^{\left(\text{global }\right)}\left({\mathit{S}}^{\setminus \left(1,j_1\right)}\right)-{\widehat{\mathit{w}}}^{\left(1\right)}\left({\mathit{S}}^{\setminus \left(1,j_1\right)}\right)∥}^2\right)-\left(L_1\left({\widehat{\mathit{w}}}^{\left(1\right)}\left(\mathit{S}\right),S_1^{\setminus j_1}\right)+\frac{\lambda }{2}{∥{\widehat{\mathit{w}}}^{\left(\text{global }\right)}\left({\mathit{S}}^{\setminus \left(1,j_1\right)}\right)-{\widehat{\mathit{w}}}^{\left(1\right)}\left(\mathit{S}\right)∥}^2\right)-\frac{1}{n_1}\ell \left({\widehat{\mathit{w}}}^{\left(1\right)}\left({\mathit{S}}^{\setminus \left(1,j_1\right)}\right),z_{1,j_1}^{\prime }\right)+\frac{1}{n_1}\ell \left({\widehat{\mathit{w}}}^{\left(1\right)}\left({\mathit{S}}^{\setminus \left(1,j_1\right)}\right),z_{j_1}^{\left(1\right)}\right)+\frac{1}{n}\ell \left({\widehat{\mathit{w}}}^{\left(1\right)}\left(\mathit{S}\right),z_{1,j_1}^{\prime }\right)-\frac{1}{n_1}\ell \left({\widehat{\mathit{w}}}^{\left(\mathit{S}\right)},z_{j_1}^{\left(1\right)}\right)-\frac{\lambda }{2}{∥{\widehat{\mathit{w}}}^{\left(\text{global }\right)}\left({\mathit{S}}^{\setminus \left(1,j_1\right)}\right)-{\widehat{\mathit{w}}}^{\left(1\right)}\left({\mathit{S}}^{\setminus \left(1,j_1\right)}\right)∥}^2+\frac{\lambda }{2}{∥{\widehat{\mathit{w}}}^{\left(\text{global }\right)}\left(\mathit{S}\right)-{\widehat{\mathit{w}}}^{\left(1\right)}\left({\mathit{S}}^{\setminus \left(1,j_1\right)}\right)∥}^2+\frac{\lambda }{2}{∥{\widehat{\mathit{w}}}^{\left(\text{global }\right)}\left({\mathit{S}}^{\setminus \left(1,j_1\right)}\right)-{\widehat{\mathit{w}}}^{\left(1\right)}\left(\mathit{S}\right)∥}^2-\frac{\lambda }{2}{∥{\widehat{\mathit{w}}}^{\left(\text{global }\right)}\left(\mathit{S}\right)-{\widehat{\mathit{w}}}^{\left(1\right)}\left(\mathit{S}\right)∥}^2+{\zeta }^{\left(1\right)}∥{\widehat{\mathit{w}}}^{\left(1\right)}\left({\mathit{S}}^{\left(1,j_1\right)}\right)-{\widehat{\mathit{w}}}^{\left(1\right)}\left(\mathit{S}\right)∥ \\ \stackrel{\left(55\right)}{\le }{\delta }^{\left(1\right)}+{\zeta }^{\left(1\right)}∥{\widehat{\mathit{w}}}^{\left(1\right)}\left({\mathit{S}}^{\setminus \left(1,j_1\right)}\right)-{\widehat{\mathit{w}}}^{\left(1\right)}\left(\mathit{S}\right)∥+\lambda ⟨{\widehat{\mathit{w}}}^{\left(\text{global }\right)}\left(\mathit{S}\right)-{\widehat{\mathit{w}}}^{\left(\text{global }\right)}\left({\mathit{S}}^{\setminus \left(1,j_1\right)}\right),{\widehat{\mathit{w}}}^{\left(\mathit{S}\right)}-{\widehat{\mathit{w}}}^{\left(1\right)}\left({\mathit{S}}^{\setminus \left(1,j_1\right)}\right)⟩+\frac{1}{n_1}\left(\ell \left({\widehat{\mathit{w}}}^{\left(1\right)}\left(\mathit{S}\right),z_{1,j_1}^{\prime }\right)-\ell \left({\widehat{\mathit{w}}}^{\left(1\right)}\left({\mathit{S}}^{\setminus \left(1,j_1\right)}\right),z_{1,j_1}^{\prime }\right)+\ell \left({\widehat{\mathit{w}}}^{\left(1\right)}\left({\mathit{S}}^{\setminus \left(1,j_1\right)}\right),z_{j_1}^{\left(1\right)}\right)-\ell \left({\widehat{\mathit{w}}}^{\left(1\right)}\left(\mathit{S}\right),z_{j_1}^{\left(1\right)}\right)\right) \\ \le {\delta }^{\left(1\right)} +{\zeta }^{\left(1\right)}∥{\widehat{\mathit{w}}}^{\left(1\right)}\left({\mathit{S}}^{\setminus \left(1,j_1\right)}\right)-{\widehat{\mathit{w}}}^{\left(1\right)}\left(\mathit{S}\right)∥+\frac{2}{n_1}\left(\sqrt{2\beta \Vert \ell {\Vert }_{\infty }}∥{\widehat{\mathit{w}}}^{\left(1\right)}\left(\mathit{S}\right)-{\widehat{\mathit{w}}}^{\left(1\right)}\left({\mathit{S}}^{\setminus \left(1,j_1\right)}\right)∥+\frac{\beta }{2}{∥{\widehat{\mathit{w}}}^{\left(1\right)}\left(\mathit{S}\right)-{\widehat{\mathit{w}}}^{\left(1\right)}\left({\mathit{S}}^{\setminus \left(1,j_1\right)}\right)∥}^2\right)+\frac{\lambda +\mu }{\mu }\left(2{\zeta }^{\left(\text{global }\right)}+\sqrt{\frac{2\lambda \mu \delta \text{ (global) }}{\lambda +\mu }}+4p_i\lambda {\epsilon }^{\left(i\right)}+2p_i\lambda ∥{\widehat{\mathit{w}}}^{\left(i\right)}\left({\mathit{S}}^{\setminus \left(i,j_1\right)}\right)-{\widehat{\mathit{w}}}^{\left(i\right)}\left(\mathit{S}\right)∥\right)\times ∥{\widehat{\mathit{w}}}^{\left(1\right)}\left(\mathit{S}\right)-{\widehat{\mathit{w}}}^{\left(1\right)}\left({\mathit{S}}^{\setminus \left(1,j_1\right)}\right)∥, \end{gathered}$$

where the last inequality is by Lemma 26 and Lemma 27. Denoting $s_1≔\Vert {\widehat{\mathit{w}}}^{\left(1\right)}\left(\mathit{S}\right)-{\widehat{\mathit{w}}}^{\left(1\right)}\left({\mathit{S}}^{\left.\setminus 1,j_1\right)}\right)\Vert$, the above inequality can be written as

$$C_{\lambda ,1}{\text{s}}_1^2-\left[\frac{2\sqrt{2\beta \Vert \ell {\Vert }_{\infty }}}{n_1}+{\zeta }^{\left(1\right)}+\frac{\lambda +\mu }{\mu }\left(2{\zeta }^{\text{(global)}}+4p_1\lambda {\epsilon }^{\left(1\right)}+\sqrt{\frac{2\lambda \mu {\delta }^{\text{(global)}}}{\lambda +\mu }}\right)\right]{\text{s}}_1-{\delta }^{\left(1\right)}\le 0$$ (61)

where

$$C_{\lambda ,1}≔\frac{1}{2}\left(\mu +\lambda \right)-\frac{\beta }{n_1}-\frac{2p_1\lambda \left(\lambda +\mu \right)}{\mu }.$$

By (48), we have

$$C_{\lambda ,1}\ge \frac{\mu +\lambda }{2}-\frac{\mu }{4}-\frac{2p_1\lambda (\lambda +\mu )}{\mu }\ge \frac{\lambda +\mu }{4}-\frac{2p_1\lambda (\lambda +\mu )}{\mu }=\frac{\lambda +\mu }{4}\cdot \left(1-\frac{8p_1\lambda }{\mu }\right)\ge \frac{\lambda +\mu }{8}.$$

In particular, $C_{\lambda ,1}>0$, and thus we can solve the quadratic inequality (61) (similar to the proof of Lemma 27) to get

$${\mathbf{s}}_1\le \frac{2\sqrt{2\beta \Vert \ell {\Vert }_{\infty }}}{C_{\lambda ,1}{n}_1}+\frac{{\zeta }^{\left(1\right)}+\frac{\lambda +\mu }{\mu }\left(2{\zeta }^{\left(\text{global }\right)}+4p_1\lambda {\epsilon }^{\left(1\right)}+\sqrt{\left.\frac{2\lambda \mu {\delta }^{\left(\text{global }\right)}}{\lambda +\mu }\right)}\right.}{C_{\lambda ,1}}+\sqrt{\frac{{\delta }^{\left(1\right)}}{C_{\lambda ,1}}}.$$

Plugging in $C_{\lambda ,1}\ge (\lambda +\mu )/8$ to the above inequality gives the desired result. ■

We are finally ready to present a proof of Lemma 24:

Proof [Proof of Lemma 24] Invoking Lemma 26, we have

$$\begin{gathered} {\gamma }_i\le \sqrt{2\beta \Vert \ell {\Vert }_{\infty }}\cdot \left(\frac{16\sqrt{2\beta \Vert \ell {\Vert }_{\infty }}}{n_i\left(\mu +\lambda \right)}+{ℰ}_{\lambda ,i}\right)+\frac{\beta }{2}{\left(\frac{16\sqrt{2\beta \Vert \ell {\Vert }_{\infty }}}{n_i\left(\mu +\lambda \right)}+{ℰ}_{\lambda ,i}\right)}^2 \\ \le \frac{32\beta \Vert \ell {\Vert }_{\infty }}{n_i\left(\mu +\lambda \right)}\cdot \left(1+\frac{\beta }{n_i\left(\mu +\lambda \right)}\right)+\sqrt{2\beta \Vert \ell {\Vert }_{\infty }}\cdot {ℰ}_{\lambda ,i}+\beta {ℰ}_{\lambda ,i}^2, \end{gathered}$$

where in the last line we have used $(a+b{)}^2\le 2a^2+2b^2$. We finish the proof by noting that $\frac{\beta }{n_i(\mu +\lambda )}\le \frac{\beta }{n_i\mu }\le 4$, where the last inequality is by (48). ■

C.4. Proof of Theorem 12

Compared to the proof of Theorem 11, we need to additionally control the estimation error of the global model.

Proposition 29 (Estimation error of the global model) Let Assumptions A(b) and B(b) hold. Then

$${\mathbb{E}}_{\mathit{S}}{∥{\tilde{\mathit{w}}}^{\text{(global) }}-{\mathit{w}}_{\mathbf{p}}^{\left(\text{global }\right)}∥}^2\le \frac{48{\beta }^2{\sigma }^2}{{\mu }^2{\lambda }^2}{\left(\sum _{i\in \left[m\right]}\frac{p_i}{\sqrt{n_i}}\right)}^2+\frac{48{\beta }^2{R}^2}{{\mu }^2}+\frac{12(\mu +\lambda {)}^2{\sigma }^2}{{\mu }^2{\lambda }^2}\sum _{i\in \left[m\right]}\frac{p_i^2}{n_i}.$$

Proof See Appendix C.4.1 ■

With the above proposition, the following result is a counterpart of Proposition 23.

Proposition 30 ($\mathit{\lambda }$-dependent bound on the IER) Let Assumptions A(a, b), B(b) and Equation (48) hold. Run ${𝒜}_{\text{FP}}$ with hyperparameters chosen as in Lemma 14 and 15. Then, for any $i\in \left[m\right]$, as long as

$$T \ge C_1\lambda (\lambda \vee 1)m\Vert \mathbf{p}{\Vert }^2{n}_i\cdot \left(p_i^{-1}\vee \left[\lambda \left(\lambda \vee 1\right)n_i\right]\right), K_T \ge C_2(\lambda +1{)}^2{n}_i\left(p_i^{-1}\vee {\lambda }^2{p}_i^2{n}_i\right),$$ (62)

the algorithm ${𝒜}_{\text{FP}}$ satisfies both

$${\mathbb{E}}_{{𝒜}_{\text{FP}},\mathit{S}}\left[{\text{IER}}_i\left({𝒜}_{\text{FP}}\right)\right]\left.\le \frac{C}{\lambda n_i}\left[\beta \Vert \ell {\Vert }_{\infty }+\frac{{\sigma }^2{\beta }^2{n}_i}{{\mu }^2}{\left(\sum _{i\in \left[m\right]}\frac{p_i}{\sqrt{n_i}}\right)}^2+{\sigma }^2{n}_i\sum _{i\in \left[m\right]}\frac{p_i^2}{n_i}\right]+C\lambda \left[\left(1+\frac{{\beta }^2}{{\mu }^2}\right)R^2+\frac{{\sigma }^2}{{\mu }^2}\sum _{i\in \left[m\right]}\frac{p_i^2}{n_i}\right)\right],$$ (63)

and

$${\mathbb{E}}_{{𝒜}_{\text{FP}},S}\left[{\text{IER}}_i\left({𝒜}_{\text{FP}}\right)\right]\le C\left(\frac{\beta \Vert \ell {\Vert }_{\infty }}{\mu n_i}+\lambda D^2\right),$$ (64)

where $C_1,C_2$ are two constants only depending on $\left(\mu ,\beta ,\Vert \ell {\Vert }_{\infty },D\right)$, and $C$ is an absolute constant.

Proof Without loss of generality we consider the first client. Our assumptions allow us to invoke Propositions 10 and 22 to get

$${\mathbb{E}}_{{𝒜}_{\text{FP}},\mathit{S}}\left[{\text{IER}}_1\right] \le {\mathbb{E}}_{{𝒜}_{\text{FP}},\mathit{S}}\left[\frac{{ℰ}_{\text{OPT}}}{p_1}+\frac{3\lambda }{2}{\left({\epsilon }^{\left(\text{global}\right)}\right)}^2+\frac{3\lambda }{2}{∥{\tilde{\mathit{w}}}^{\text{(global)}}-{\mathit{w}}_{\mathbf{p}}^{\left(\text{global}\right)}∥}^2+\frac{3\lambda }{2}{R}^2+\frac{320\beta \Vert \ell {\Vert }_{\infty }}{n_1(\mu +\lambda )}\right]$$

We first show that the expected value of ${ℰ}_{\text{OPT}}/p_1$ and $\lambda {\left({\epsilon }^{\text{(global)}}\right)}^2$ are both bounded above by a constant multiple of $\frac{\beta \Vert \ell {\Vert }_{\infty }}{n_1(\mu +\lambda )}$. Indeed, by the estimates we have established in Equations (50) and (52), it suffices to require

$${T\gtrsim }_{\left(\mu ,\beta ,\Vert \mu {\Vert }_{\infty },D\right)}\lambda \left(\lambda \vee 1\right)m\left|\right|P\left|\right|2_{n1},$$

and

$${K_T\gtrsim }_{\left(\mu ,\beta ,\Vert \mu {\Vert }_{\infty },D\right)}\frac{(\lambda \vee 1{)}^2{n}_1}{p_1},{T\gtrsim }_{\left(\mu ,\beta ,\Vert \mu {\Vert }_{\infty },D\right)}\frac{\lambda \left(\lambda \vee 1\right)m\Vert \mathbf{p}{\Vert }^2{n}_1}{p_1},$$

respectively. And the above two displays, combined with (49), is exactly (62). (64) then follows from the compactness of $𝒲$. To prove (63), we invoke Proposition 29 to get

$$\begin{gathered} {\mathbb{E}}_{{𝒜}_{\text{FP}},S}\left[{\text{IER}}_1\right]\lesssim \frac{\beta \Vert \ell {\Vert }_{\infty }}{n_1\left(\mu +\lambda \right)}+\lambda \left(1+\frac{{\beta }^2}{{\mu }^2}\right)R^2+\frac{{\beta }^2{\sigma }^2}{{\mu }^2\lambda }{\left(\sum _{i\in \left[m\right]}\frac{p_i}{\sqrt{n_i}}\right)}^2+\frac{(\mu +\lambda {)}^2{\sigma }^2}{{\mu }^2\lambda }\sum _{i\in \left[m\right]}\frac{p_i^2}{n_i} \\ \lesssim \frac{\beta \Vert \ell {\Vert }_{\infty }}{n_1\lambda }+\lambda \left(1+\frac{{\beta }^2}{{\mu }^2}\right)R^2+\frac{{\beta }^2{\sigma }^2}{{\mu }^2\lambda }{\left(\sum _{i\in \left[m\right]}\frac{p_i}{\sqrt{n_i}}\right)}^2+\left(\frac{{\sigma }^2}{\lambda }+\frac{\lambda {\sigma }^2}{{\mu }^2}\right)\sum _{i\in \left[m\right]}\frac{p_i^2}{n_i}, \end{gathered}$$

and (64) follows by rearranging terms. ■

We now present our proof of Theorem 12

Proof [Proof of Theorem 12| Without loss of generality we consider the first client. Since all $n_i\prime$s are of the same order, it suffices to show

$$\mathbb{E}\left[{\text{IER}}_i\left({𝒜}_{\text{FP}}\right)\right]\lesssim \left[\left(\mu +{\mu }^{-1}\right)\left(\beta \Vert \ell {\Vert }_{\infty }+\frac{{\sigma }^2{\beta }^2+{\beta }^2+{\sigma }^2}{{\mu }^2}\right)+\mu D^2\right]\cdot \left(\frac{R}{\sqrt{N/m}}\wedge \frac{1}{N/m}+\frac{\sqrt{m}}{N}\right)$$ (65)

We define the following two events:

$$\text{A}≔\left\{R\ge \sqrt{\frac{m}{N}}\right\},\text{ B}≔{\text{A}}^c=\left\{R<\sqrt{\frac{m}{N}}\right\},$$

and we set

$$\lambda =\frac{c_{\text{A}}m}{D^{2}N}\cdot {\mathbb{1}}_{\text{A}}+c_{\text{B}}\sqrt{\frac{m}{R^{2}N+1}}\cdot {\mathbb{1}}_{\text{B}},$$

where $c_{\text{A}},c_{\text{B}}$ are two constants to be specified later. We consider two cases:

1. If A holds, then from (64) we have ${\mathbb{E}}_{{𝒜}_{\text{FP}},\mathit{S}}\left[{\text{IER}}_1\right]\lesssim \left(\frac{\beta \Vert \ell {\Vert }_{\infty }}{\mu }+c_{\text{A}}\right)\cdot \frac{1}{N/m}$, provided $\lambda p_{\text{max}}\le \mu /16$. Note that $\lambda p_{\text{max}}\asymp \frac{c_{\text{A}}}{D^{2}N}\le c_{\text{A}}/D^2$. So we can choose $\lambda \asymp \mu D^2$, which gives

   $${\mathbb{E}}_{{𝒜}_{\text{FP}},S}\left[{\text{IER}}_1\right]\lesssim \left(\frac{\beta \Vert \ell {\Vert }_{\infty }}{\mu }+\mu D^2\right)\cdot \frac{1}{N/m}\le \text{right-hand side of }\left(65\right).$$
2. If B holds, and if $\lambda p_{\text{max}}\le \mu /16$ holds, then from (63) we have

   $$\begin{gathered} {\mathbb{E}}_{{𝒜}_{\text{FP}},S}\left[{\text{IER}}_1\right]\lesssim \frac{1}{\lambda N/m}\left(\beta \Vert \ell {\Vert }_{\infty }+\frac{{\sigma }^2{\beta }^2}{{\mu }^2}+{\sigma }^2\right)+\lambda \left[\left(1+\frac{{\beta }^2}{{\mu }^2}\right)R^2+\frac{{\sigma }^2}{{\mu }^{2}N}\right] \\ \lesssim \left(\beta \Vert \ell {\Vert }_{\infty }+\frac{{\sigma }^2{\beta }^2+{\beta }^2+{\sigma }^2}{{\mu }^2}\right)\cdot \left(\frac{1}{\lambda N/m}+\lambda \left(R^2+N^{-1}\right)\right) \\ =\left(\beta \Vert \ell {\Vert }_{\infty }+\frac{{\sigma }^2{\beta }^2+{\beta }^2+{\sigma }^2}{{\mu }^2}\right)\cdot \left(\frac{\sqrt{R^{2}N+1}}{c_{\text{B}}N/\sqrt{m}}+\frac{c_{\text{B}}}{N}\sqrt{m\left(R^{2}N+1\right)}\right) \\ \le \left(\beta \Vert \ell {\Vert }_{\infty }+\frac{{\sigma }^2{\beta }^2+{\beta }^2+{\sigma }^2}{{\mu }^2}\right)\cdot \left(\frac{R}{c_{\text{B}}\sqrt{N/m}}+\frac{1}{c_{\text{B}}N/\sqrt{m}}+\frac{c_{\text{B}}\sqrt{m}R}{\sqrt{N}}+\frac{c_{\text{B}}\sqrt{m}}{N}\right) \\ =\left(\beta \Vert \ell {\Vert }_{\infty }+\frac{{\sigma }^2{\beta }^2+{\beta }^2+{\sigma }^2}{{\mu }^2}\right)\cdot \left(c_{\text{B}}+c_{\text{B}}^{-1}\right)\left(\frac{R}{\sqrt{N/m}}+\frac{\sqrt{m}}{N}\right). \end{gathered}$$

Note that $p_{\text{max}}\lambda \le c_{\text{B}}{p}_{\text{max}}\sqrt{m}\asymp c_{\text{B}}/\sqrt{m}\le c_{\text{B}}$. So to satisfy $p_{\text{max}}\lambda \le \mu /16$, we can choose $c_{\text{B}}\asymp \mu$. This gives

$$\begin{gathered} {\mathbb{E}}_{{𝒜}_{\text{FP}},\mathit{S}}\left[{\text{IER}}_1\right] \lesssim \left(\beta \Vert \ell {\Vert }_{\infty }+\frac{{\sigma }^2{\beta }^2+{\beta }^2+{\sigma }^2}{{\mu }^2}\right)\left(\mu +{\mu }^{-1}\right)\cdot \left(\frac{R}{\sqrt{\frac{N}{m}}}+\frac{\sqrt{m}}{N}\right) \\ \le \text{right-hand side of }\left(65\right). \end{gathered}$$

The desired result follows by combining the above two cases together. ■

C.4.1. Proof of Proposition 29: Estimation Error of the Global Model

We begin by proving a useful lemma.

Lemma 31 (Estimating ${\mathit{w}}_{\star }^{\left(\mathit{i}\right)}$ given the knowledge of ${\mathit{w}}_{\mathbf{p}}^{\left(\text{global}\right)}$) Let Assumption A(b) hold. Then for any $i\in \left[m\right]$, we have

$$∥{\mathit{w}}_{\star }^{\left(i\right)}-{\text{Prox}}_{L_i/\lambda }\left({\mathit{w}}_{\mathbf{p}}^{\left(\text{global }\right)}\right)∥\le \frac{2}{\mu +\lambda }∥\nabla L_i\left({\mathit{w}}_{\star }^{\left(i\right)},S_i\right)+\lambda \left({\mathit{w}}_{\star }^{\left(i\right)}-{\mathit{w}}_{\mathbf{p}}^{\left(\text{global }\right)}\right)∥.$$

Proof This follows from an adaptation of the arguments in Theorem 7 of Foster et al. (2019). By strong convexity, we have

$$\begin{gathered} ⟨\nabla L_i\left({\mathit{w}}_{\star }^{\left(i\right)},S_i\right)+\lambda \left({\mathit{w}}_{\star }^{\left(i\right)}-{\mathit{w}}_{\mathbf{p}}^{\left(\text{global }\right)}\right),{\text{Prox}}_{L_i/\lambda }\left({\mathit{w}}_{\mathbf{p}}^{\left(\text{global }\right)}\right)-{\mathit{w}}_{\star }^{\left(i\right)}⟩+\frac{\mu +\lambda }{2}{∥{\mathit{w}}_{\star }^{\left(i\right)}-{\text{Prox}}_{L_i/\lambda }\left({\mathit{w}}_{\mathbf{p}}^{\left(\text{global }\right)}\right)∥}^2 \\ \le L_i\left({\mathit{w}}_{\star }^{\left(i\right)},S_i\right)+\frac{\lambda }{2}{∥{\mathit{w}}^{\left(\text{global }\right)}-{\mathit{w}}_{\star }^{\left(i\right)}∥}^2-L_i\left({\text{Prox}}_{\frac{L_i}{\lambda }}\left({\mathit{w}}_{\mathbf{p}}^{\left(\text{global }\right)}\right),S_i\right)-\frac{\lambda }{2}{∥{\mathit{w}}_{\mathbf{p}}^{\left(\text{global }\right)}-{\text{Prox}}_{\frac{L_i}{\lambda }}\left({\mathit{w}}_{\mathbf{p}}^{\left(\text{global }\right)}\right)∥}^2 \\ \le 0. \end{gathered}$$

If $∥{\mathit{w}}^{\left(i\right)}-{\text{Prox}}_{L_i/\lambda }\left({\mathit{w}}_{\mathbf{p}}^{\left(\text{global}\right)}\right)∥=0$ we are done. Otherwise, Cauchy-Schwartz inequality applied to the above display gives the desired result. ■

Now, since ${\sum }_{i\in \left[m\right]}{p}_i{F}_i$ is ${\mu }_F=\mu \lambda /(\mu +\lambda )$-strongly convex, we have

$$\begin{gathered} ⟨\sum _{i\in \left[m\right]}{p}_i\nabla F_i\left({\mathit{w}}_{\mathbf{p}}^{\left(\text{global }\right)}\right),{\tilde{\mathit{w}}}^{\left(\text{global }\right)}-{\mathit{w}}_{\mathbf{p}}^{\left(\text{global }\right)}⟩+\frac{{\mu }_F}{2}{∥{\tilde{\mathit{w}}}^{\left(\text{global }\right)}-{\mathit{w}}_{\mathbf{p}}^{\left(\text{global }\right)}∥}^2 \\ \le \sum _{i\in \left[m\right]}{p}_i{F}_i\left({\tilde{\mathit{w}}}^{\text{(global) })}-\sum _{i\in \left[m\right]}{p}_i{F}_i\left({\mathit{w}}_{\mathbf{p}}^{\left(\text{global }\right)}\right)\right. \\ \le 0 \end{gathered}$$

If $∥{\tilde{\mathit{w}}}^{\text{(global)}}-{\mathit{w}}_{\mathbf{p}}^{\text{(global)}}∥=0$ we are done. Otherwise, by Cauchy-Schwartz inequality, we get

$$\begin{gathered} ∥{\tilde{\mathit{w}}}^{\left(\text{global }\right)}-{\mathit{w}}_{\mathbf{p}}^{\left(\text{global }\right)}∥ \\ \le \frac{2}{{\mu }_F}∥\sum _{i\in \left[m\right]}{p}_i\nabla F_i\left({\mathit{w}}_{\mathbf{p}}^{\left(\text{global }\right)}\right)∥ \\ =\frac{2}{{\mu }_F}∥\sum _{i\in \left[m\right]}{p}_i\nabla L_i\left({\text{Prox}}_{\frac{L_i}{\lambda }}\left({\mathit{w}}_{\mathbf{p}}^{\left(\text{global }\right)}\right),S_i\right)∥ \\ \le \frac{2}{{\mu }_F}∥\sum _{i\in \left[m\right]}{p}_i\left(\nabla L_i\left({\text{Prox}}_{\frac{L_i}{\lambda }}\left({\mathit{w}}_{\mathbf{p}}^{\left(\text{global }\right)}\right),S_i\right)-\nabla L_i\left({\mathit{w}}_{\star }^{\left(i\right)},S_i\right)\right)∥+\frac{2}{{\mu }_F}∥\sum _{i\in \left[m\right]}{p}_i\nabla L_i\left({\mathit{w}}_{\star }^{\left(i\right)},S_i\right)∥ \\ \stackrel{\left(*\right)}{\le }\frac{2\beta }{{\mu }_F}\sum _{i\in \left[m\right]}{p}_i∥{\text{Prox}}_{\frac{L_i}{\lambda }}\left({\mathit{w}}_{\mathbf{p}}^{\left(\text{global }\right)}\right)-{\mathit{w}}_{\star }^{\left(i\right)}∥+\frac{2}{{\mu }_F}∥\sum _{i\in \left[m\right]}{p}_i\nabla L_i\left({\mathit{w}}_{\star }^{\left(i\right)},S_i\right)∥ \\ \stackrel{\left(**\right)}{\le }\frac{4\beta }{{\mu }_F\left(\mu +\lambda \right)}\sum _{i\in \left[m\right]}{p}_i∥\nabla L_i\left({\mathit{w}}_{\star }^{\left(i\right)},S_i\right)+\lambda \left({\mathit{w}}_{\star }^{\left(i\right)}-{\mathit{w}}_{\mathbf{p}}^{\left(\text{global }\right)}\right)∥+\frac{2}{{\mu }_F}∥\sum _{i\in \left[m\right]}{p}_i\nabla L_i\left({\mathit{w}}_{\star }^{\left(i\right)},S_i\right)∥ \\ \le \frac{4\beta }{\mu \lambda }\sum _{i\in \left[m\right]}{p}_i∥\nabla L_i\left({\mathit{w}}_{\star }^{\left(i\right)},S_i\right)∥+\frac{4\beta R}{\mu }+\frac{2(\mu +\lambda )}{\mu \lambda }∥\sum _{i\in \left[m\right]}{p}_i\nabla L_i\left({\mathit{w}}_{\star }^{\left(i\right)},S_i\right)∥, \end{gathered}$$

where (*) is by smoothness of $L_i$ and (**) is by Lemma 31. Thus, we have

$${∥{\tilde{\mathit{w}}}^{\text{(global)}}-{\mathit{w}}_{\mathbf{p}}^{\text{(global)}}∥}^2 \le \frac{48{\beta }^2}{{\mu }^2{\lambda }^2}{\left(\sum _{i\in \left[m\right]}{p}_i∥\nabla L_i\left({\mathit{w}}_{\star }^{\left(i\right)},S_i\right)∥\right)}^2+\frac{48{\beta }^2{R}^2}{{\mu }^2}+\frac{12(\mu +\lambda {)}^2}{{\mu }^2{\lambda }^2}{∥\sum _{i\in \left[m\right]}{p}_i\nabla L_i\left({\mathit{w}}_{\star }^{\left(i\right)},S_i\right)∥}^2$$ (66)

Note that

$${\left(\sum _{i\in \left[m\right]}{p}_i∥\nabla L_i\left({\mathit{w}}_{\star }^{\left(i\right)},S_i\right)∥\right)}^2=\sum _{i\in \left[m\right]}{p}_i^2{∥\nabla L_i\left({\mathit{w}}_{\star }^{\left(i\right)},S_i\right)∥}^2+\sum _{i\ne s}{p}_i{p}_s∥\nabla L_i\left({\mathit{w}}_{\star }^{\left(i\right)},S_i\right)∥∥\nabla L_s\left({\mathit{w}}_{\star }^{\left(s\right)},S_s\right)∥.$$

Taking expectation at both sides, we arrive at

$${\mathbb{E}}_{\mathit{S}}\left[{\left(\sum _{i\in \left[m\right]}{p}_i∥\nabla L_i\left({\mathit{w}}_{\star }^{\left(i\right)},S_i\right)∥\right)}^2\right]\le \sum _{i\in \left[m\right]}\frac{p_i^2{\sigma }^2}{n_i}+\sum _{i\ne s}\frac{p_i{p}_s{\sigma }_i{\sigma }_s}{\sqrt{n_i{n}_s}}={\sigma }^2{\left(\sum _{i\in \left[m\right]}\frac{p_i}{\sqrt{n_i}}\right)}^2.$$

Meanwhile, we have

$${\mathbb{E}}_{\mathit{S}}{∥\sum _{i\in \left[m\right]}{p}_i\nabla L_i\left({\mathit{w}}_{\star }^{\left(i\right)},S_i\right)∥}^2\le \sum _{i\in \left[m\right]}\frac{p_i^2{\sigma }^2}{n_i}.$$

The desired result follows by plugging the previous two displays to (66).

D. Details on Experiments

In each round (among 100 rounds) of simulation, we first generate ${\mathit{w}}_{\star }\in {\mathbb{R}}^{100}$ with i.i.d. standard Gaussian entries, and we set each local model ${\mathit{w}}_{\star }^{\left(i\right)}={\mathit{w}}_{\star }+R\cdot {\mathit{v}}_i$, where ${\mathit{v}}_i\in {\mathbb{R}}^{100}$ is a random unit vector that has negative correlation with ${\mathit{w}}_{\star }$ and we vary $R$ from 0 to 20. The dataset for the $i$-th client is then generated by a logistic regression model. We apply FedAvg (Algorithm 1), PurelocalTraining, and FedAvg followed by fine tuning, as well as FedProx (Algorithm 2) to this collection of datasets.

For FedAvg, we assume full participation (i.e., ${𝒞}_t=\left[m\right]$) and we set the number of communication rounds $T=20$ and global step size ${\eta }_t=0.8$. In its local training stage, we run SGD for 5 epochs with step size 0.2. For PureLocalTraining, we run SGD with step size 0.2 for 20 ⋅ 5=100 epochs. For the fine tuning strategy, we first run FedAvg (with the same hyperparameter as the previous case) and then for each client run SGD for 15 epochs with step size 0.2. For FedProx, we again assume full participation, and we set the number of communication rounds $T=20$, global step size ${\eta }_t^{\text{global}}=0.8$, local rounds $K_t=5$, and local step size ${\eta }_{t,k}^{\left(i\right)}=0.2$. In all the experiments, the batch size is set to 16.