Hybrid-DCA: A double asynchronous approach for stochastic dual coordinate ascent

Abstract

In prior works, stochastic dual coordinate ascent (SDCA) has been parallelized in a multi-core environment where the cores communicate through shared memory, or in a multi-processor distributed memory environment where the processors communicate through message passing. In this paper, we propose a hybrid SDCA framework for multi-core clusters, the most common high performance computing environment that consists of multiple nodes each having multiple cores and its own shared memory. We distribute data across nodes where each node solves a local problem in an asynchronous parallel fashion on its cores, and then the local updates are aggregated via an asynchronous across-node update scheme. The proposed double asynchronous method converges to a global solution for L-Lipschitz continuous loss functions, and at a linear convergence rate if a smooth convex loss function is used. Extensive empirical comparison has shown that our algorithm scales better than the best known shared-memory methods and runs faster than previous distributed-memory methods. Big datasets, such as one of 280 GB from the LIBSVM repository, cannot be accommodated on a single node and hence cannot be solved by a parallel algorithm. For such a dataset, our hybrid algorithm takes less than 30 seconds to achieve a duality gap of 10⁻⁵ on 16 nodes each using 12 cores, which is significantly faster than the best known distributed algorithms, such as CoCoA+, that take more than 160 seconds on 16 nodes.

1. Introduction

The immense growth of data has made it important to efficiently solve large scale machine learning problems. It is necessary to take advantage of modern high performance computing (HPC) environments such as multi-core settings where the cores communicate through shared memory, or multi-processor distributed memory settings where the processors communicate by passing messages. In particular, a large class of supervised learning formulations, including support vector machines (SVMs), logistic regression, ridge regression and many others, solve the following generic regularized risk minimization (RRM) problem: given a set of instance-label pairs of data points (x_i,y_i), i = 1, …, n,

$$\underset{\mathbf{w}\in {ℝ}^d}{\min }P(\mathbf{w}):=\frac{1}{n}\sum _{i=1}^n\varphi ({\mathbf{x}}_i^{⊺}\mathbf{w};y_i)+\frac{\text{λ}}{2}\text{g}(\mathbf{w}),$$ (1)

where $y_i\in \mathbb{R}$ is the label for the data point ${\mathbf{x}}_i\in {\mathbb{R}}^d$, $\mathbf{w}\in {\mathbb{R}}^d$ is the linear predictor to be optimized, ϕ is a loss function that is convex with respect to its first argument, λ is a regularization parameter that balances between the loss and a regularizer g(w), which for instance can take the squared ℓ₂-norm ${\parallel \mathbf{w}\parallel }_2^2$.

Many efficient sequential algorithms have been developed in the past decades to solve (1), e.g., stochastic gradient descent (SGD) [25], or alternating direction method of multipliers (ADMM) [2]. Especially, (stochastic) dual coordinate ascent (DCA) algorithm [18] has been one of the most widely used algorithms for solving (1). It efficiently optimizes the following dual formulation (2)

$$\underset{\mathit{\alpha }\in {ℝ}^n}{\max }D(\mathit{\alpha }):=-\frac{1}{n}\sum _{i=1}^n{\varphi }^{*}(-{\alpha }_i)-\frac{\lambda }{2}{g}^{*}\left(\frac{1}{\lambda n}\mathbf{X}\mathit{\alpha }\right),$$ (2)

where $\mathbf{X}=[{\mathbf{x}}_1,{\mathbf{x}}_2,\dots ,{\mathbf{x}}_n]\in {\mathbb{R}}^{d\times n}$, ϕ*(u) and g*(v) are the convex conjugates of ϕ(z;y) (or in short ϕ(z) where the scalar z = x^Tw) and g(w), respectively. The conjugate of the loss function ϕ(z) is defined as ϕ*(u) = max_z(zu − ϕ(z)). Let ∇g*(v) be the gradient of g* with respect to v where $\mathbf{v}(\alpha )=\frac{1}{\lambda n}\mathbf{X}\alpha$. We have

$$\mathbf{w}(\mathit{\alpha })=\nabla g^{*}(\mathbf{v}).$$ (3)

It is known from duality theory that if α* is an optimal dual solution, then the vector w* = w(α*) is an optimal primal solution and P(w*) = D(α*). The dual objective has a separate dual variable α_i associated with each training data point x_i. The stochastic DCA updates dual variables, one at a time, while maintaining the primal variables by calculating (3) from the dual variables.

Recently, many efforts have been undertaken to solve Problem (1) in a distributed or parallel framework. It has been shown that distributed DCA algorithms have comparable and sometimes even better convergence than SGD-based or ADMM-based distributed algorithms [23]. The distributed DCA algorithms can be grouped into two sets. The first set contains synchronous algorithms in which a random dual variable is updated by each processor and the primal variables are synchronized across the processors in every iteration [8,11,23]. This approach incurs a large communication overhead. The second set of algorithms avoids communication overhead by exploiting the shared memory in a multi-core setting [7] where the primal variables are stored in a primary memory shared across all the processors. Further speedups have been obtained by using (asynchronous) atomic memory operations instead of costly locks for shared memory updates [7,16]. Nevertheless, this approach is difficult to scale up for big datasets that cannot be fully accommodated in the shared memory. This leads to a challenging question: how do we scale up the asynchronous shared memory approach for big data while maintaining the speed up?

We address this challenge by proposing and implementing a hybrid strategy. The modern HPC platforms can be viewed as a collection of K nodes interconnected through a network as shown in Fig. 1(a). Each node contains a memory shared among R processing cores. Our strategy exploits this architecture by equally distributing the data across the local shared memory of the K nodes. Each of the R cores within a node runs a computing thread that asynchronously updates a random dual variable from those associated with the data allocated to the node. Each node also runs a communicating thread. One of the communicating threads is designated as a master and the rest are workers. After every round of H local iterations in each computing thread, each worker thread sends the local update to the master. After accumulating the local updates from S of the K workers, the master broadcasts the global update to the contributing workers. However, to avoid a slower worker falling back too far, the master ensures that in every Γ consecutive global updates there is at least one local update from each worker. Fig. 1(b) shows how our scheme is a generalization of the existing approaches: for K = 1, our setup coincides with the shared memory multi-core setting [7] and for R = 1, S = K our setup coincides with the synchronous algorithms in distributed memory setting [8,11,23]. With a proper adjustment of the parameters H, S, Γ our strategy could balance the computation time of the first setting with the communication time of the second one, while ensuring scalability in big data applications.

Fig. 1.

(a) A simplified view of the modern HPC system and (b) Algorithms on this architecture.

Thus, our contributions are (1) we propose and analyze a hybrid asynchronous shared memory and asynchronous distributed memory implementation (Hybrid-DCA) of the mostly used DCA algorithm to solve (1); (2) we prove a strong guarantee of convergence for L-Lipschitz continuous loss functions, and further linear convergence when a smooth convex loss function is used; and (3) the experimental results using our light-weight OpenMP+MPI implementation show that our algorithms are much faster than existing distributed memory algorithms [8,11], and easily scale up with the volume of data in comparison with the shared memory based algorithms [7] as the shared memory size is limited.

2. Related work

Sequential Algorithms.

SGD is the oldest and simplest method for solving problem (1). Though SGD is easy to implement and converges to modest accuracy quickly, it requires a long tail of iterations to reach ‘good’ solutions and also requires adjusting a step-size parameter. On the other hand, SDCA methods are free of learning-rate parameters and have faster convergence rate around the end [14,15]. A modified SGD has also been proposed with faster convergence by switching to SDCA after quickly reaching a modest solution [18]. Recently, ‘variance reduced’ modifications to the original SGD have also caught attention. These modifications estimate stochastic gradients with corrections to reduce the estimation variance. Mini-batch algorithms are also proposed to update several dual variables (data points) in a batch rather than a single data point per iteration[22]. Mini-batch versions of both SGD and SDCA have slower convergence when the batch size increases[17,19]. These sequential algorithms become ineffective when the datasets get bigger.

Distributed Algorithms.

In the early single communication scheme [5,12,13], a dataset is ‘decomposed’ into smaller parts that can be solved independently. The final solution is reached by ‘accumulating’ the partial solutions using a single round of communications. This method has limited utility because most datasets cannot be decomposed in such a way. Using the primal-dual relationship (3), fully distributed algorithms of DCA are later developed where each processor updates a separate α_i which is then used to update w(α), and synchronizes w across all processors (e.g., CoCoA [8]). To trade off communications vs computations, a processor can solve its subproblem with H dual updates before synchronizing the primal variable (e.g., CoCoA+ [11], DisDCA [23]). In [11,23], a more general framework is proposed in which the subproblem can be solved using not only SDCA but also any other sequential solver that can guarantee a Θ-approximation of the local solution at a processor for some Θ ∈ (0, 1]. Nevertheless, the synchronized update to the primal variables has the inherent drawback that the overall algorithm runs at a speed of the slowest processor even when there are fast processors [1].

Parallel Algorithms.

Multi-core shared memory systems have also been exploited, where the primal variables are maintained in a shared memory, removing the communication cost. However, updates to shared memory requires synchronization primitives, such as locks, which again slows down computation. Recent methods [7,10] avoid locks by exploiting (asynchronous) atomic memory updates in modern memory systems. There is even a wild version in [7] that takes arbitrarily one of the simultaneous updates. Though the shared memory algorithms are faster than the distributed versions, they have an inherent drawback of being not scalable, as there can be only a few cores in a processor board.

Other Distributed Methods for RRM.

Besides distributed DCA methods, there are several recent distributed versions of other algorithms with faster convergence, including distributed Newton-type methods (DISCO [28], DANE [20]) and distributed stochastic variance reduced gradient method (DSVRG [9]). It has been shown that they can achieve the same accurate solution using fewer rounds of communication, however, with additional computational overhead. In particular, DISCO and DANE need to solve a linear system in each round, which could be very expensive for higher dimensions. DSVRG requires each machine to load and store a second subset of the data sampled from the original training data, which also increases its running time.

The ADMM [2] and quasi-Newton methods such as L-BFGS also have distributed solutions. These methods have low communication cost, however, their inherent drawback of computing the full batch gradient does not give computation vs communications trade-off. In the context of consensus optimization, [26] gives an asynchronous distributed ADMM algorithm but that does not directly apply to solving (1).

To the best of our knowledge, this paper is the first to propose, implement and analyze a hybrid approach exploiting modern HPC architecture. Our approach is the amalgamation of three different ideas – (1) CoCoA+/DisDCA distributed framework, (2) asynchronous multi-core shared-memory solver [7] and (3) asynchronous distributed approach [27] – taking the best of each of them. In a sense ours is the first algorithm which asynchronously uses updates which themselves have been computed using asynchronous methods.

3. The Proposed Algorithm

At the core of our algorithm, the data are distributed across K nodes and each node, called a worker, repeatedly solves a perturbed dual formulation on its data partition and sends the local update to one of the workers additionally designated as the master which merges the local updates and sends back the accumulated global update to the workers to solve the subproblem once again, unless a global convergence is reached. Let ${\mathcal{I}}_k\subseteq \{1,2,\dots ,n\}$, k = 1, …, K denote the indices of the data and the dual variables residing on node k and $n_k=\mid {\mathcal{I}}_k\mid$. For any $\alpha \in {\mathbb{R}}^n$ let α_[k] denote the vector in ${\mathbb{R}}^n$ defined in such a way that the ith component (α_[k]i = α_i if $i\in {\mathcal{I}}_k$, 0 otherwise, so that α = ∑_k α_[k]. Let ${\mathbf{X}}_{[k]}\in {\mathbb{R}}^{d\times n}$ denote the matrix consisting of the columns of the $\mathbf{X}\in {\mathbb{R}}^{d\times n}$ indexed by ${\mathcal{I}}_k$ and replaced with zeros in all other columns, so that X = ∑_kX_[k].

Ideally, the dual problem solved by node k is (2) with X, α replaced by X_[k], α_[k], respectively, and hence is independent of other nodes. However, following the efficient practical implementation in [11,23], we let the workers communicate among them a vector $\mathbf{v}\in {\mathbb{R}}^d$, an estimate of $\mathbf{w}(\alpha )=\frac{1}{\lambda n}\mathbf{X}\alpha$ that summarizes the last known global solution α. Also following [11,23] for faster convergence, each worker in our algorithm solves the following perturbed local dual problem, which we henceforth call the subproblem:

$$\begin{gathered} \underset{{\mathit{\delta }}_{[k]}\in {ℝ}^n}{\max }{D}_k({\mathit{\delta }}_{[k]};\mathbf{v},{\mathit{\alpha }}_{[k]}):=-\frac{1}{n_k}\sum _{i\in {\mathcal{I}}_k}{\varphi }^{*}(-{\alpha }_i-{\delta }_i)-\frac{\mathrm{\lambda }}{2S}{g}^{*}(\mathbf{v}) \\ -\langle \frac{1}{n}{\mathbf{X}}_{[k]}^{⊺}\nabla g^{*}(\mathbf{v}),{\mathit{\delta }}_{[k]}\rangle -\frac{\mathrm{\lambda }\sigma }{2}{\Vert \frac{1}{\mathrm{\lambda }n}{\mathbf{X}}_{[k]}{\mathit{\delta }}_{[k]}\Vert }^2 \end{gathered}$$ (4)

where δ_[k] denotes the local (incremental) update to the dual variable δ_[k], the bounding barrier parameter S denotes the number of workers from which the updates would be merged by the master in a global iteration and the scaling parameter σ measures the difficulty of solving the given data partition (see [11,23]) and must be chosen such that

$$\sigma \ge {\sigma }_{\mathit{min}}:=\nu \underset{\mathit{\alpha }\in {ℝ}^n}{\max }\frac{{\Vert \mathbf{X}\mathit{\alpha }\Vert }^2}{{\sum }_{k=1}^K{\Vert {\mathbf{X}}_{[k]}{\mathit{\alpha }}_{[k]}\Vert }^2}$$ (5)

where the aggregation parameter $\nu \in [\frac{1}{S},1]$ is the weight given by the master to each of local updates from the contributing workers while computing the global update. Unlike the synchronous all reduce approach in [11], our asynchronous method merges the local updates from only S out of K nodes in each global update and the second term in the objective of our subproblem (4) has denominator S in stead of K. By Lemma 3.2 in [11], σ ≔ νS is a safe choice to hold condition (5).

3.1. Asynchronous updates by cores in a worker node

In each communication round, each worker k solves its subproblem using a parallel asynchronous DCA method [7] on the R cores. Let the data partition ${\mathcal{I}}_k$ stored in the shared memory be logically divided into R subparts where subpart ${\mathcal{I}}_{k,r}\subseteq {\mathcal{I}}_k$, r = 1, …, R, is exclusively used by core r. In each of the H iterations, core r chooses a random coordinate $i\in {\mathcal{I}}_{k,r}$ and updates δ_[k] in the ith unit direction by a step size ε computed using a single variable optimization problem:

$$\epsilon =\underset{\epsilon \in ℝ}{\arg \max }{D}_k(\epsilon {\mathbf{e}}_i;\mathbf{v},{\mathit{\alpha }}_{[k]}+{\mathit{\delta }}_{[k]})$$ (6)

which has a closed form solution for SVM problems [4], and a solution using an iterative solver for logistic regression problems [24]. The local updates to v are also maintained appropriately. Because the coordinates used by any two cores are randomly chosen in parallel, the corresponding updates to δ_[k] are independent of each other. Thus, there might be conflicts in the updates to v if the corresponding columns (the allocated examples in different cores) in X have nonzero values in the same row (resulting in different updates at the same element position of v). We use lock-free atomic memory updates to handle such conflicts. When all cores complete H iterations, worker k sends the accumulated update Δv from the current round to the master; waits until it receives the globally updated v from the master; and repeats for another round unless the master indicates termination.

Algorithm 1:

Hybrid-DCA: Worker k

Algorithm 2:

Hybrid-DCA: Master

3.2. Merging updates from workers by master

If the master had to wait for the updates from all the workers, it could compute the global updates only after the slowest worker finished. To avoid this problem, we use bounded barrier: in each round t, the master waits for updates from only a subset ${\mathcal{P}}_S$ of workers of size S ≤ K, and sends them back the global update ${\mathbf{v}}^{(t+1)}={\mathbf{v}}^{(t)}+\nu {\sum }_{k\in {\mathcal{P}}_S^{(t)}}\Delta {\mathbf{v}}_k^{(t)}$. However, due to this relaxation, there might be some slow workers with out-of-date v. When updates from such workers are merged by the master, it may degrade the quality of the global solution and hence may cause slow convergence or even divergence. We ensure sufficient freshness of the updates using bounded delay: the master makes sure that no worker has a stale update older than Γ rounds. This asynchronous approach has two benefits: (1) the overall progress is no more bottlenecked by the slowest processor, and (2) the total number of communications is reduced. On the flip side, convergence may get slowed down for very small S or very large Γ.

Example.

Fig. 2 shows a possible sequence of important events in a run of our algorithm on a dataset having n = 12 data points in d = 3 dimensions using K = 3 nodes each having R = 2 cores such that each core works with only $\mid {\mathcal{I}}_{k,r}\mid =2$ data points. The activities in solving the subproblem using H = 1 local iterations in a round is shown in a rectangular box. For the first subproblem, core 1 and core 2 in worker 1 randomly select dual coordinates such that the corresponding data points have nonzero entries in the dimensions {1, 3} and {1, 2, 3}, respectively. Each core first reads the entries of v corresponding to these nonzero data dimensions, and then computes the updates [0.1, 0, 0.7] and [0.15, 0.5, 0.4], respectively, and finally applies these updates to v (where v₁ is first updated to be 0.1 + 0.15 = 0.25 from both of the cores, then v₂ is updated to 0.5 from core 2 while v₃ is updated to 0.7 from core 1, and then v₃ is augmented by 0.4 from core 2 to reach 0.7 + 0.4 = 1.1). The atomic memory updates ensure that all the conflicting writes to v, such as v₁ in the first write-cycle, happen completely. At the end of H local iterations by each core, worker 1 sends Δv = [0.25, 0.5, 1.1] to the master, the responsibility of which is shared by one of the 3 nodes, but shown separately in the figure. By this time, the faster workers 2 and 3 already complete 3 rounds. As S = 2, the master takes first 2 updates from ${\mathcal{P}}_S^{(1)}={\mathcal{P}}_S^{(2)}=\{2,3\}$ and computes the global updates using ν = 1. However, as Γ = 2, the master holds back the third updates from workers 2,3 until the first update from worker 1 reaches the master. The subsequent events in the run are omitted in the figure.

Fig. 2.

Sequence of important events in an example run of Hybrid-DCA where n = 12, d = 3, K = 3, R = 2, S = 2, Γ = 2, ν = 1.

3.3. Communication cost analysis

In each communication round, the algorithms based on synchronous updates on all K nodes require 2K transmissions, each consisting of all values of v or Δv. Half of these transmissions are from the workers to the master and the rest are from the master to the workers. Whereas, our asynchronous update scheme requires 2S transmissions in each round.

4. Convergence Analysis

In this section we prove the convergence of the global solution computed by our hybrid algorithm. We prove for the case of regularizer g(w) = ‖w‖² as an example; the proof can be similarly extended to other regularizes g(w). For this special case, g*(v) = ‖v‖², ∇g*(v) = 2v, the simplified dual formulation is the following

$$\underset{\mathit{\alpha }\in {ℝ}^n}{\max }D(\mathit{\alpha }):=-\frac{1}{n}{\sum _{i=1}^n{\varphi }^{*}(-{\alpha }_i)-\frac{\mathrm{\lambda }}{2}\Vert \left(\frac{1}{\mathrm{\lambda }n}\mathbf{X}\mathit{\alpha }\right)\Vert }^2,$$ (7)

and the corresponding subproblem formulation is the following

$$\underset{{\mathit{\delta }}_{[k]}\in {ℝ}^n}{\max }{D}_k({\mathit{\delta }}_{[k]};\mathbf{v},{\mathit{\alpha }}_{[k]}):=-\frac{1}{n_k}\sum _{i\in {\mathcal{I}}_k}{\varphi }^{*}(-{\alpha }_i-{\delta }_i)-\frac{\mathrm{\lambda }}{2S}{\Vert \mathbf{v}\Vert }^2-\langle \frac{1}{2n}{\mathbf{X}}_{[k]}^{⊺}\mathbf{v},{\mathit{\delta }}_{[k]}\rangle -\frac{\mathrm{\lambda }\sigma }{2}{\Vert \frac{1}{\mathrm{\lambda }n}{\mathbf{X}}_{[k]}{\mathit{\delta }}_{[k]}\Vert }^2.$$ (8)

The analysis is divided into two parts. First we show that the solution of the subproblem computed by each node locally is indeed not far from the optimum of the subproblem. Using this result on the subproblem, we next show the convergence of the global solution. Though our proofs for the two parts are based on the works [7] and [11], respectively, we need to make significant adjustments in the proofs due to our modified framework handling two cascaded levels of asynchronous updates.

In our analysis we focus on all the events that are important for the local updates that are merged by the master in global update t. Fig. 3 shows an example where the master merges local updates from S = 2 workers.

Fig. 3.

Timeline of important events for the local updates from two workers that are merged by master at global update t.

4.1. Near optimality of the solution to the local subproblem

In this section we prove that the solution returned by the parallel asynchronous stochastic DCA solver used by each worker k in Algorithm 1 is not far from the optimal solution for the subproblem (4).

Definition 1 (Θ-approximate). For given v, α_[k], a solution δ_[k] to the subproblem (4) is said to be Θ-approximate, Θ ∈ [0, 1), if

$$\mathbb{E}\left[D_k({\mathit{\delta }}_{[k]}^{*};\mathbf{v},{\mathit{\alpha }}_{[k]})-D_k({\mathit{\delta }}_{[k]};\mathbf{v},{\mathit{\alpha }}_{[k]})\right]\le \Theta \left(D_k({\mathit{\delta }}_{[k]}^{*};\mathbf{v},{\mathit{\alpha }}_{[k]})-D_k(0;\mathbf{v},{\mathit{\alpha }}_{[k]})\right)$$ (9)

where ${\delta }_{[k]}^{\ast }$ is an optimum solution to (4).

Though our proof is based on the results in [7], the main challenge here is to tackle the following two modifications in our approach: (1) the solver here solves only a part of the dual problem and (2) the subproblem is now perturbed (Section 3). While the first modification is simply handled by considering the updates by the cores in worker k only, the second modification needs changes in each step of the proof in [7]. We give below the complete details of each of the steps.

Worker k solves subproblem (4) by applying total R × H updates, each of its R cores makes H updates. To show Θ-approximate, we need to show that sufficient progress is made between any two successive updates. However, each of the cores makes multiple atomic memory writes in an update, the updates made by different cores are interleaved and hence it is difficult to demarcate two successive updates. Nevertheless, depending upon the order cores select a data point $\in {\mathcal{I}}_k$ as in step 6 of Algorithm 1 we assign a node-level counter j for each of the total R × H updates in node k and let the index i(j) denote the data point selected for update j ∈ {1, …, RH}. Fig. 3 shows an example of local updates i(j).

In each update, a worker core computes step size ε using line 6 of Algorithm 1 and then applies ε in i(j)th axis to α_[k]. However, there are few subtle points to notice that happen due to the atomic updates. Firstly, when a core computes ε it starts with a v but by the time it reads a coordinate of v some other core might have already modified some other coordinates. So the effective v that a core uses to compute the increment ε might not be the actual v at the memory, and in fact it might not exist at all in the memory at any time. Let $\stackrel{‒}{v}$ denote the effective v that a core uses to compute ε, and let $\widehat{v}$ denote the actual v value in the memory. Fig. 4 helps readers connect the different notation and updates used in the proofs in this section.

Fig. 4.

Relationship among different approximations of α.

For all $i\in {\mathcal{I}}_k$, we have the following definitions:

$$h_i(u):=\frac{{\varphi }_i^{*}(-u)}{n{\Vert {\mathbf{x}}_i\Vert }^2}+\frac{\mathrm{\lambda }}{2}\left(\frac{1}{S}-\frac{1}{\sigma }\right)\frac{{\Vert \mathbf{w}\Vert }^2}{{\Vert {\mathbf{x}}_i\Vert }^2}$$

$${\text{prox}}_i(s):=\underset{u}{\arg \min }\frac{1}{2}{(u-s)}^2+h_i(u)$$

$$\begin{gathered} T_i(\mathbf{w},s):=\underset{u}{\arg \max }-\frac{1}{\sigma }\frac{\mathrm{\lambda }}{2}{\Vert \mathbf{w}\Vert }^2-\frac{1}{n}{\mathbf{w}}^{⊺}{\mathbf{x}}_i(u-s)-\frac{\mathrm{\lambda }}{2}\sigma {(\frac{1}{\mathrm{\lambda }n}{\mathbf{x}}_i(u-s))}^2 \\ -\frac{1}{n}{\varphi }_i^{*}(-u)-\frac{\mathrm{\lambda }}{2}(\frac{1}{K}{\Vert \mathbf{w}\Vert }^2-\frac{1}{\sigma }{\Vert \mathbf{w}\Vert }^2) \\ =\underset{u}{\arg \max }-\frac{\mathrm{\lambda }}{2}{\Vert \frac{\mathbf{w}}{\sqrt{\sigma }}+\frac{\sqrt{\sigma }}{\mathrm{\lambda }n}(u-s){\mathbf{x}}_i\Vert }^2-{\Vert {\mathbf{x}}_i\Vert }^2{h}_i(u) \\ =\underset{u}{\arg \min }\frac{1}{2}(u-(s{-\frac{\mathrm{\lambda }n{\mathbf{w}}^{⊺}{\mathbf{x}}_i}{\sigma {\Vert {\mathbf{x}}_i\Vert }^2}))}^2+h_i(u), \end{gathered}$$

where $\mathbf{w}\in {\mathbb{R}}^d$ denotes any fixed vector, $s\in \mathbb{R}$, and prox(s) denotes the proximal operator. We can see the connection of the above operator to the proximal operator: $T_i(\mathbf{w},s)={\mathrm{prox}}_i\left(s-\frac{{\mathbf{w}}^{⊺}{\mathbf{x}}_i}{\sigma {\parallel {\mathbf{x}}_i\parallel }^2}\right)$. Here both h_i(u) and T_i(w,s) were revised from [7] to satisfy the subproblem (4).

Let $\stackrel{‒}{\mathbf{X}}$ denote the normalized data matrix at k-th local atomic solver with omitted notation _[k] where each row is ${\stackrel{‒}{\mathbf{x}}}_i^{⊺}={\mathbf{x}}_i^{⊺}∕\parallel {\mathbf{x}}_i\parallel$, $i\in {\mathcal{I}}_k$. Define $M_{[k],i}={\max }_{\mathcal{D}\subseteq [d]}\parallel {\sum }_{t\in \mathcal{D}}{\stackrel{‒}{\mathbf{X}}}_{(:,t)}{X}_{(i,t)}\parallel$, M = max_k max_i M_[k],i over all the local atomic solvers, where [d] is the set of all the feature indices, and ${\stackrel{‒}{\mathbf{X}}}_{(:,t)}$ is the t-th column of $\stackrel{‒}{\mathbf{X}}$. Moreover, R_min is defined as the minimum value of global data matrix, i.e. R_min = min_i=1,…,n ‖x_i‖². Then, we define that:

Definition 2 (Local atomic dual variables). Here we omit _[k] in the proofs of the local atomic solver.

$$\begin{array}{cc}{\beta }_t^{l+1}=\{\begin{array}{ccc}{T}_t\left({\widehat{\mathbf{w}}}^l,{\beta }_c^l\right)& \hfill \text{if}\hfill & c=i(l),\\ {\beta }_c^l\hfill & \text{if}& c\ne i(l)\end{array},\hfill & {\epsilon }^l={\beta }_{i(l)}^{l+1}-{\beta }_{i(l)}^l,\\ {\tilde{\beta }}^{l+1}=T\left({\widehat{\mathbf{w}}}^l,{\mathit{\beta }}^l\right),\hfill & {\overline{\beta }}^{l+1}=T\left({\overline{\mathbf{w}}}^l,{\mathit{\beta }}^l\right),\end{array}$$

where ${\beta }^l={\alpha }_{[k]}+\nu {\delta }_{[k]}^l$ denotes the l-th sequence generated by a specific k-th local atomic solver, ${\widehat{\mathbf{w}}}^l$ denotes the actual values of w maintained at update l in the local atomic solver; i(l) indicates the index selected at l-th update; and ${\stackrel{‒}{\mathbf{w}}}^l$ refers to the “accurate” w if all cores are synchronously updated at iteration l. Note that, ${\tilde{\beta }}_{i(l)}^{l+1}={\beta }_{i(l)}^{l+1}$ and ${\tilde{\beta }}^{l+1}=\mathrm{prox}\left({\beta }^l-\frac{\mathrm{\lambda }n}{\sigma }\stackrel{‒}{\mathbf{X}}{\widehat{\mathbf{w}}}^l\right)$. Since v and α_[k] will not be changed when solving the local subproblem D_k(δ_[k]; v, α_[k]), we denote Q^σ(β^l) as the objective value of the dual subproblem at l-th update and omit the subscripts _[k].

Assumption 1 (Lipschitz Continuous). The global problem objective (2) is L_max-Lipschitz continuous and therefore, its local subproblems objective (4) are at most L_max-Lipschitz continuous.

The following propositions are cited from [7], and we use these results in our proof.

Proposition 1 (Expectation of Dual Variables).

$${\mathbb{E}}_{i(l)}\left({\Vert {\mathit{\beta }}^{l+1}-{\mathit{\beta }}^l\Vert }^2\right)=\frac{1}{n}{\Vert {\tilde{\mathit{\beta }}}^{l+1}-{\mathit{\beta }}^l\Vert }^2,$$ (10)

Proposition 2 (Boundary of Asynchronous Variables).

$$\Vert {\overline{\mathbf{X}}\overline{\mathbf{w}}}^l-{\overline{\mathbf{X}}\widehat{\mathbf{w}}}^l\Vert \le \frac{1}{\mathrm{\lambda }n}M\sum _{c=l-\gamma }^l\left|{\epsilon }^c\right|,$$ (11)

Proposition 3.

$$\left|T_i({\mathbf{w}}_1,s_1)-T_i({\mathbf{w}}_2,s_2)\right|\le \left|s_1-s2+\frac{{({\mathbf{w}}_1-{\mathbf{w}}_2)}^{⊺}{\mathbf{x}}_i}{{\Vert {\mathbf{x}}_i\Vert }^2}\right|,$$ (12)

Proposition 4. Let M ≥ 1, $q=\frac{6(\gamma +1)eM}{\sqrt{n}}$, ρ = (1 + q)², and $\theta ={\sum }_{t=1}^{\gamma }{\rho }^{1∕2}$. If q(γ + 1) ≤ 1 and σ ≥ 1, then ρ^(γ+1)/2 ≤ e, and

$${\rho }^{-1}\le 1-\frac{4}{\sqrt{n}}-\frac{4M+4M\theta }{\sqrt{n}}\le 1-\frac{4}{\sqrt{n}}-\frac{4M+4M\theta }{\sigma \sqrt{n}},$$ (13)

Proposition 5 (Properties of Dual Concave Function). For all j > 0, we have

$$Q^{\sigma }\left({\mathit{\beta }}^l\right)\le Q^{\sigma }\left({\overline{\mathit{\beta }}}^{l+1}\right)-\frac{\sigma {\Vert {\mathbf{x}}_{i(l)}\Vert }^2}{2}{\Vert {\mathit{\beta }}^l-{\overline{\mathit{\beta }}}^{l+1}\Vert }^2,$$ (14)

$$Q^{\sigma }\left({\mathit{\beta }}^l\right)\ge Q^{\sigma }\left({\overline{\mathit{\beta }}}^{l+1}\right)-\frac{L_{\mathit{max}}}{2}{\Vert {\mathit{\beta }}^l-{\overline{\mathit{\beta }}}^{l+1}\Vert }^2$$ (15)

Proof. Two properties of dual concave function are stated as follows.

• the strong convexity of Q^σ(β^l): as all conjugate functions are convex, so it is clear that Q^σ(β^l) is σ ‖x_i(l)‖²-strongly convex.

• the Lipschitz continuous gradient of Q^σ(β^l): refer to Assumption 1. □

Because of the atomic updates, the step size computation may not include all the latest updates, but we assume all the updates before the (l − γ)-th update have already been written into v.

Assumption 2 (Bounded Delay of Local Updates, γ).

$${(\gamma +1)}^2\le \frac{\sqrt{n_k}}{6eM},\mathit{where}e\mathit{is\; the\; Euler}’\mathit{s\; number}.$$ (16)

This assumption restricts the maximum allowed local delay γ by M and n_k.

Lemma 6. Under Assumption 2, Definition 2, and $\rho ={\left(1+\frac{6(\gamma +1)eM}{{\sqrt{n}}_k}\right)}^2$. Then, the local subproblem satisfies:

$$\mathbb{E}\left[{\Vert {\mathit{\beta }}_{[k]}^{l-1}-{\tilde{\mathit{\beta }}}_{[k]}^l\Vert }^2\right]\le \rho \mathbb{E}\left[{\Vert {\mathit{\beta }}_{[k]}^l-{\tilde{\mathit{\beta }}}_{[k]}^{l+1}\Vert }^2\right].$$ (17)

Not that l ≠ h, represents the lth update to ω in a local solver but not the hth iteration of one core.

Proof. We omit the subscript _[k] in the notations, which specifies the kth data partition, in the proof. We prove Eq. (17) by induction. As shown in [7], we have

$${\Vert {\mathit{\beta }}^{l-1}-{\tilde{\mathit{\beta }}}^l\Vert }^2-{\Vert {\mathit{\beta }}^l-{\tilde{\mathit{\beta }}}^{l+1}\Vert }^2\le 2\Vert {\mathit{\beta }}^{l-1}-{\tilde{\mathit{\beta }}}^l\Vert \Vert {\mathit{\beta }}^l-{\tilde{\mathit{\beta }}}^{l+1}-{\mathit{\beta }}^{l-1}+{\tilde{\mathit{\beta }}}^l\Vert .$$ (18)

The second factor in the r.h.s of Eq (18) is bounded as follows with the revisions:

$$\begin{gathered} \Vert {\beta }^l-{\tilde{\beta }}^{l+1}-{\beta }^{l-1}+{\tilde{\beta }}^l\Vert \\ \le \Vert {\mathit{\beta }}^l-{\mathit{\beta }}^{l-1}\Vert +\Vert \text{prox}\left({\mathit{\beta }}^l-\frac{\mathrm{\lambda }n}{\sigma }{\overline{\mathbf{X}}\widehat{\mathbf{w}}}^l\right)-\text{prox}\left({\mathit{\beta }}^{l-1}-\frac{\mathrm{\lambda }n}{\sigma }{\overline{\mathbf{X}}\widehat{\mathbf{w}}}^{l-1}\right)\Vert \\ \le \Vert {\mathit{\beta }}^l-{\mathit{\beta }}^{l-1}\Vert +\Vert \left({\mathit{\beta }}^l-\frac{\mathrm{\lambda }n}{\sigma }{\overline{\mathbf{X}}\widehat{\mathbf{w}}}^l\right)-\left({\mathit{\beta }}^{l-1}-\frac{\mathrm{\lambda }n}{\sigma }{\overline{\mathbf{X}}\widehat{\mathbf{w}}}^{l-1}\right)\Vert \\ \le 2\Vert {\mathit{\beta }}^l-{\mathit{\beta }}^{l-1}\Vert +\frac{\mathrm{\lambda }n}{\sigma }\Vert {\overline{\mathbf{X}}\widehat{\mathbf{w}}}^l-{\overline{\mathbf{X}}\widehat{\mathbf{w}}}^{l-1}\Vert \\ =2\Vert {\mathit{\beta }}^l-{\mathit{\beta }}^{l-1}\Vert +\frac{\mathrm{\lambda }n}{\sigma }\Vert {\overline{\mathbf{X}}\widehat{\mathbf{w}}}^l-{\overline{\mathbf{X}}\overline{\mathbf{w}}}^l+{\overline{\mathbf{X}}\overline{\mathbf{w}}}^l-{\overline{\mathbf{X}}\overline{\mathbf{w}}}^{l-1}+{\overline{\mathbf{X}}\overline{\mathbf{w}}}^{l-1}-{\overline{\mathbf{X}}\widehat{\mathbf{w}}}^{l-1}\Vert \\ \le 2\Vert {\mathit{\beta }}^l-{\mathit{\beta }}^{l-1}\Vert +\frac{\mathrm{\lambda }n}{\sigma }\left(\Vert {\overline{\mathbf{X}}\overline{\mathbf{w}}}^l-{\overline{\mathbf{X}}\overline{\mathbf{w}}}^{l-1}\Vert +\Vert {\overline{\mathbf{X}}\widehat{\mathbf{w}}}^l-{\overline{\mathbf{X}}\overline{\mathbf{w}}}^l\Vert +\Vert {\overline{\mathbf{X}}\overline{\mathbf{w}}}^{l-1}-{\overline{\mathbf{X}}\widehat{\mathbf{w}}}^{l-1}\Vert \right) \\ \le \left(2+2\frac{\mathrm{\lambda }n}{\sigma }\frac{M}{\mathrm{\lambda }n}\right)\Vert {\mathit{\beta }}^l-{\mathit{\beta }}^{l-1}\Vert +2\frac{\mathrm{\lambda }n}{\sigma }\frac{M}{\mathrm{\lambda }n}\sum _{c=l-\gamma -1}^{l-2}\left|{\epsilon }^c\right|(\text{Proposition}2) \end{gathered}$$ (19)

$$\le \left(2+2\frac{M}{\sigma }\right)\Vert {\mathit{\beta }}^l-{\mathit{\beta }}^{l-1}\Vert +2\frac{M}{\sigma }\sum _{c=l-\gamma -1}^{l-2}\left|{\epsilon }^c\right|$$ (20)

Now we start the induction. Although some steps may be the same as the steps in [7], we still keep them here to make the proof self-contained.

Induction Hypothesis.

We prove the following equivalent statement. For all j,

$$\mathbb{E}\left({\Vert {\mathit{\beta }}^{l-1}-{\tilde{\mathit{\beta }}}^l\Vert }^2\right)\le \rho \mathbb{E}\left({\Vert {\mathit{\beta }}^l-{\tilde{\mathit{\beta }}}^{l+1}\Vert }^2\right),$$

Induction Basis.

When l = 1,

$$\begin{gathered} \mathbb{E}\left({\Vert {\mathit{\beta }}^0-{\tilde{\mathit{\beta }}}^1\Vert }^2\right)-\mathbb{E}\left({\Vert {\mathit{\beta }}^1-{\tilde{\mathit{\beta }}}^2\Vert }^2\right) \\ \le 2E\left(\Vert {\mathit{\beta }}^0-{\tilde{\mathit{\beta }}}^1\Vert \Vert {\mathit{\beta }}^1-{\tilde{\mathit{\beta }}}^2-{\mathit{\beta }}^0-{\tilde{\mathit{\beta }}}^1\Vert \right) \\ \le \left(4+4\frac{M}{2}\right)\mathbb{E}(\Vert {\mathit{\beta }}^0-{\tilde{\mathit{\beta }}}^1\Vert \Vert {\mathit{\beta }}^0-{\mathit{\beta }}^1\Vert ). \end{gathered}$$

By Proposition 1 and AM–GM inequality, which for any b₁, b₂ > 0 and any c > 0, we have

$$b_1{b}_2\le \frac{1}{2}(c{b}_1^2+c^{-1}{b}_2^2)$$ (21)

Therefore, we have

$$\begin{gathered} \mathbb{E}\left(\Vert {\mathit{\beta }}^0-{\tilde{\mathit{\beta }}}^1\Vert \Vert {\mathit{\beta }}^0-{\mathit{\beta }}^1\Vert \right) \\ \le \frac{1}{2}\mathbb{E}\left(\sqrt{n}{\Vert {\mathit{\beta }}^0-{\mathit{\beta }}^1\Vert }^2+\frac{1}{\sqrt{n}}{\Vert {\mathit{\beta }}^0-{\tilde{\mathit{\beta }}}^1\Vert }^2\right) \\ =\frac{1}{2}\mathbb{E}\left(\frac{1}{\sqrt{n}}{\Vert {\mathit{\beta }}^0-{\tilde{\mathit{\beta }}}^1\Vert }^2+\frac{1}{\sqrt{n}}{\Vert {\mathit{\beta }}^0-{\tilde{\mathit{\beta }}}^1\Vert }^2\right)\left(\text{Proposition}1\right) \\ =\frac{1}{\sqrt{n}}\mathbb{E}\left({\Vert {\mathit{\beta }}^0-{\tilde{\mathit{\beta }}}^1\Vert }^2\right) \end{gathered}$$

Therefore,

$$\begin{gathered} \mathbb{E}\left({\Vert {\mathit{\beta }}^0-{\tilde{\mathit{\beta }}}^1\Vert }^2\right)-\mathbb{E}\left({\Vert {\mathit{\beta }}^1-{\tilde{\mathit{\beta }}}^2\Vert }^2\right) \\ \le \left(\frac{4}{\sqrt{n}}+\frac{4M}{\sigma \sqrt{n}}\right)\mathbb{E}\left({\Vert {\mathit{\beta }}^0-{\tilde{\mathit{\beta }}}^1\Vert }^2\right), \end{gathered}$$

which implies

$$\begin{gathered} \mathbb{E}\left({\Vert {\mathit{\beta }}^0-{\tilde{\mathit{\beta }}}^1\Vert }^2\right)\le {\left(1-\frac{4}{\sqrt{n}}-\frac{4M}{\sigma \sqrt{n}}\right)}^{-1}\mathbb{E}\left({\Vert {\mathit{\beta }}^1-{\tilde{\mathit{\beta }}}^2\Vert }^2\right) \\ \le \rho \mathbb{E}\left({\Vert {\mathit{\beta }}^1-{\tilde{\mathit{\beta }}}^2\Vert }^2\right), \end{gathered}$$

where the last inequality is based on Proposition 4 and the fact θM ≥ 1.

Induction Step.

By the induction hypothesis, we assume

$$\mathbb{E}\left({\Vert {\mathit{\beta }}^{h-1}-{\tilde{\mathit{\beta }}}^h\Vert }^2\right)\le \rho \mathbb{E}\left({\Vert {\mathit{\beta }}^h-{\tilde{\mathit{\beta }}}^{h+1}\Vert }^2\right)\forall h\le l-1.$$ (22)

To show

$$\mathbb{E}\left({\Vert {\mathit{\beta }}^{l-1}-{\tilde{\mathit{\beta }}}^l\Vert }^2\right)\le \rho \mathbb{E}\left({\Vert {\mathit{\beta }}^l-{\tilde{\mathit{\beta }}}^{l+1}\Vert }^2\right),$$

we firstly show that for all h < l,

$$\begin{gathered} \mathbb{E}\left(\Vert {\mathit{\beta }}^h-{\mathit{\beta }}^{h+1}\Vert \Vert {\mathit{\beta }}^{l-1}-{\tilde{\mathit{\beta }}}^l\Vert \right) \\ \le \frac{1}{2}\mathbb{E}\left(\sqrt{n}{\rho }^{(h+1-l)/2}{\Vert {\mathit{\beta }}^h-{\mathit{\beta }}^{h+1}\Vert }^2+\frac{1}{\sqrt{n}}{\rho }^{(l-1-h)/2}{\Vert {\mathit{\beta }}^{l-1}-{\tilde{\mathit{\beta }}}^l\Vert }^2\right)(\text{Eq.}21)) \\ =\frac{1}{2}\mathbb{E}\left(\sqrt{n}{\rho }^{(h+1-l)/2}\mathbb{E}\left({\Vert {\mathit{\beta }}^h-{\mathit{\beta }}^{h+1}\Vert }^2\right)+\frac{1}{\sqrt{n}}{\rho }^{(l-1-h)/2}{\Vert {\mathit{\beta }}^{l-1}-{\tilde{\mathit{\beta }}}^l\Vert }^2\right) \\ =\frac{1}{2}\mathbb{E}\left(\frac{1}{\sqrt{n}}{\rho }^{(h+1-l)/2}{\Vert {\mathit{\beta }}^h-{\tilde{\mathit{\beta }}}^{h+1}\Vert }^2+\frac{1}{\sqrt{n}}{\rho }^{(l-1-h)/2}{\Vert {\mathit{\beta }}^{l-1}-{\tilde{\mathit{\beta }}}^l\Vert }^2\right)(\text{Proposition}1) \\ \le \frac{1}{2}\mathbb{E}\left(\frac{1}{\sqrt{n}}{\rho }^{(h+1-l)/2}{\rho }^{l-c-1}{\Vert {\mathit{\beta }}^{l-1}-{\tilde{\mathit{\beta }}}^l\Vert }^2+\frac{1}{\sqrt{n}}{\rho }^{(l-1-h)/2}{\Vert {\mathit{\beta }}^{l-1}-{\tilde{\mathit{\beta }}}^l\Vert }^2\right)(\text{Eq.}22)) \\ \le \frac{{\rho }^{(l-1-h)/2}}{\sqrt{n}}\mathbb{E}\left({\Vert {\mathit{\beta }}^{l-1}-{\tilde{\mathit{\beta }}}^l\Vert }^2\right). \end{gathered}$$ (23)

Let $\theta ={\sum }_{h=1}^{\gamma }{\rho }^{h∕2}$. We have

$$\begin{gathered} \mathbb{E}\left(\Vert {\mathit{\beta }}^{l-1}-{\tilde{\mathit{\beta }}}^l{\Vert }^2\right)-\mathbb{E}\left(\Vert {\mathit{\beta }}^l-{\tilde{\mathit{\beta }}}^{l+1}\Vert \right)\le \mathbb{E}\left(2\Vert {\mathit{\beta }}^{l-1}-{\tilde{\mathit{\beta }}}^l\Vert \left(\right(2+2\frac{M}{\sigma })\right)\Vert {\mathit{\beta }}^{l-1}-{\mathit{\beta }}^l\Vert +2\frac{M}{\sigma }\Vert {\mathit{\beta }}^{h-1}-{\mathit{\beta }}^h\Vert \left)\right)(\mathrm{Eqs.}(18),(19)) \\ =\left(4+4\frac{M}{\sigma }\right)\mathbb{E}(\Vert {\mathit{\beta }}^{l-1}-{\tilde{\mathit{\beta }}}^l\Vert \Vert {\mathit{\beta }}^{l-1}-{\mathit{\beta }}^l\Vert )+4\frac{M}{\sigma }\sum _{c=l-\gamma -1}^{l-1}\mathbb{E}(\Vert {\mathit{\beta }}^{l-1}-{\tilde{\mathit{\beta }}}^l\Vert \Vert {\mathit{\beta }}^{c-1}-{\mathit{\beta }}^c\Vert ) \\ \le \frac{4\sigma +4M}{\sigma \sqrt{n}}\mathbb{E}(\Vert {\mathit{\beta }}^{l-1}-{\tilde{\mathit{\beta }}}^l{\Vert }^2)+\frac{4M}{\sigma \sqrt{n}}\mathbb{E}(\Vert {\mathit{\beta }}^{l-1}-{\tilde{\mathit{\beta }}}^l\Vert )\sum _{c=l-\gamma -1}^{l-2}{\rho }^{(l-1-c)/2}(\text{Eq.}(23)) \\ \le \frac{4\sigma +4M}{\sigma \sqrt{n}}\mathbb{E}(\Vert {\mathit{\beta }}^{l-1}-{\tilde{\mathit{\beta }}}^l{\Vert }^2)+\frac{4M}{\sigma \sqrt{n}}\theta \mathbb{E}(\Vert {\mathit{\beta }}^{l-1}-{\tilde{\mathit{\beta }}}^l\Vert ) \\ \le \left(\frac{4}{\sqrt{n}}+\frac{4M+4M\theta }{\sigma \sqrt{n}}\right)\mathbb{E}(\Vert {\mathit{\beta }}^{l-1}-{\tilde{\mathit{\beta }}}^l{\Vert }^2) \end{gathered}$$

which implies that

$$\begin{gathered} \mathbb{E}\left({\Vert {\mathit{\beta }}^{l-1}-{\tilde{\mathit{\beta }}}^l\Vert }^2\right)\le \frac{1}{1-\frac{4}{\sqrt{n}}-\frac{4M+4M\theta }{\sigma \sqrt{n}}}\mathbb{E}\left({\Vert {\mathit{\beta }}^l-{\tilde{\mathit{\beta }}}^{l+1}\Vert }^2\right) \\ \le \rho \mathbb{E}\left({\Vert {\mathit{\beta }}^{l-1}-{\tilde{\mathit{\beta }}}^l\Vert }^2\right) \end{gathered}$$

by Proposition 4. □

Lemma 6 implies that the asynchronous updates will not pull the solution away from the optimal solution too much even if the directions of the updates are wrong.

Definition 3 (Global Error Bound). For a convex function $f:{\mathbb{R}}^n\to \mathbb{R}$, the optimization problem: min_β f(β) admits a global error bound if there is a constant κ such that

$$\Vert \mathit{\beta }-P_S(\mathit{\beta })\Vert \le \kappa \Vert T(\mathit{\beta })-\mathit{\beta }\Vert ,$$ (24)

where P_S(⋅) is the Euclidean projection to the set of optimal solutions, and $T:{\mathbb{R}}^n\to {\mathbb{R}}^n$ is the operator defined as

$$T_i(\mathit{\beta })=\arg \underset{u}{\min }f(\mathit{\beta }+(u-{\beta }_i){\mathbf{e}}_h)\forall i\in [n].$$

The optimization problem admits a relaxed condition called global error bound from the beginning if (24) holds for any β satisfying f(β) ≤ F for some constant F.

Assumption 3. The local subproblem formulation (4) admits the global error bound from the beginning for $F=Q({\delta }_{[k]}^{(j)};{\mathbf{v}}^{(j)},{\alpha }_{[k]}^{(j)})$ and any update j.

The global error bound in the local subproblem helps prove that our subproblem solver achieves significant improvement after each update. It has been shown that when the loss functions are hinge loss or squared hinge loss, the global problem formulation (2) does indeed satisfy the global error bound condition [7]. Then, for the local subproblem (4), it still satisfy the global error bound within the subset α_[k].

Assumption 4 (Bounded M, L_max).

$$2L_{\mathit{max}}\left(1+\frac{e^2{\gamma }^2{M}^2}{{\sigma }^2{n}_k}\right)\left(\frac{e^2{\gamma }^2{M}^2}{{\sigma }^2{n}_k}\right)\le 1$$

Lemma 7 (Convergence for Subproblem). When Assumptions 2–4 hold, the solutions computed in two successive updates by the local subproblem solver has a linear convergence rate in expectation, i.e.,

$$\mathbb{E}\left[D_k({\mathit{\delta }}_{[k]}^{*})-D_k({\mathit{\delta }}_{[k]}^{(j)})\right]\le \Theta \left[D_k({\mathit{\delta }}_{[k]}^{*})-D_k({\mathit{\delta }}_{[k]}^{(j-1)})\right]$$

where ${\delta }_{[k]}^{(j)}={\delta }_{[k]}^{(j)}+{\sum }_{h=1}^H{\sum }_{r=1}^R{\mathit{\beta }}_{[k]}^{h,r}$ is the δ_[k] after the jth update,

$$\eta =1-\frac{\kappa R_{\mathit{min}}}{2n{L}_{\mathit{max}}}\left(1-\frac{2L_{\mathit{max}}}{R_{\mathit{min}}}\left(1+\frac{e^2{\gamma }^2{M}^2}{{\sigma }^2\tilde{n}}\right)\left(\frac{e^2{\gamma }^2{M}^2}{{\sigma }^2\tilde{n}}\right)\right),$$

$\tilde{n}={\max }_k{n}_k$ is the size of the largest data part, and

$$\Theta ={\eta }^{RH}.$$ (25)

Proof. We also omit the subscript _[k] of the notations in the proof. We can bound the expected distance $\mathbb{E}\left({\mid {\stackrel{‒}{\mathit{\beta }}}^{j+1}-{\tilde{\mathit{\beta }}}^{l+1}\mid }^2\right)$ by the following derivation.

$$\begin{gathered} \mathbb{E}\left({\Vert {\overline{\mathit{\beta }}}^{l+1}-{\tilde{\mathit{\beta }}}^{l+1}\Vert }^2\right)=\mathbb{E}\left(\sum _{t=1}^n{(T_t({\overline{\mathbf{w}}}^l,{\beta }_t^l)-T_t({\widehat{\mathbf{w}}}^l,{\beta }_t^l))}^2\right) \\ \le \mathbb{E}\left(\sum _{t=1}^n{\left(\frac{\mathrm{\lambda }n{({\overline{\mathbf{w}}}^l-{\widehat{\mathbf{w}}}^l)}^{⊺}{\mathbf{x}}_t}{\sigma {\Vert {\mathbf{x}}_t\Vert }^2}\right)}^2\right)(\text{Proposition}3) \\ =\frac{{\mathrm{\lambda }}^2{n}^2}{{\sigma }^2}\mathbb{E}\left({\Vert \overline{\mathbf{X}}({\overline{\mathbf{w}}}^l-{\widehat{\mathbf{w}}}^l)\Vert }^2\right) \\ \le \frac{M^2}{{\mathrm{\lambda }}^2{n}^2}\frac{{\mathrm{\lambda }}^2{n}^2}{{\sigma }^2}\mathbb{E}\left({\left(\sum _{t=l-\gamma }^{l-1}\Vert {\mathit{\beta }}^t-{\mathit{\beta }}^{t+1}\Vert \right)}^2\right)(\text{Proposition}2) \\ \le \frac{M^2}{{\sigma }^2}\mathbb{E}\left(\gamma {\left(\sum _{t=l-\gamma }^{l-1}\Vert {\mathit{\beta }}^t-{\mathit{\beta }}^{t+1}\Vert \right)}^2\right)(\text{Cauchy Schwarz Inequality}) \\ \le \frac{\gamma M^2}{{\sigma }^2}\mathbb{E}(\gamma \left(\sum _{t=1}^{\gamma }{\rho }^t{\Vert {\mathit{\beta }}^l-{\mathit{\beta }}^{l+1}\Vert }^2\right))(\text{Lemma}6) \\ \le \frac{\gamma M^2}{{\sigma }^{2}n}(\sum _{t=1}^{\gamma }{\rho }^t)\mathbb{E}\left({\Vert {\mathit{\beta }}^l-{\tilde{\mathit{\beta }}}^{l+1}\Vert }^2\right)(\text{Proposition}1) \\ \le \frac{{\gamma }^2{M}^2}{{\sigma }^{2}n}{\rho }^{\gamma }\mathbb{E}\left({\Vert {\mathit{\beta }}^l-{\tilde{\mathit{\beta }}}^{l+1}\Vert }^2\right) \\ \le \frac{{\gamma }^2{M}^2{e}^2}{{\sigma }^{2}n}\mathbb{E}\left({\Vert {\mathit{\beta }}^l-{\tilde{\mathit{\beta }}}^{l+1}\Vert }^2\right).(\text{Proposition}4) \end{gathered}$$ (26)

Moreover,

$$\begin{gathered} \mathbb{E}\left({\Vert {\overline{\mathit{\beta }}}^l-{\mathit{\beta }}^{l+1}\Vert }^2\right)=\mathbb{E}\left({\Vert {\overline{\mathit{\beta }}}^{l+1}-{\tilde{\mathit{\beta }}}^{l+1}+{\tilde{\mathit{\beta }}}^{l+1}-{\mathit{\beta }}^l\Vert }^2\right) \\ \le \mathbb{E}\left(2\left({\Vert {\overline{\mathit{\beta }}}^{l+1}-{\tilde{\mathit{\beta }}}^{l+1}\Vert }^2+{\Vert {\tilde{\mathit{\beta }}}^{l+1}-{\mathit{\beta }}^l\Vert }^2\right)\right)(\text{Cauchy-Schwarz}) \\ \le 2\left(1+\frac{{\gamma }^2{M}^2{e}^2}{{\sigma }^{2}n}\right)\mathbb{E}\left({\Vert {\tilde{\mathit{\beta }}}^{l+1}-{\mathit{\beta }}^l\Vert }^2\right) \end{gathered}$$ (27)

The bound of the increase of local objective function value by

$$\begin{gathered} \mathbb{E}(Q^{\sigma }({\mathit{\beta }}^{l+1}))-\mathbb{E}(Q^{\sigma }({\mathit{\beta }}^l))=\mathbb{E}\left(-\left(Q^{\sigma }({\mathit{\beta }}^l)-Q^{\sigma }\left({\overline{\mathit{\beta }}}^{l+1}\right)\right)\right)-\mathbb{E}\left((Q^{\sigma }\left({\overline{\mathit{\beta }}}^{l+1}\right)-Q^{\sigma }({\mathit{\beta }}^{l+1}))\right) \\ \ge \mathbb{E}\left(\frac{\sigma {\Vert {\mathbf{x}}_{i(l)}\Vert }^2}{2}{\Vert {\mathit{\beta }}^l-{\overline{\mathit{\beta }}}^{l+1}\Vert }^2\right)-\mathbb{E}\left(\frac{L_{\mathit{max}}}{2}{\Vert {\mathit{\beta }}^{l+1}-{\overline{\mathit{\beta }}}^{l+1}\Vert }^2\right)(\text{Proposition}5) \\ \ge \frac{R_{\mathit{min}}}{2n}\mathbb{E}\left({\Vert {\mathit{\beta }}^l-{\overline{\mathit{\beta }}}^{l+1}\Vert }^2\right)-\frac{L_{\mathit{max}}}{2n}\mathbb{E}\left({\Vert {\tilde{\mathit{\beta }}}^{l+1}-{\overline{\mathit{\beta }}}^{l+1}\Vert }^2\right) \\ \ge \frac{R_{\mathit{min}}}{2n}\mathbb{E}\left({\Vert {\mathit{\beta }}^l-{\overline{\mathit{\beta }}}^{l+1}\Vert }^2\right)-\frac{L_{\mathit{max}}}{2n}\frac{{\gamma }^2{M}^2{e}^2}{{\sigma }^{2}n}\mathbb{E}\left({\Vert {\tilde{\mathit{\beta }}}^{l+1}-{\overline{\mathit{\beta }}}^l\Vert }^2\right)(\text{Eq.}26) \\ \ge \frac{R_{\mathit{min}}}{2n}\mathbb{E}\left({\Vert {\mathit{\beta }}^l-{\overline{\mathit{\beta }}}^{l+1}\Vert }^2\right)-\frac{2L_{\mathit{max}}}{2n}\frac{{\gamma }^2{M}^2{e}^2}{{\sigma }^{2}n}\left(1+\frac{{\gamma }^2{M}^2{e}^2}{{\sigma }^{2}n}\right)\mathbb{E}\left({\Vert {\overline{\mathit{\beta }}}^{l+1}-{\mathit{\beta }}^l\Vert }^2\right)(\text{Eq.}(27)) \\ \ge \frac{R_{\mathit{min}}}{2n}\left(1-\frac{2L_{\mathit{max}}}{2n}\left(1+\frac{{\gamma }^2{M}^2{e}^2}{{\sigma }^{2}n}\right)\left(\frac{{\gamma }^2{M}^2{e}^2}{{\sigma }^{2}n}\right)\right)\times \mathbb{E}\left({\Vert {\overline{\mathit{\beta }}}^{l+1}-{\mathit{\beta }}^l\Vert }^2\right) \\ \ge \frac{\kappa R_{\mathit{min}}}{2n}\left(1-\frac{2L_{\mathit{max}}}{2n}\left(1+\frac{{\gamma }^2{M}^2{e}^2}{{\sigma }^{2}n}\right)\left(\frac{{\gamma }^2{M}^2{e}^2}{{\sigma }^{2}n}\right)\right)\times \mathbb{E}\left({\Vert Ps({\mathit{\beta }}^l)-{\mathit{\beta }}^l\Vert }^2\right) \\ \ge \frac{\kappa R_{\mathit{min}}}{2n{L}_{\mathit{max}}}\left(1-\frac{2L_{\mathit{max}}}{2n}\left(1+\frac{{\gamma }^2{M}^2{e}^2}{{\sigma }^{2}n}\right)\left(\frac{{\gamma }^2{M}^2{e}^2}{{\sigma }^{2}n}\right)\right)\times \mathbb{E}(Q^{\sigma *}-Q^{\sigma }({\mathit{\beta }}^l)) \end{gathered}$$

Therefore,

$$\begin{gathered} Q^{\sigma *}-\mathbb{E}(Q^{\sigma }({\mathit{\beta }}^{l+1}))=Q^{\sigma *}-\mathbb{E}(Q^{\sigma }({\mathit{\beta }}^l))-(\mathbb{E}(Q^{\sigma }({\mathit{\beta }}^{l+1})-\mathbb{E}(Q^{\sigma }({\mathit{\beta }}^l)))) \\ \le \eta (Q^{\sigma *}-\mathbb{E}(Q^{\sigma }({\mathit{\beta }}^l))) \end{gathered}$$

Let us assume that ${\beta }_{[k]}^{\ast }$ is the optimal solution of the subproblem (4) denoted as:

$${\beta }_{[k]}^{*}=\arg \underset{{\beta }_{[k]}\in {\mathbb{R}}^{n}k}{\max }{Q}^{\sigma }({\beta }_{[k]};\overline{\mathbf{w}}).$$ (28)

According to above proof of Lemma 7, the local atomic solver has a linear convergence rate in expectation, that is,

$$Q^{\sigma }({\beta }_{[k]}^{*};\overline{\mathbf{w}})-\mathbb{E}\left(Q^{\sigma }({\beta }_{[k]}^{j+1};\overline{\mathbf{w}})\right)\le \eta \left(\mathbb{E}\left(Q^{\sigma }({\beta }_{[k]}^{*};\overline{\mathbf{w}})-Q^{\sigma }({\beta }_{[k]}^j;\overline{\mathbf{w}})\right)\right)$$

It is obvious that Θ = η^RH. Thus, we can easily get the induction as

$$\begin{gathered} Q^{\sigma }({\beta }_{[k]}^{*};\overline{\mathbf{w}})-\mathbb{E}(Q^{\sigma }({\beta }_{[k]}^{RH};\overline{\mathbf{w}})) \\ \le \eta (Q^{\sigma }({\beta }_{[k]}^{*};\overline{\mathbf{w}})-\mathbb{E}(Q^{\sigma }({\beta }_{[k]}^{RH-1};\overline{\mathbf{w}}))) \\ \le {\eta }^2(\mathbb{E}(Q^{\sigma }({\beta }_{[k]}^{*};\overline{\mathbf{w}})-Q^{\sigma }({\beta }_{[k]}^{RH-2};\overline{\mathbf{w}}))) \\ \le \cdots \le \Theta (Q^{\sigma }({\beta }_{[k]}^{*};\overline{\mathbf{w}})-\mathbb{E}(Q^{\sigma }({\beta }_{[k]}^0;\overline{\mathbf{w}}))). \end{gathered}$$

Notice that ${\beta }_{[k]}^0$ are the starting points of the local atomic solver and ${\beta }_{[k]}^{R,H}$ are the final results of β_[k] of the local atomic solver. So the following equations hold for the global problem:

$$\begin{gathered} {\beta }_{[k]}^0={\alpha }_{[k]} \\ {\beta }_{[k]}^{R,H}-{\beta }_{[k]}^0={\mathit{\delta }}_{[k]} \\ {\beta }_{[k]}^{*}-{\beta }_{[k]}^0={\mathit{\delta }}_{[k]}^{*} \end{gathered}$$

Therefore, we have:

$${\rm E}[D_k({\mathit{\delta }}_{[k]}^{\ast };v,{\mathit{\alpha }}_{[k]})-D_k({\mathit{\delta }}_{[k]};v,{\mathit{\alpha }}_{[k]})]\le \Theta [D_k({\mathit{\delta }}_{[k]}^{\ast };v,{\mathit{\alpha }}_{[k]})-D_k(0;v,{\mathit{\alpha }}_{[k]})]$$

with Θ = η^RH. □

4.2. Convergence of global solution

Although we have showed that the local subproblem solver outputs a Θ-approximate solution, we cannot directly apply the results of [11] for the global solution because our algorithm uses updates from only a subset S ≤ K of workers which is unlike the synchronous all-reduce of the updates from all workers used in [11]. We need to handle this asynchronous nature of the global updates, just like we handled asynchronous updates for the local subproblem.

Let us consider the global updates in the order the master computed them (at global time t in Fig. 3). To prove convergence, it is customary to show that the global objective progresses sufficiently in each round, i.e., there is sufficient change from D(α^(t)) to D(α^(t+1)) where α^(t) denotes the value of the dual variable α distributed as ${\alpha }_{[k]}^{(t)}$ across all the workers k at the time master computed tth global update v^(t). For simplicity we assume that each worker updates ${\alpha }_{[k]}^{(t)}$ in step 11 of Algorithm 1 as soon as it receives global update from the master. Thus, α^(t+1) = α^(t) + νδ^(t) where ${\delta }^{(t)}={\sum }_k{\delta }_{[k]}^{(t)}$ and ${\delta }_{[k]}^{(t)}$ denotes the increment to ${\alpha }_{[k]}^{(t)}$ computed by worker k if $k\in {\mathcal{P}}_S^{(t)}$, 0 otherwise.

If $k\in {\mathcal{P}}_S^{(t)}$ then the update ${\delta }_{[k]}^{(t)}$ has already been included in v^(t). However, if $k\notin {\mathcal{P}}_S^{(t)}$ then it may not be included. Let ξ be such that for all l ≤ ξ and for all k, ${\delta }_{[k]}^{(l)}$ has been included in v^(t). By the design of our algorithm, t − Γ < ξ ≤ t. Let ${\widehat{\alpha }}^{(t)}$ be defined as follows: ${\widehat{\alpha }}_{[k]}^{(t)}={\alpha }_{[k]}^{(t)}$, $\forall k\in {\mathcal{P}}_S^{(t)}$ and $={\alpha }_{[k]}^{(\xi -1)}$ for the latest (ξ − 1) for which the update is already included in global v, $\forall k\notin {\mathcal{P}}_S^{(t)}$. Let w^(t), ${\widehat{\mathbf{w}}}^{(t)}$ be w(α^(t)) and $\mathbf{w}({\widehat{\alpha }}^{(t)})$ respectively. Note that ${\mathbf{w}}^{(t)}={\widehat{\mathbf{w}}}^{(t)}+\frac{\nu }{\mathrm{\lambda }n}{\sum }_{l=\xi }^t\mathbf{X}{\delta }^{(l)}$. For a vector expression (χ), let (χ)_i represent the ith element of the vector resulting from the expression (χ).

Lemma 8. For any dual α^(t), ${\delta }^{(t)}\in {\mathbb{R}}^n$, primal ${\widehat{\mathbf{w}}}^{(t)}=\mathbf{w}({\widehat{\alpha }}^{(t)})$ and real values ν and σ satisfying (5), it holds that

$$\begin{gathered} D({\mathit{\alpha }}^{(t+1)})=D\left({\mathit{\alpha }}^{(t)}+\nu \underset{k\in {\mathcal{P}}_S^{(t)}}{\Sigma }{\mathit{\delta }}_{[k]}^{(t)}\right) \\ \ge (1-\nu )D(\widehat{\mathit{\alpha }})+\nu \underset{k\in {\mathcal{P}}_S^{(t)}}{\Sigma }{D}_k({\mathit{\delta }}_{[k]}^{(t)};{\mathit{\alpha }}_{[k]},\widehat{w}) \\ -\frac{\mathrm{\lambda }}{2}\left(\frac{2v}{\mathrm{\lambda }n}\underset{k\notin {\mathcal{P}}_S^{(t)}}{\Sigma }w{(\widehat{\mathit{\alpha }})}^{⊺}X\left(\underset{l=\xi }{\overset{t}{\Sigma }}{\mathit{\delta }}_{[k]}^{(l)}\right) \\ +{\left(\frac{\nu }{\mathrm{\lambda }n}\right)}^2{\Vert \underset{k\notin {\mathcal{P}}_S^{(t)}}{\Sigma }\underset{c=\xi }{\overset{t}{\Sigma }}X{\mathit{\delta }}_{[k]}^{(c)}\Vert }^2\right) \\ -\frac{2{\nu }^2}{\mathrm{\lambda }{n}^2}{\left(\underset{\mathit{k}\notin {\mathcal{P}}_S^{(t)}}{\Sigma }\underset{l=\xi }{\overset{t}{\Sigma }}{\mathit{\delta }}_{[k]}^{(l)}\right)}^{⊺}{X}^{⊺}X\underset{k\in {\mathcal{P}}_S}{\Sigma }{\mathit{\delta }}_{[k]}^{(t)} \\ -\frac{1}{n}\underset{k\notin {\mathcal{P}}_S^{(t)}}{\Sigma }\left(\underset{i\in {\mathcal{I}}_k}{\Sigma }{\varphi }_i^{\ast }(-\widehat{\mathit{\alpha }}-\nu \underset{c=\xi }{\overset{t}{\Sigma }}{{\mathit{\delta }}_{[k]}^{(c)})}_i)\right). \end{gathered}$$ (29)

Proof. Assume that $\mathcal{I}={\bigcup }_{k\in {\mathcal{P}}_S}{\mathcal{I}}_k$. Then, we have

$$\begin{gathered} D\left({\mathit{\alpha }}^{(t)}+\nu \sum _{k\in {\mathcal{P}}_S^{(t)}}{\mathit{\delta }}_{[k]}^{(t)}\right)=-\frac{1}{n}{\varphi }_i^{*}\left(-{\widehat{\alpha }}_i-\nu {\left(\sum _{k\in {\mathcal{P}}_S^{(t)}}\sum _{c=\xi }^t{\mathit{\delta }}_{[k]}^{(c)}+\sum _{k\in {\mathcal{P}}_S^{(t)}}{\mathit{\delta }}_{[k]}^{(t)}\right)}_i\right) \\ -\frac{\lambda }{2}\Vert \frac{1}{\lambda n}\mathbf{X}\left(\widehat{\mathit{\alpha }}+\nu \sum _{k\notin {\mathcal{P}}_S^{(t)}}\sum _{c=\xi }^t{\mathit{\delta }}_{[k]}^{(c)}+\nu \sum _{k\in {\mathcal{P}}_S^{(t)}}{\mathit{\delta }}_{[k]}^{(t)}\right){\Vert }^2 \\ =-\frac{1}{n}\sum _{k\in {\mathcal{P}}_S^{(t)}}\left(\sum _{i\in {\mathcal{I}}_k}{\varphi }_i^{*}(-(1-\nu ){\widehat{\mathit{\alpha }}}_i-\nu {({\widehat{\mathit{\alpha }}}_i+{\mathit{\delta }}_{[k]}^{(t)})}_i)\right) \\ -\frac{1}{n}\sum _{k\in {\mathcal{P}}_S^{(t)}}\left(\sum _{i\in {\mathcal{I}}_k}{\varphi }_i^{*}((-\widehat{\mathit{\alpha }}-\nu \sum _{c=\xi }^t{\mathit{\delta }}_{[k]}^{(c)}{)}_i)\right)-\frac{\lambda }{2}(\Vert \mathbf{w}(\widehat{\mathit{\alpha }}){\Vert }^2+\frac{2\nu }{\lambda n}\sum _{k\in {\mathcal{P}}_S^{(t)}}\mathbf{w}{(\widehat{\mathit{\alpha }})}^{⊺}\mathbf{X}{\mathit{\delta }}_{[k]}^{(t)}+{\left(\frac{\nu }{\lambda n}\right)}^2\Vert \sum _{k\in {\mathcal{P}}_S^{(t)}}\mathbf{X}{\mathit{\delta }}_{[k]}^{(t)}{\Vert }^2)-\frac{\lambda }{2}(\frac{2\nu }{\lambda n}\sum _{k\in {\mathcal{P}}_S^{(t)}}\mathbf{w}{(\widehat{\mathit{\alpha }})}^{⊺}\mathbf{X}(\sum _{l=\xi }^t{\mathit{\delta }}_{[k]}^{(l)})+{\left(\frac{\nu }{\lambda n}\right)}^2\Vert \sum _{k\notin {\mathcal{P}}_S^{(t)}}\sum _{c=\xi }^t\mathbf{X}{\mathit{\delta }}_{[k]}^{(c)}{\Vert }^2)-\frac{2{\nu }^2}{\lambda n^2}{\left(\sum _{k\notin {\mathcal{P}}_S^{(t)}}\sum _{l=\xi }^t{\mathit{\delta }}_{[k]}^{(l)}\right)}^{⊺}{\mathbf{X}}^{⊺}\mathbf{X}\sum _{k\in {\mathcal{P}}_S}{\mathit{\delta }}_{[k]}^{(t)}) \\ \ge -\frac{1}{n}\sum _{k\in {\mathcal{P}}_S^{(t)}}(\sum _{i\in {\mathcal{I}}_k}((1-\nu ){\varphi }_i^{*}(-{\widehat{\alpha }}_i)+\nu {\varphi }_i^{*}{(-({\widehat{\mathit{\alpha }}}_i+{\mathit{\delta }}_{[k]}^{(t)})}_i))) \\ -\frac{\lambda }{2}\left(\Vert \widehat{w}{\Vert }^2+\frac{2\nu }{\lambda n}\sum _{k\in {\mathcal{P}}_S^{(t)}}{\widehat{w}}^{⊺}\mathbf{X}{\mathit{\delta }}_{[k]}^{(t)}+{\left(\frac{\nu }{\lambda n}\right)}^2\Vert \sum _{k\in {\mathcal{P}}_S^{(t)}}\mathbf{X}{\mathit{\delta }}_{[k]}^{(t)}){\Vert }^2\right) \\ -\frac{\lambda }{2}\left(\frac{2\nu }{\lambda n}\sum _{k\notin {\mathcal{P}}_S^{(t)}}\mathbf{w}{(\widehat{\mathit{\alpha }})}^{⊺}\mathbf{X}(\sum _{l=\xi }^t{\mathit{\delta }}_{[k]}^{(l)}))+{\left(\frac{\nu }{\lambda n}\right)}^2\Vert \sum _{k\notin {\mathcal{P}}_S^{(t)}}\sum _{c=\xi }^t\mathbf{X}{\mathit{\delta }}_{[k]}^{(c)}\Vert 2\right)-\frac{2{\nu }^2}{\lambda n^2}{(\sum _{k\notin {\mathcal{P}}_S^{(t)}}\sum _{I=\xi }^t{\mathit{\delta }}_{[k]}^{(l)}))}^{⊺}{\mathbf{X}}^{⊺}\mathbf{X}\sum _{k\in {\mathcal{P}}_S^{(t)}}^t{\mathit{\delta }}_{[k]}^{(t)} \\ -\frac{1}{n}\sum _{k\notin {\mathcal{P}}_S^{(t)}}\left(\sum _{i\in {\mathcal{I}}_k}{\varphi }_i^{*}((-\widehat{\mathit{\alpha }}-\nu \sum _{c=\xi }^t{\mathit{\delta }}_{[k]}^{(c)}{)}_i)\right)(\text{mean value theorem},\text{concave}) \\ =\underset{(1-\nu )D(\widehat{\mathit{\alpha }})}{-\frac{1}{n}\sum _{k=1}^n\left(\sum _{i\in {\mathcal{I}}_k}(1-\nu ){\varphi }_i^{*}(-{\widehat{\alpha }}_i)\right)-(1-\nu )\frac{\lambda }{2}}\Vert \mathbf{w}(\widehat{\mathit{\alpha }}){\Vert }^2 \\ +\frac{1}{n}\sum _{k\notin {\mathcal{P}}_S^{(t)}}(\sum _{i\in I_k}(1-v){\varphi }_i^{*}(-{\widehat{\alpha }}_i)) \\ +v\sum _{k\notin p_s^{(t)}}(-\frac{1}{n}\sum _{i\in I_k}{\varphi }_i^{*}(-{({\widehat{\alpha }}_i+{\delta }_{[k]}^{(t)})}_i-\frac{1}{S}\frac{\lambda }{2}\left|\right|w(\widehat{\alpha })|{|}^2-\frac{1}{n}w{(\widehat{\alpha })}^{T}X{\delta }_{[K]}^{(t)}-\frac{\lambda }{2}\sigma |\left|\frac{1}{\lambda n}X{\delta }_{[K]}^{(t)}\right|{|}^2) \\ -\frac{\lambda }{2}(\frac{2v}{\lambda n}\sum _{k\in p_S^{(t)}}w{(\widehat{\alpha })}^{T}X(\sum _{c=\xi }^t{\delta }_{[K]}^{(t)}))+{(\frac{v}{\lambda n})}^2\left|\right|\sum _{k\notin p_s^{(t)}}\sum _{c=\xi }^t{\delta }_{[K]}^{c)})\left|\right|2) \\ -\frac{2v}{\lambda n}{(\sum _{k\notin p_s^{(t)}}\sum _{I=\xi }^t{\delta }_{[K]}^{(t)}))}^T{X}^{T}X\sum _{k\in p_s^{(t)}}^t{\delta }_{[K]}^{(t)}) \\ -\frac{1}{n}\sum _{k\notin p_S^{(t)}}(\sum _{i\in I_k}{\varphi }_i^{*}((-\widehat{\alpha }-v)\sum _{c=\xi }^t{\delta }_{[K]}^{(t)}{)}_i)) \\ =(1-v)D(\widehat{\alpha })+v\sum _{k\in p_s^{(t)}}{D}_k{\delta }_{[K]}^{(t)};{\alpha }_{[k]},\widehat{w})+\frac{1-v}{n}\sum _{k\notin p_s^{(t)}}(\sum _{i\in I_k}{\varphi }_i^{*}(-{\widehat{\alpha }}_i)) \\ -\frac{\lambda }{2}(\frac{2v}{\lambda n}\sum _{k\in p_S^{(t)}}w{(\widehat{\alpha })}^{T}X(\sum _{c=\xi }^t{\delta }_{[K]}^{(t)}))+{(\frac{v}{\lambda n})}^2\left|\right|\sum _{k\notin p_s^{(t)}}\sum _{c=\xi }^t{\delta }_{[K]}^{c)})\left|\right|2) \\ -\frac{2v^2}{\lambda n^2}{(\sum _{k\notin p_s^{(t)}}\sum _{I=\xi }^t{\delta }_{[K]}^{(t)}))}^T{X}^{T}X\sum _{k\in p_s^{(t)}}^t{\delta }_{[K]}^{(t)}) \\ -\frac{1}{n}\sum _{k\notin p_S^{(t)}}(\sum _{i\in I_k}{\varphi }_i^{*}((-\widehat{\alpha }-v)\sum _{c=\xi }^t{\delta }_{[K]}^{(t)}{)}_i)) \end{gathered}$$

Assumption 5 (Bounded Delay of Global Updates, Γ). There exists a $\varrho <e^{\frac{2}{\Gamma +1}}$ such that

$${\Vert {\mathit{\delta }}^{(t-1)}\Vert }^2\le \varrho {\Vert {\mathit{\delta }}^{(t)}\Vert }^2.$$ (30)

Lemma 9 (Global Convergence at Each Iteration). If ${\varphi }_i^{\ast }$ are all (1/μ)-strongly convex and Assumptions 2-5 are satisfied then for any s ∈ [0, 1], any round t of Algorithm 2 satisfies

$$\mathbb{E}[D({\mathit{\alpha }}^{(t+1)})-D({\mathit{\alpha }}^{(t)})]\ge \Psi (1-\Theta )\left(sG(\widehat{\mathit{\alpha }})-\frac{\sigma }{2\mathrm{\lambda }}{\left(\frac{s}{n}\right)}^2\widehat{R}\right)$$ (31)

where

$$\Psi ≔\nu \left(1-\frac{(\Gamma +1)e^{2}M{L}_{\mathit{max}}}{\mathrm{\lambda }n}+\frac{S\Gamma M{L}_{\mathit{max}}}{K\mathrm{\lambda }n}\right)\le 1,\mathit{and}$$ (32)

$$\widehat{R}≔-\frac{\mathrm{\lambda }\mu n(1-s)}{\sigma s}{\Vert \widehat{\mathbf{u}}-\widehat{\mathit{\alpha }}\Vert }^2+{\sum _{k=1}^K\Vert X{(\widehat{\mathbf{u}}-\widehat{\mathit{\alpha }})}_{[k]}\Vert }^2,$$ (33)

for $\widehat{\mathbf{u}}\in {\mathbb{R}}^n$ with $-{\widehat{u}}_i\in \partial {\varphi }_i(\mathbf{w}{(\widehat{\alpha })}^{⊺}{\mathbf{x}}_i)$.

Proof. For sake of notation, we will write α instead of α^t, w instead of w(α^t), $\widehat{\mathbf{w}}$ instead of $\mathbf{w}(\widehat{\alpha })$ and δ instead of δ^t.

Now, the expected change of the dual objective is

$$\begin{gathered} \mathbb{E}[D({\mathit{\alpha }}^t)-D({\mathit{\alpha }}^{(t+1)})]=\mathbb{E}[D({\mathit{\alpha }}^t)-D(\widehat{\mathit{\alpha }})+D(\mathit{\alpha })-D({\mathit{\alpha }}^{(t+1)})] \\ =\mathbb{E}[D({\mathit{\alpha }}^t)-D(\mathit{\alpha })]+\mathbb{E}[D(\widehat{\mathit{\alpha }})-D({\mathit{\alpha }}^{(t+1)})] \end{gathered}$$

Thus, it is a summation of two parts. Let us estimate both parts as follows,

$$\begin{gathered} \mathbb{E}[D({\alpha }^t)-D(\widehat{\alpha })]=\mathbb{E}\left[-\frac{1}{n}\sum _{i\notin \mathcal{I}}{\varphi }_i^{\ast }(-{\widehat{\alpha }}_i-\nu \sum _{c=\xi }^t{\mathit{\delta }}^{(c)})-\frac{1}{n}\sum _{k\in {\mathcal{P}}_S}(\sum _{i\in {\mathcal{I}}_k}{\varphi }_i^{\ast }(-{\widehat{\alpha }}_i)\right)-\frac{\mathrm{\lambda }}{2}{\Vert \frac{1}{\mathrm{\lambda }n}\sum _{k\notin {\mathcal{P}}_s}X\left({\widehat{\mathit{\alpha }}}_{[k]}+\nu \sum _{c=\xi }^t{\mathit{\delta }}_{[k]}^{(c)}\right)\Vert }^2+\frac{1}{n}\sum _{k=1}^n{\varphi }^{\ast }(-{\widehat{\mathit{\alpha }}}_{[k]})+\frac{\mathrm{\lambda }}{2}{\Vert \widehat{\mathbf{w}}\Vert }^2] \\ =\mathbb{E}\left[-\frac{1}{n}\sum _{i\notin \mathcal{I}}{\varphi }_i^{\ast }(-{\widehat{\alpha }}_i-\nu \sum _{c=\xi }^t{\mathit{\delta }}^{(c)})+\frac{1}{n}\sum _{k\notin {\mathcal{P}}_s}(\sum _{i\in {\mathcal{I}}_k}{\varphi }_i^{\ast }(-{\widehat{\alpha }}_i)\right]-\frac{\mathrm{\lambda }}{2}\mathbb{E}\left[\frac{2\nu }{\mathrm{\lambda }n}\sum _{k\notin {\mathcal{P}}_S}w({\widehat{\mathit{\alpha }}}^{⊺}X\left(\sum _{c=\xi }^t{\delta }_{[k]}^{(i)}\right)+{\left(\frac{\nu }{\mathrm{\lambda }n}\right)}^2{\Vert \sum _{k\notin {\mathcal{P}}_S}\sum _{c=\xi }^{t}X{\mathit{\delta }}_{[k]}^{(c)}\Vert }^2\right] \\ \mathbb{E}[D(\widehat{\mathit{\alpha }})-D({\mathit{\alpha }}^{(t+1)})]=\mathbb{E}\left[D(\widehat{\mathit{\alpha }})-D(\alpha +\nu \sum _{k\in {\mathcal{P}}_S}{\mathit{\delta }}_{[k]}^{(t)})\right]\le \mathbb{E}[D(\mathit{\alpha })-(1-\nu )D(\widehat{\mathit{\alpha }})-\nu \sum _{k\in {\mathcal{P}}_S}{D}_k({\mathit{\delta }}_{[k]}^{(t)};{\mathit{\alpha }}_{[k]},\widehat{\mathbf{w}})+\frac{\nu }{n}\sum _{k\notin {\mathcal{P}}_S}(\sum _{i\in {\mathcal{I}}_k}{\varphi }_i^{\ast }(-{\widehat{\alpha }}_i))]-\mathbb{E}\left[-\frac{1}{n}\sum _{k\notin {\mathcal{P}}_S}(\sum _{i\in {\mathcal{I}}_k}{\varphi }_i^{\ast }{((-\widehat{\mathit{\alpha }}-\nu \sum _{c=\xi }^t{\mathit{\delta }}_{[k]}^{(c)})}_i))+\frac{1}{n}\sum _{k\notin {\mathcal{P}}_S}(\sum _{i\in {\mathcal{I}}_k}{\varphi }_i^{\ast }(-{\widehat{\alpha }}_i)\right)]+\frac{\mathrm{\lambda }}{2}\mathbb{E}\left[\frac{2\nu }{\mathrm{\lambda }n}\sum _{k\notin {\mathcal{P}}_S}w{(\widehat{\alpha })}^{⊺}X\left(\sum _{c=\xi }^t{\mathit{\delta }}_{[k]}^{(c)}\right)+{\left(\frac{\nu }{\mathrm{\lambda }n}\right)}^2{\Vert \sum _{k\notin {\mathcal{P}}_S}\sum _{c=\xi }^{t}X{\mathit{\delta }}_{[k]}^{(c)}\Vert }^2\right]+\mathbb{E}\left[\frac{2{\nu }^2}{\mathrm{\lambda }{n}^2}{\left(\sum _{k\notin {\mathcal{P}}_S}\sum _{c=\xi }^t{\mathit{\delta }}_{[k]}^{(c)}\right)}^{⊺}{X}^{⊺}X\sum _{k\in {\mathcal{P}}_S}{\mathit{\delta }}_{[k]}^{(t)}\right](\text{Lemma}8) \end{gathered}$$

Therefore,

$$\begin{gathered} (\text{sum of previous two inequalities})=\nu \mathbb{E}[D({\alpha }^t)-D({\alpha }^{(t+1)})]=\mathbb{E}[D({\alpha }^t)-D({\alpha }^t)]+\mathbb{E}[D(\alpha )-D({\alpha }^{(t+1)})] \\ \le \mathbb{E}[D(\alpha )-(1-\nu )D(\alpha )-\nu \sum _{k\in {\mathcal{P}}_s}{D}_k({\delta }_{[k]}^{(t)};{\alpha }_{[k]},w)+\frac{\nu }{n}\sum _{k\notin {\mathcal{P}}_s}(\sum _{i\in I_k}{\varphi }_i^{\ast }(-{\alpha }_i))] \\ +\mathbb{E}[\frac{2{\nu }^2}{\mathrm{\lambda }{n}^2}{(\sum _{k\notin {\mathcal{P}}_s}\sum _{c=\xi }^t{\delta }_{[k]}^{(c)})}^T{X}^{T}X\sum _{k\in {\mathcal{P}}_s}{\delta }_{[k]}^{(t)}] \\ (sumofprevioustwoinequality) \\ =\nu \mathbb{E}[D(\alpha )-\sum _{k=1}^K{D}_k({\delta }_{[k]}^{\ast };{\alpha }_{[k]},w)+\sum _{k=1}^K{D}_k({\delta }_{[k]}^{\ast };{\alpha }_{[k]},w)-\sum _{k=1}^K{D}_k(0;{\alpha }_{[k]},w)] \\ +\mathbb{E}[\frac{2{\nu }^2}{\mathrm{\lambda }{n}^2}{(\sum _{k\notin {\mathcal{P}}_s}\sum _{c=\xi }^t{\delta }_{[k]}^{(c)})}^T{X}^{T}X\sum _{k\in {\mathcal{P}}_s}{\delta }_{[k]}^{(t)}] \\ \le \nu (\Theta (\sum _{k=1}^K{D}_k({\delta }_{[k]}^{\ast };{\alpha }_{[k]},w)-\underset{D(\widehat{\alpha })}{\underbrace{\sum _{k=1}^K{D}_k(0;{\alpha }_{[k]},w)}})+D(\alpha )-\sum _{k=1}^K{D}_k({\delta }_{[k]}^{\ast };{\alpha }_{[k]},w)) \\ +\mathbb{E}[\frac{{\nu }^2}{\mathrm{\lambda }{n}^2}{(\sum _{c=\xi }^t{\delta }_{[k]}^{(c)})}^T{X}^{T}X\sum _{k\in {\mathcal{P}}_s}{\delta }_{[k]}^{(t)}] \\ =\nu (1-\Theta )D(\alpha )-\sum _{k=1}^K{D}_k({\delta }_{[k]}^{\ast };{\alpha }_{[k]},w)+\mathbb{E}[\frac{2{\nu }^2}{\mathrm{\lambda }{n}^2}{(\sum _{k\notin {\mathcal{P}}_s}\sum _{c=\xi }^t{\delta }_{[k]}^{(c)})}^T{X}^{T}X\sum _{k\in {\mathcal{P}}_s}{\delta }_{[k]}^{(t)}] \\ \le \nu (1-\Theta )D(\alpha )-\sum _{k\in {\mathcal{P}}_s}{D}_k({\delta }_{[k]}^{\ast };{\alpha }_{[k]},w) \\ +\frac{\nu }{\mathrm{\lambda }n}\underset{A}{\underbrace{(\mathbb{E}[{\Vert \sum _{k\notin {\mathcal{P}}_s}X(\sum _{c=\xi }^t{\delta }_{[k]}^{(c)})\Vert }^2]+\mathbb{E}[{\Vert \sum _{k\in {\mathcal{P}}_s}X{\delta }_{[k]}^{(t)}\Vert }^2])}} \\ (a^2+b^2\ge 2ab) \end{gathered}$$

Before bounding the term A, we need the following proposition:

Proposition 10.

$$\begin{gathered} \mathbb{E}\left[{\Vert X{\mathit{\delta }}_{[k]}^{(t)}\Vert }^2\right]=\frac{1}{S}\mathbb{E}\left[\sum _{k\in {\mathcal{P}}_s}{\Vert X{\mathit{\delta }}_{[k]}^{(t)}\Vert }^2\right] \\ =\frac{1}{K}\mathbb{E}\left[\sum _{k=1}^K{\Vert X{\mathit{\delta }}_{[k]}^{(t)}\Vert }^2\right]\forall k\in \{1,\dots ,K\}. \end{gathered}$$

Now, let us bound the term A. We have

$$\begin{gathered} A=\mathbb{E}\left[{\Vert \sum _{k\notin {\mathcal{P}}_s}X\left(\sum _{c=\xi }^t{\delta }_{[k]}^{(c)}\right)\Vert }^2\right]+\mathbb{E}\left[{\Vert \sum _{k\in {\mathcal{P}}_s}X{\mathit{\delta }}_{[k]}^{(t)}\Vert }^2\right] \\ \le \mathbb{E}\left[(\Gamma +1){\sum _{c=t-\Gamma }^t\Vert \sum _{k\notin {\mathcal{P}}_s}X{\mathit{\delta }}_{[k]}^c\Vert }^2\right]+\mathbb{E}\left[S\sum _{k\in {\mathcal{P}}_s}{\Vert X{\mathit{\delta }}_{[k]}^{(t)}\Vert }^2\right](\text{Cauchy Schwarz Inequality}) \\ \le \mathbb{E}\left[\frac{(K-S)(\Gamma +1)M}{K}\sum _{k=1}^K\sum _{c=t-\Gamma }^{t-1}{\Vert {\mathit{\delta }}_{[k]}^c\Vert }^2\right]+\mathbb{E}\left[\frac{SM}{K}\sum _{k=1}^K{\Vert {\mathit{\delta }}_{[k]}^{(t)}\Vert }^2\right](\text{Propositions}2\&10) \\ \le \mathbb{E}\left[\frac{(K-S)(\Gamma +1)M}{K}\left(\sum _{c=t-\Gamma }^t{\varrho }^c\right)\sum _{k=1}^K{\Vert {\mathit{\delta }}_{[k]}^{(t)}\Vert }^2\right]+\mathbb{E}\left[\frac{SM}{K}\sum _{k=1}^K{\Vert {\mathit{\delta }}_{[k]}^{(t)}\Vert }^2\right](\text{By}(30)) \\ \le \frac{(K-S)(\Gamma +1)M{L}_{\mathit{max}}}{K}\sum _{c=t-\Gamma }^t{\varrho }^c(D(\widehat{\mathit{\alpha }})-\sum _{k=1}^K{D}_k({\mathit{\delta }}_{[k]}^{(t)};{\mathit{\alpha }}_{[k]},\widehat{\mathbf{w}}))+\frac{SM{L}_{\mathit{max}}}{K}\sum _{k=1}^K\left(D(\widehat{\mathit{\alpha }})-D_k({\mathit{\delta }}_{[k]}^{(t)};{\mathit{\alpha }}_{[k]}^{(t)},\widehat{\mathbf{w}})\right)(\text{Proposition}5) \end{gathered}$$

where $D(\widehat{\alpha })=D_k(0;{\alpha }_{[k]},\widehat{\mathbf{w}})$. Thus, Eq. (9) can be rewritten as,

$$\begin{gathered} \mathbb{E}\left[D_k({\mathit{\delta }}_{[k]}^{\ast };{\mathit{\alpha }}_{[k]},\widehat{\mathbf{w}})-D_k({\mathit{\delta }}_{[k]}^{(t)};{\mathit{\alpha }}_{[k]},\widehat{\mathbf{w}})\right]\le \Theta (D_k({\mathit{\delta }}_{[k]}^{\ast };{\mathit{\alpha }}_{[k]},\widehat{\mathbf{w}})-D(\widehat{\mathit{\alpha }}))+D(\widehat{\mathit{\alpha }})-D(\widehat{\mathit{\alpha }}) \\ D(\widehat{\mathit{\alpha }})-D_k({\mathit{\delta }}_{[k]}^{(t)};{\mathit{\alpha }}_{[k]},\widehat{\mathbf{w}})\le (1-\Theta )D(\widehat{\mathit{\alpha }})-(1-\Theta )D_k({\mathit{\delta }}_{[k]}^{\ast };{\mathit{\alpha }}_{[k]},\widehat{\mathbf{w}}) \\ D(\widehat{\mathit{\alpha }})-D_k({\mathit{\delta }}_{[k]}^{(t)};{\mathit{\alpha }}_{[k]},\widehat{\mathbf{w}})\le -(1-\Theta )(D_k({\mathit{\delta }}_{[k]}^{\ast };{\mathit{\alpha }}_{[k]},\widehat{\mathbf{w}})-D(\widehat{\mathit{\alpha }})) \end{gathered}$$ (34)

Then, A can be bounded as,

$$\begin{gathered} A\le -\frac{(K-S)(\Gamma +1)M{L}_{\max }}{K}{\varrho }^{\Gamma +1}(1-\Theta )\sum _{k=1}^K(D(\widehat{\mathit{\alpha }})-D_k({\mathit{\delta }}_{[k]}^{*};{\mathit{\alpha }}_{[k]},\widehat{\mathbf{w}}))-\frac{SM{L}_{\mathit{max}}}{K}(1-\Theta )\sum _{k=1}^K(D(\widehat{\mathit{\alpha }})-D_k({\mathit{\delta }}_{[k]}^{*};{\mathit{\alpha }}_{[k]},\widehat{\mathbf{w}}))(\text{By}(34)) \\ \le -\frac{(K-S)(\Gamma +1)e^{2}M{L}_{\max }}{2}(1-\Theta )\sum _{k=1}^K(D(\widehat{\mathit{\alpha }})-D_k({\mathit{\delta }}_{[k]}^{*};{\mathit{\alpha }}_{[k]},\widehat{\mathbf{w}}))-\frac{SM{L}_{\mathit{max}}}{K}(1-\Theta )\sum _{k=1}^K(D(\widehat{\mathit{\alpha }})-D_k({\mathit{\delta }}_{[k]}^{*};{\mathit{\alpha }}_{[k]},\widehat{\mathbf{w}}))(\text{Assumption}5) \end{gathered}$$

By substituting A, we have

$$\begin{gathered} \mathbb{E}[D({\mathit{\alpha }}^t)-D({\mathit{\alpha }}^{(t+1)})]\le \nu \left(1-\frac{(\Gamma +1)e^{2}M{L}_{\mathit{max}}}{\mathrm{\lambda }n}+\frac{S\Gamma M{L}_{\mathit{max}}}{K\mathrm{\lambda }n}\right)(1-\Theta )\sum _{k=1}^K(D(\widehat{\alpha })-D_k({\mathit{\delta }}_{[k]}^{\ast })) \\ \mathbb{E}[D({\mathit{\alpha }}^t)-D({\mathit{\alpha }}^{(t+1)})]\le \Psi (1-\Theta )\sum _{k=1}^K(D(\widehat{\mathit{\alpha }})-D_k({\mathit{\delta }}_{[k]}^{\ast })) \end{gathered}$$

Using the Eq. C in the proof of Lemma 5 in [11], we can show that

$$\begin{gathered} \mathbb{E}[D({\mathit{\alpha }}^t)-D({\mathit{\alpha }}^{(t+1)})]\le \Psi (1-\Theta )\left(-sG(\widehat{\mathit{\alpha }})-\frac{1}{2\mu }(1-s)s\frac{1}{n}{\Vert \widehat{\mathbf{u}}-\widehat{\mathit{\alpha }}\Vert }^2+\frac{\sigma }{2\mathrm{\lambda }}{\left(\frac{s}{n}\right)}^2\sum _{k=1}^K\Vert X{(\widehat{\mathbf{u}}-\widehat{\mathit{\alpha }})}_{[k]}\Vert \right) \\ =\Psi (1-\Theta )\left(-sG(\widehat{\mathit{\alpha }})+\frac{\sigma }{2\mathrm{\lambda }}{\left(\frac{s}{n}\right)}^2\widehat{R}\right)\square \end{gathered}$$

Remark. When S (the minimal number of workers required to update before a global update is communicated) is fixed in (32), Ψ will be approach 0 when Γ (the maximal delay allowed, for the slowest worker) becomes larger and larger since (Γ + 1)e² > Γ. In other words, the improvement from D(α^(t)) to D(α^(t+1)) at iteration t becomes smaller when a larger delay is allowed among the workers.

When Γ is fixed in (32), Ψ will be larger when S is set larger. The improvement from D(α^(t)) to D(α^(t+1)) at iteration t will be more significant when more updates from different workers are taken into account. However, when Γ = 0, Ψ will be independent from S because all updates have to be gathered by the master.

Using the main results in [11] and combining Lemma 7 with Lemma 9 yield the following two convergence results, one for smooth loss functions and the other for the Lipschitz continuous loss functions. The theorems use the quantities σ_max = max_k σ_k, σ_sum = ∑_k σ_kn_k where ∀k, ${\sigma }_k={\max }_{{\alpha }_{[k]}\in {\mathbb{R}}^n}{\parallel \mathbf{X}{\alpha }_{[k]}\parallel }^2∕{\parallel {\alpha }_{[k]}\parallel }^2$.

Theorem 11 (Global convergence for (1/μ)-Smooth Functions). If the loss functions ϕ_i are all (1/μ)-smooth, then in T₁ iterations Algorithm 2 finds a solution with objective atmost ε_D from the optimal, i.e., $\mathbb{E}[D({\alpha }^{\ast })-D({\alpha }^{(T_1)})]\le {∊}_D$ whenever $T_1\ge C_1\log \frac{1}{{∊}_D}$ where $C_1=\frac{1}{\Psi (1-ϴ)}(1+\frac{{\sigma }_{\mathit{max}}\sigma }{\nu \mathrm{\lambda }n})$ and Θ is given by (25). Furthermore, in T₂ iterations, it finds a solution with duality gap atmost ϵ_gap, i.e., $\mathbb{E}[P(\mathbf{w}({\alpha }^{(T_2)}))-D({\alpha }^{(T_2)})]\le {∊}_{\text{gap}}$ whenever $T_2\ge C_1\log \frac{C_1}{{∊}_D}$.

Theorem 12 (Global convergence for L-Lipschitz functions). If the loss functions ϕ_i are all L-Lipschitz, then in T₁ iterations Algorithm 2 finds a solution with duality gap at most ϵ_gap, i.e., $\mathbb{E}[P(\mathbf{w}(\stackrel{‒}{\alpha }))-D(\stackrel{‒}{\alpha })]\le {∊}_{\text{gap}}$ for the average iterate $\stackrel{‒}{\alpha }=\frac{1}{T_1-T_0}{\sum }_{t=T_0+1}^{T_1-1}{\alpha }^{(t)}$ whenever $T_1\ge T_0+\max \{\lceil \frac{1}{\Psi (1-ϴ)}\rceil ,\frac{4L^2{\sigma }_{\text{sum}}\sigma }{\mathrm{\lambda }{n}^2{∊}_{\text{gap}}\Psi (1-ϴ)}\}$, and $T_0\ge \max \{0,\lceil \frac{1}{\Psi (1-ϴ)}\log \frac{2\mathrm{\lambda }{n}^2(D({\alpha }^{\ast }-D({\alpha }^{(0)})}{4L^2{\sigma }_{\text{sum}}\sigma }\rceil \}+\max \{0,\frac{2}{\Psi (1-ϴ)}(\frac{8L^2{\sigma }_{\text{sum}}\sigma }{\mathrm{\lambda }{n}^2{∊}_{\text{gap}}}-1)\}$ and Θ is given by (25).

Theorem 12 establishes the convergence for L-Lipschitz continuous loss functions, and Theorem 11 proves a linear convergence rate for smooth convex loss functions.

5. Experimental Results

We implemented our algorithm in C++ using Open MPI and OpenMP. All our experiments were conducted on the Biowulf cluster at the National Institutes of Health, USA, using up to K = 16 nodes where each node has 58 GB main memory and R = 16 cores each with a 2.6 GHz Xeon E5–2650v2 processor and 20MB secondary cache. Though the cores in the cluster are hyperthreading enabled, we did not use hyperthreading mode for our experiments. Each node in the cluster runs exactly one MPI task corresponding to a worker which in turn runs one OpenMP thread on each core available within the node. The main thread in a worker handles the inter-node communication and the root MPI task works as the master. For MPI node scheduling we used the slurm task scheduler with the settings --ntasks–per–node=1 and --threads–per–core=1 whereas for OpenMP thread scheduling we used a simple CPU affinity scheduler that always assigned thread i to physical core i where i ∈ {1, …, 16}.

5.1. Datasets

We evaluated our algorithm against three other algorithms on four binary classification datasets, rcv1, webspam, kddb and splicesite from the LIBSVM [3] website (https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/binary.html) as shown in Table 1. For datasets rcv1 and kddb we used separate training and test data files downloaded directly from the website. As there was no separate test data file for webspam, we divided the data file into two parts and used the first part containing 80% of the datapoints as the training set and the remaining as the test set. For splicesite we used the test file on the website as our training set (to better test the scalability of our algorithm as the test file was bigger). We chose the datasets in such a way that we had representatives of several scenarios: data sample-heavy rcv1 where n ⪢ d, feature-heavy webspam where n ⪡ d, both sample and feature heavy kddb where both n ≈ d are high, and the bigdataset splicesite which was more dense in both ways.

Table 1.

Datasets.

Dataset details	rcv1	webspam	kddb	splicesite
Training set	rcv1_train.binary	webspam_*_trigram	kddb	splice_site.t
File size	1.2 GB	20 GB	5.1 GB	280 GB
Training size n	677,399	280,000	19,264,097	4,627,840
Number of features d	47,236	16,609,143	29,890,095	11,725,480
Non-zero entries nnz	49,556,258	1,045,051,224	566,345,888	15,383,587,858
Test set	rcv1_test.binary	webspam_*_trigram	kddb.t	-
Test size	677,399	69,632	748,401	-

5.2. Comparison of algorithms

We experimented with the following four algorithms:

• Baseline: a sequential implementation of stochastic dual coordinate ascent (DCA) [6] which runs on a single core of a single node

• CoCoA+: a MPI based distributed implementation of stochastic DCA [11] which runs on multiple nodes, however, each node uses a single core

• PassCoDe: an OpenMP based parallel implementation of stochastic DCA [7] which runs on a single node, however, the node uses multiple cores

• Hybrid-DCA: an OpenMP+MPI implementation of our hybrid parallel distributed approach for stochastic DCA which runs on multiple nodes and each node uses multiple cores.

5.3. Parameter settings

We evaluated all the four algorithms for the hinge loss, though other loss functions could be tested too, with the regularization parameter λ. In our experiments with three values λ ∈ {10⁻³, 10⁻⁴, 10⁻⁵}, we observed similar patterns of results and we reported the results for λ = 10⁻⁴ only. All the three parallel/distributed algorithms, namely PassCoDe, CoCoA+, Hybrid-DCA, have a global parameter G denoting the number of basic updates to the dual variable α that are made in each global round. The parameter G acts as a tradeoff between the progress on the dual objective and the time taken in each global round. In the original implementation in [7] PassCoDe sets G equal to the number of datapoints n in the dataset whereas CoCoA implementation in [11] uses a smaller G than n.

We experimented with different values of G and found G = 40000, 30000, 2000000 for the datasets rcv1, webspam, kddb, respectively, gave the best results in our empirical study. In our implementation, for PassCoDe on t cores each thread made about H = G/t local updates, for PassCoDe on p nodes, each MPI task made H = G/p local updates and for Hybrid-DCA on p nodes each with t cores, each thread within an MPI task made H = G/(p × t) local updates so that all the three algorithms made total G updates in a global round. Though the sequential algorithm Baseline did not have any local iteration parameter, for better comparison we computed performance metrics such as time taken after every G updates and treated such G updates as a global round. We set aggregation parameters ν = 1, and the scaling parameter σ = K for both CoCoA+, Hybrid-DCA as recommended in [11].

Our Hybrid-DCA algorithm has additional parameters, the bounded barrier S and bounded delay Γ so that updates from only S out of p workers are incorporated in each global round with a maximum delay of Γ rounds for any update from the workers. In our implementation of Hybrid-DCA, we treated the two cases slightly differently: synchronous Hybrid-DCA with S = p where the updates from the workers were merged using the collective operation MPI_Allreduce, and asynchronous Hybrid-DCA with S < p where the updates from a subset of S out of p workers were merged and distributed using basic MPI “send and receive” commands. For all our experiments, we ran algorithms 10 times for each setting and reported the average of measured values.

5.4. Optimization performance

Fig. 5 shows the progress of duality gap achieved by the four algorithms on the three relatively smaller datasets rcv1, webspam and kddb. We chose the number of nodes (p ≤ K) and the number of cores (t ≤ R) per node such that the total number of worker cores (p × t) was the same (16) for all algorithms except Baseline. For Hybrid-DCA we set the parameters as the bounding barrier S = p and the delay Γ = 1 so that updates from all workers were incorporated in each global round. However, for p = t = 4 we also experimented with S = 3 and Γ = 4 to compare how Hybrid-DCA performed when updates from only 3 out of 4 workers were incorporated in each round with a maximum delay of 4 rounds for any update from the workers. The duality gap was measured as P(w) − D(α) where the primal estimate w was computed as (w = v) w = w(α) at the end of each global round. However, when S < K it was not possible for the master in Hybrid-DCA to gather the parts of P(w) from all workers at the end of each global round. To workaround in such a case, we let the master temporarily store w in disk after each round and at the end of all stipulated rounds, the workers computed the respective parts of P(w) from the stored w and the master computed the duality gap using a series of synchronous all-reduce computations from all the workers.

Fig. 5.

Performance of different solvers on three datasets, rcv1 (left column), webspam (middle column), and kddb (right column), in terms of the progress of the duality gap across the number of rounds (top row) and across the wall time taken (bottom row).

The bottom row of Fig. 5 shows the progress of the duality gap over time, while the top row shows the progress after each global round. In terms of progress across global rounds, the algorithms performed somewhat equally except for S < p where Hybrid-DCA had slightly slower progress on duality gap as the updates from one of the workers was missing in each round. In terms of time, there was no clear winner of the three parallel/distributed algorithms. CoCoA+ showed an advantage over PassCoDe on datasets with a smaller number of datapoints n, such as rcv1 and webspam, because the costly inter-node communication complexity was O(n) per round. For kddb where the dataset had more non-zero data elements, PassCoDe performed better. For all the three datasets the performance of Hybrid-DCA came in between CoCoA+ and PassCoDe by balancing inter-core and inter-node communications. Since the asynchronous Hybrid-DCA with S < p missed updates from some of the workers, took longer time as expected than synchronous Hybrid-DCA with S = p.

5.5. Test accuracy

To compare the quality of the solution obtained by each algorithm we used test datasets from LIBSVM binary repository the details of which are given in Table 1 and in Section 5.1. Note that the webspam dataset in LIBSVM did not have a separate test dataset. We divided it into two parts and took the first 280,000 datapoints in training and the remaining 69, 632 datapoints for test. For each of the empirical datatsets, we computed the accuracy, i.e., the fraction of test datapoints that were classified correctly by each of the algorithms and plotted the results on Fig. 6. In terms of the progress on accuracy across rounds, all the algorithms performed somewhat equally except for Hybrid-DCA with S < p which missed updates from one of the workers. Though, CoCoA+ apparently worked better on kddb, the margin was about 0.0005. However, in the long run all algorithms reached the same accuracy level. The progress on accuracy across time can be explained by the progress on duality gap across time as explained in Section 5.4.

Fig. 6.

Performance of different solvers on three datasets, rcv1 (left column), webspam (middle column), and kddb (right column), in terms of the progress of the duality gap across the number of rounds (top row) and across the wall time taken (bottom row).

5.6. Speedup

Speedup evaluates the improvement in performance of an algorithm as a function of the number of cores used. However, the exact definition of speedup varies in the literature. For example, PassCoDe computes speedup as the improvement in runtime to execute a fixed number of rounds, whereas CoCoA+ uses a more refined notion of speedup for an optimization problem like RRM defined in equation (1) and computes speedup as the improvement in runtime to achieve a fixed level of duality gap. In our experiments, we evaluated the algorithms using both notions of speedup. Let TR_A(p, t, r) and TD_A(p, t, ε) denote the time taken by an algorithm A using p nodes each with t cores to complete r rounds and to achieve duality gap ϵ, respectively. We formally define the two notions of speedup as follows²:

$$\begin{gathered} {\text{Speedup}}_A(p,t,r)=\frac{{\mathit{TR}}_A(p,t,r)}{{\mathit{TR}}_{\text{Baseline}}(1,1,r)},\text{and} \\ {\text{Proficiency}}_A(p,t,ϵ)=\frac{{\mathit{TD}}_A(p,t,ϵ)}{{\mathit{TD}}_{\text{Baseline}}(1,1,ϵ)} \end{gathered}$$

We ran sufficient rounds (≥ 100) of each of the four algorithms and computed Speedup_A(p, t, 100) and Proficiency_A(p, t, 10⁻³) for all algorithms A except Baseline, as shown in the top two rows of Fig. 7. PassCoDe can be run only on a single node; so we varied only the number of cores. Because CoCoA+ could use only 1 core per node. We ran CoCoA+ and Hybrid-DCA with t = 1 cores on p ∈ {1, 2, 4, 8, 12, 16, 20, 24, 28, 31} nodes and plotted the results separately. We also ran synchronous Hybrid-DCA on p ∈ {2, 4, 8} nodes each with t ∈ {2, 4, 6, 8, 10, 12, 13, 14, 15, 16} cores, S = p, γ = 1 and plotted the results separately for each p fixed and varying t. For p = 8 nodes and the same set of possible number of cores, we additionally plotted the results for asynchronous Hybrid-DCA with S = 6, Γ = 4.

Fig. 7.

Speedup of different parallel or distributed solvers with respect to the sequential implementation Baseline.

Our first observation on the results of speedup experiments was that though proficiency and speedup were computed differently, they turned out to follow a similar trend. In fact, speedup was slightly higher than proficiency as the merging of parallel/distributed updates in every global round reduced the quality of the merged update in comparison with the pure updates of Baseline. We also observed that the speedup and proficiency of the algorithms followed a trend similar to the performance seen in Section 5.4, i.e., CoCoA+ performed better on datasets that were either sample-heavy or feature heavy but not both, PassCoDe performed better on dataset that was both sample-heavy and feature heavy, and Hybrid-DCA performed in between. Furthermore, asynchronous Hybrid-DCA ran slower than synchronous Hybrid-DCA.

While investigating why Hybrid-DCA was slower than our expectation, we observed that the overhead of OpenMP in maintaining the parallel threads was significant on the datasets rcv1 and webspam as evident from the performance of CoCoA.t1 and Hybrid-DCA.t1 where their only difference was the overhead of maintaining a single OpenMP thread. However, this overhead was relatively small in comparison with the overall computation in the sample and feature heavy kddb dataset. However, we also noticed that CoCoA+ and PassCoDe did not scale well beyond 20 nodes and 14 cores, respectively.

Further investigation revealed two drawbacks of our implementation of MPI based inter-node communication. Firstly, MPI was inherently single threaded. Even for t > 1, for the whole duration when the workers in Hybrid-DCA sent their local updates to the master and received back the merged global update from the master, only the main thread in the workers was active and all other threads were idle. Though there has been academic research such as [21] on making MPI communications multi-threaded taking advantage of OpenMP threads available within the same task, the research has not been incorporated in the standard MPI implementations. This inherent drawback hindered Hybrid-DCA to take full advantage of all the cores available. The second drawback was the way we implemented asynchronous inter-node data transfers in Hybrid-DCA for S < p. In the absence of a collective operation for receiving updates from only a subset of all workers in the standard MPI implementation, we implemented such a collective operation using primitive MPI “send and receive” operations that lacked the performance improvement of CoCoA+ that utilized the optimized MPI collectives such as MPI_Allreduce.

To mask the effect of the two drawbacks in the MPI, we drew the plots for speedup and proficiency, as shown in the bottom two rows of Fig. 7, after ignoring the time involved in MPI communications. Though CoCoA+ also received the advantage of ignoring MPI time and showed better performance for datasets rcv1 and webspam, it became totally outperformed by Hybrid-DCA on the sample-heavy kddb dataset. We also observed better performance of asynchronous Hybrid-DCA than synchronous Hybrid-DCA. In summary, as expected theoretically, we see almost the same speedup for a fixed number of total p × t cores irrespective of individual values of p and t. The drop in performance of Hybrid-DCA on t > 14 cores could be due to 1) the increase of time taken in an atomic memory update as the number of threads increase, and 2) for higher number of threads the delay bound of local updates may violate the assumption in (16). Both the aspects of atomic memory writes for higher number of threads have been investigated and worked around in a follow-up paper [25] of original PassCoDe paper [7], however, the incorporation of a similar fix in our implementation is out of scope for this paper.

We ignored MPI time for the computation of speedup and proficiency only for the experiments described in the section, for the experiments elsewhere we do include MPI time while measuring the wall time.

5.7. Effects of the parameter S

Fig. 8 shows the results of varying S ∈ {2, 3, 4, 6, 8} with fixed Γ = 10 on p = 8 nodes each with t = 8 cores. When S < p/2, only a minority of the workers contributed in a round and the duality gap did not progress smoothly below some certain level. On the other hand, when at least half of the workers contributed in each round, it was possible to achieve the same duality level obtained using all the workers. However, the reduction in time per round was eventually eaten by the larger number of rounds required to achieve the same duality gap. Nevertheless, we will see in a later section that the approach was useful for HPC platforms with heterogeneous nodes, unlike ours, where the waiting for updates from all workers had larger penalty per round, or for the case, where the need was to run for a specified number of rounds and quickly achieved a reasonably good duality gap.

Fig. 8.

Effect of varying S on p = 8 worker nodes, with Γ fixed at 10.

5.8. Effects of the parameter Γ

Fig. 9 shows the results of varying Γ ∈ {1, 2, 3, 4, 10} with fixed S = 6 on p = 8 nodes each with t = 8 cores. We did not see much effect of Γ as the HPC platform used for our experiments had homogeneous nodes. Our experimentation showed that even if we used Γ = 10, the stale value at any worker was for at most 4 rounds. We expect to see a larger variance of staleness in case of heterogeneous nodes.

Fig. 9.

Effect of varying Γ on p = 8 worker nodes, with S fixed at 8.

5.9. Effects of workload and processing power

When the HPC system had homogeneous nodes, our experimental results showed that asynchronous Hybrid-DCA did not change much when varying S and Γ. To show usefulness of asynchronous Hybrid-DCA on heterogeneous systems, we introduced imbalance in the setup for an experiment on p = 4 nodes each with t = 4 cores in two different ways: 1) by varying workload and 2) independently varying processing power. To vary workload, instead of distributing the datapoints equally on all 4 nodes, we loaded one of the nodes heavily by distributing the datapoints in the ratios 1:1:1:10. Similarly, to see the effect of heterogeneous processing speed, we introduced 10 seconds of delay in one of the nodes.

We compared the performance of synchronous Hybrid-DCA (S = 4) and asynchronous Hybrid-DCA (S = 3, Γ = 10) in the two imbalanced scenarios as well as on the usual balanced-load, homogeneous-speed scenario by plotting the progress on duality gap across rounds and wall time as shown in Fig. 10. For both scenarios of imbalanced load and heterogeneous speed the performance of synchronous Hybrid-DCA degraded significantly in terms of both across round and across time. However, the asynchronous Hybrid-DCA was able to mitigate the effects of imbalance completely in terms of performance across rounds in both scenarios and across wall time in heterogeneous speed scenario. For imbalanced load, asynchronous Hybrid-DCA was not able to improve performance as the time taken by the heavily loaded node dominated the overall time.

Fig. 10.

Effect of varying workload and processing power on p = 4 worker nodes.

5.10. Performance on a big dataset

We experimented our hybrid algorithm on the big dataset splicesite of size about 280 GB and compared with the previous best algorithm CoCoA+. Because of the enormous size, the dataset could not be accommodated on a single node and hence PassCoDe could not be run on this dataset. In this experiment, we used the number of global iterations G = 1000, 000. The results are shown in Fig. 11 where the progress of duality gap across the rounds of communication is shown on the left and across the wall time on the right. To achieve a duality gap of at least 10⁻⁵ on 16 nodes, CoCoA+ took about 165 seconds. On the other hand Hybrid-DCA on 16 nodes each using 12 cores took about 29 seconds to achieve the same duality gap giving approximately 6-fold improvement, showing enough evidence about the scalability of our algorithm. One could also argue that CoCoA+ can be run on all these 16 × 12 = 192 cores, treating each core as a distributed node. However, when we experimented with this mode of CoCoA+, we found that CoCoA+ did not reached the duality gap 10⁻⁵ in a stipulated 100 rounds. We also experimented CoCoA+ on 16 × 8 = 128 cores treating each core as a node and found out that though it performed better than 16 × 1 cores in the initial few rounds but worse in the later rounds. Moreover, it was outperformed by Hybrid-DCA in terms of both the number of rounds and the time taken even on 16 nodes each using 8 cores.

Fig. 11.

Performance of Hybrid-DCA on big dataset splicesite.

6. Conclusions

In this paper, we have presented a hybrid parallel and distributed asynchronous stochastic dual coordinate ascent algorithm utilizing modern HPC platforms with many nodes of multi-core shared-memory systems. We analyze the convergence properties of this novel algorithm which uses asynchronous updates at two cascading levels: inter-cores and inter-nodes. Experimental results show that our algorithm is faster than the state-of-the-art distributed algorithms and scales better than the state-of-the-art parallel algorithms. The effectiveness of our approach in practical implementations can be increased further by a combination of (1) optimizing overhead of OpenMP threads, (2) incorporating multi-threaded implementation [21] of MPI operation, and (3) fixing the issues of larger delays in atomic memory writes for larger number of threads as given in [26].

Biographies

Soumitra Pal received a B.E.degree in Computer Science and Technology from Bengal Engineering and Science University, Howrah, India in 2000 and M.Tech. and Ph.D. degrees in Computer Science and Engineering from IIT Bombay in 2007 and 2013, respectively. He was a postdoctoral research fellow at the University of Connecticut, USA during 2014–6. He also worked with Texas Instruments, India as a Software Design Engineer during 2000–5. His research interests are in algorithm design, optimization, bioinformatics and parallel and distributed systems.

Tingyang Xu is a Ph.D. student in the Department of Computer Science and Engineering, the University of Connecticut. Since 2011, he is a member of the Laboratory of Machine Learning and Health Informatics led by Professor Jinbo Bi. His main areas of research interests are machine learning in longitudinal and time series data and high performance computing for optimization problems. He is also a student member of the Institute of Electrical and Electronics Engineers (IEEE).

Tianbao Yang is an Assistant Professor of the Computer Science Department at the University of Iowa. He received the Ph.D. degree in Computer Science from Michigan State University in 2012. He worked as a researcher in GE Global Research from 2012 to 2013 and in NEC Laboratories America, Inc. from 2013 to 2014. He has board interests in machine learning and has focused on several research topics, including social network analysis and large scale optimization in machine learning. He has won the Mark Fulk Best student paper award at 25th Conference on Learning Theory (COLT) in 2012. He also served as program committee for several conferences, including AAAI’15, AAAI’12, CIKM’12, ’13, IJCAI’13, ACML’12.

Sanguthevar Rajasekaran received his M.E. degree in Automation from the Indian Institute of Science (Bangalore) in 1983, and his Ph.D. degree in Computer Science from Harvard University in 1988. Currently he is the UTC Chair Professor of Computer Science and Engineering at the University of Connecticut and the Director of Booth Engineering Center for Advanced Technologies (BECAT). Before joining UConn, he has served as a faculty member in the CISE Department of the University of Florida and in the CIS Department of the University of Pennsylvania. During 2000–2002 he was the Chief Scientist for Arcot Systems. His research interests include Parallel Algorithms, Bioinformatics, Data Mining, Randomized Computing, Computer Simulations and Combinatorial Optimization. He has published over 150 articles in journals and conferences. He has co-authored two texts on algorithms and co-edited four books on algorithms and related topics. He is an IEEE Fellow and an elected member of the Connecticut Academy of Science and Engineering.

Jinbo Bi received a Ph.D. degree in mathematics from Rensselaer Polytechnic Institute, USA, and a master degree in Electrical Engineering and Automatic Control from Beijing Institute of Technology, China. She is an associate professor of Computer Science and Engineering at the University of Connecticut. Prior to her current appointment, she worked with Siemens Medical Solutions on computer aided diagnosis research and Partners Healthcare on clinical decision support systems. Her research interests include machine learning, data mining, bioinformatics and biomedical informatics.