Algorithm 3 Mini-batch stochastic I-ADMM (sI-ADMM)

1:initialize: $\{z^1=x_i^1=y_i^1=\mathbf{0},|i\in \mathcal{N}\}$, batch size M; 2:Local Data Allocation: 3:for agent $i\in \mathcal{N}$ do 4:    divide ${\mathcal{D}}_i$ labeled data into $K_i$ equally disjoint partitions and denote each partition as ${\mathbf{\xi }}_{i,j},j\in {\mathcal{K}}_i$; 5:    for ECN $j\in {\mathcal{K}}_i$ do 6:        allocate ${\mathbf{\xi }}_{i,j}$ to ECN j; 7:        partition ${\mathbf{\xi }}_{i,j}$ examples into multiple batches with each size $M/K_i$; 8:    end for 9:end for 10:Updating Process: 11:for$k=1,2,\dots$do 12:    Steps of Active Agent $i=i_k=(k-1)modN+1$: 13:    receive token $z^k$; 14:    broadcast local variable $x_i^k$ to ECNs ${\mathcal{K}}_i$; 15:    ECN $j\in {\mathcal{K}}_i$ computes gradient in parallel: 16:      receive local primal variable $x_i^k$; 17:      select batch $I_{i,j}^k=mmod\lfloor |{\mathbf{\xi }}_{i,j}|·K_i/M\rfloor$; 18:      update gradient $g_{i,j}=\frac{K_i}{M}{\sum }_{l=1}^{\frac{M}{K_i}}\nabla F_i(x_i^k;{\xi }_{i,l}^k)$; 19:      transmit $g_{i,j}$ to the connected agent; 20:    until the $K_i$-th responded message is received; 21:     update gradient via gradient summation: $${\mathcal{G}}_i(x_i^k;{\mathbf{\xi }}_i^k)=\frac{1}{K_i}\sum _{j=1}^{K_i}{g}_{i,j};$$ (17) 22:    update ${\mathbf{x}}^{k+1}$ according to (16a); 23:    update ${\mathbf{y}}^{k+1}$ according to (16b); 24:    update $z^{k+1}$ according to (15c); 25:    send token $z^{k+1}$ to agent $i_{k+1}$ via link ($i_k,i_{k+1}$); 26:    until the stopping criterion is satisfied. 27:end for