Algorithm 1.

ϵ-Differentially Private DOLA

1:	Input: $f_t^i(w):=\mathcal{l}\left(w,x_t^i,y_t^i\right),i\in [1,m]$ and $t\in [0,T]$; initial points $w_0^1,\dots ,w_0^m$; doubly stochastic matrix $A_t\text{=}\left(a_{ij}(t)\right)\in R^{m\times m}$; maximum iterations T.
2:	for $t=0,\dots ,T$ do
3:	for each node $i=1,\dots ,m$ do
4:	$b_t^i=\sum _{j=1}^m{a}_{ij}(t+1)\left(w_t^i+{\sigma }_t^i\right)$, where ${\sigma }_t^j$ is a Laplace noise vector in ${ℝ}^n$;
5:	$g_t^j\leftarrow \nabla f_t^i\left(b_t^i,{\xi }_t^i\right)$;
6:	$w_{t+1}^i=\text{P}\mathrm{ro}\left[b_t^i-{\alpha }_{t+1}\cdot g_t^i\right]$, where ${\sigma }_t^i~\mathit{Lap}(\mu )$;
	(Projection onto W)  ($Lap(\mu )$ defined in (11))
7:	broadcast the output $\left(w_{t+1}^i+{\sigma }_{t+1}^i\right)\text{to}\mathcal{G}{(t+1)}_i$;
8:	end for
9:	end for