论文笔记(7)-"Local Newton: Reducing Communication Bottleneck for Distributed Learning"

Main idea

Authors demonstrated a second-order optimization method and incorporated the curvature information to reduce the communication cost.

Algorithm

They have proposed two algorithm: in the first algorithm, there is only once gradient direction computation, i.e., $L=1$

$L\ge 1$ in the second algorithm, called adaptive Local Newton:

Assumption and convergence analysis

In their work, only the weighted function $f$ is required to be $M$ smooth and $\kappa$ strongly convex and the norm of the local gradient $\Vert \mathrm{\nabla }{f}_i(x)\Vert$ is upper bounded.

Actually, they work on the homogeneous setting: local data sets comes from the same distribution and $X_i$ is the submatrix of $X$. Based on it, they manifested local function $f_i$ is also $M(1-ϵ)$ smooth and $\kappa (1+ϵ)$ strongly convex ($\mathbf{\text{Lemma 4.1}}$) with probability at least $1-\delta$.

As they said the sketch of the proof is following:

1. Reduce $f({\overline{w}}_{t+1})-f({\overline{w}}_t)$ to $\frac{1}{K}\sum _i{f}_i(w_{t+1}^i)-f_i(w_t^i)$
2. Bound $f_i(w_{t+1}^i)-f_i(w_t^i)\le -\phi \Vert g_t^i{\Vert }^2$($\mathbf{\text{Lemma A.2}}$)
3. local gradient $g_t^i$ is closed to$g_t$ by perturbed iterate analysis

conclusion

• The underlying homogeneous assumption is impractical in federated learning.

• For DL, solving the hessian matrix is difficult and approximation is more applicable.

• At last, I want to briefly introduce the Giant: At first, the server will broadcast the weighted gradient $g$ to users to compute local Hessian matrix.

  

Reference

• Local Newton: Reducing Communication Bottleneck for Distributed Learning
• GIANT: Globally Improved Approximate Newton Method for Distributed Optimization