Fed-Sophia: A Communication-Efficient Second-Order Federated Learning Algorithm
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866916282480197632 |
|---|---|
| author | Elbakary, Ahmed Issaid, Chaouki Ben Shehab, Mohammad Seddik, Karim ElBatt, Tamer Bennis, Mehdi |
| author_facet | Elbakary, Ahmed Issaid, Chaouki Ben Shehab, Mohammad Seddik, Karim ElBatt, Tamer Bennis, Mehdi |
| contents | Federated learning is a machine learning approach where multiple devices collaboratively learn with the help of a parameter server by sharing only their local updates. While gradient-based optimization techniques are widely adopted in this domain, the curvature information that second-order methods exhibit is crucial to guide and speed up the convergence. This paper introduces a scalable second-order method, allowing the adoption of curvature information in federated large models. Our method, coined Fed-Sophia, combines a weighted moving average of the gradient with a clipping operation to find the descent direction. In addition to that, a lightweight estimation of the Hessian's diagonal is used to incorporate the curvature information. Numerical evaluation shows the superiority, robustness, and scalability of the proposed Fed-Sophia scheme compared to first and second-order baselines. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_06655 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Fed-Sophia: A Communication-Efficient Second-Order Federated Learning Algorithm Elbakary, Ahmed Issaid, Chaouki Ben Shehab, Mohammad Seddik, Karim ElBatt, Tamer Bennis, Mehdi Machine Learning Artificial Intelligence Distributed, Parallel, and Cluster Computing Federated learning is a machine learning approach where multiple devices collaboratively learn with the help of a parameter server by sharing only their local updates. While gradient-based optimization techniques are widely adopted in this domain, the curvature information that second-order methods exhibit is crucial to guide and speed up the convergence. This paper introduces a scalable second-order method, allowing the adoption of curvature information in federated large models. Our method, coined Fed-Sophia, combines a weighted moving average of the gradient with a clipping operation to find the descent direction. In addition to that, a lightweight estimation of the Hessian's diagonal is used to incorporate the curvature information. Numerical evaluation shows the superiority, robustness, and scalability of the proposed Fed-Sophia scheme compared to first and second-order baselines. |
| title | Fed-Sophia: A Communication-Efficient Second-Order Federated Learning Algorithm |
| topic | Machine Learning Artificial Intelligence Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2406.06655 |