Sparse Incremental Aggregation in Multi-Hop Federated Learning
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918194771394560 |
|---|---|
| author | Mukherjee, Sourav Razmi, Nasrin Dekorsy, Armin Popovski, Petar Matthiesen, Bho |
| author_facet | Mukherjee, Sourav Razmi, Nasrin Dekorsy, Armin Popovski, Petar Matthiesen, Bho |
| contents | This paper investigates federated learning (FL) in a multi-hop communication setup, such as in constellations with inter-satellite links. In this setup, part of the FL clients are responsible for forwarding other client's results to the parameter server. Instead of using conventional routing, the communication efficiency can be improved significantly by using in-network model aggregation at each intermediate hop, known as incremental aggregation (IA). Prior works [1] have indicated diminishing gains for IA under gradient sparsification. Here we study this issue and propose several novel correlated sparsification methods for IA. Numerical results show that, for some of these algorithms, the full potential of IA is still available under sparsification without impairing convergence. We demonstrate a 15x improvement in communication efficiency over conventional routing and a 11x improvement over state-of-the-art (SoA) sparse IA. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_18200 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Sparse Incremental Aggregation in Multi-Hop Federated Learning Mukherjee, Sourav Razmi, Nasrin Dekorsy, Armin Popovski, Petar Matthiesen, Bho Distributed, Parallel, and Cluster Computing Machine Learning Signal Processing This paper investigates federated learning (FL) in a multi-hop communication setup, such as in constellations with inter-satellite links. In this setup, part of the FL clients are responsible for forwarding other client's results to the parameter server. Instead of using conventional routing, the communication efficiency can be improved significantly by using in-network model aggregation at each intermediate hop, known as incremental aggregation (IA). Prior works [1] have indicated diminishing gains for IA under gradient sparsification. Here we study this issue and propose several novel correlated sparsification methods for IA. Numerical results show that, for some of these algorithms, the full potential of IA is still available under sparsification without impairing convergence. We demonstrate a 15x improvement in communication efficiency over conventional routing and a 11x improvement over state-of-the-art (SoA) sparse IA. |
| title | Sparse Incremental Aggregation in Multi-Hop Federated Learning |
| topic | Distributed, Parallel, and Cluster Computing Machine Learning Signal Processing |
| url | https://arxiv.org/abs/2407.18200 |