Log-Scale Quantization in Distributed First-Order Methods: Gradient-based Learning from Distributed Data
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910779230388224 |
|---|---|
| author | Doostmohammadian, Mohammadreza Qureshi, Muhammad I. Khalesi, Mohammad Hossein Rabiee, Hamid R. Khan, Usman A. |
| author_facet | Doostmohammadian, Mohammadreza Qureshi, Muhammad I. Khalesi, Mohammad Hossein Rabiee, Hamid R. Khan, Usman A. |
| contents | Decentralized strategies are of interest for learning from large-scale data over networks. This paper studies learning over a network of geographically distributed nodes/agents subject to quantization. Each node possesses a private local cost function, collectively contributing to a global cost function, which the considered methodology aims to minimize. In contrast to many existing papers, the information exchange among nodes is log-quantized to address limited network-bandwidth in practical situations. We consider a first-order computationally efficient distributed optimization algorithm (with no extra inner consensus loop) that leverages node-level gradient correction based on local data and network-level gradient aggregation only over nearby nodes. This method only requires balanced networks with no need for stochastic weight design. It can handle log-scale quantized data exchange over possibly time-varying and switching network setups. We study convergence over both structured networks (for example, training over data-centers) and ad-hoc multi-agent networks (for example, training over dynamic robotic networks). Through experimental validation, we show that (i) structured networks generally result in a smaller optimality gap, and (ii) log-scale quantization leads to a smaller optimality gap compared to uniform quantization. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_00621 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Log-Scale Quantization in Distributed First-Order Methods: Gradient-based Learning from Distributed Data Doostmohammadian, Mohammadreza Qureshi, Muhammad I. Khalesi, Mohammad Hossein Rabiee, Hamid R. Khan, Usman A. Systems and Control Signal Processing Optimization and Control Decentralized strategies are of interest for learning from large-scale data over networks. This paper studies learning over a network of geographically distributed nodes/agents subject to quantization. Each node possesses a private local cost function, collectively contributing to a global cost function, which the considered methodology aims to minimize. In contrast to many existing papers, the information exchange among nodes is log-quantized to address limited network-bandwidth in practical situations. We consider a first-order computationally efficient distributed optimization algorithm (with no extra inner consensus loop) that leverages node-level gradient correction based on local data and network-level gradient aggregation only over nearby nodes. This method only requires balanced networks with no need for stochastic weight design. It can handle log-scale quantized data exchange over possibly time-varying and switching network setups. We study convergence over both structured networks (for example, training over data-centers) and ad-hoc multi-agent networks (for example, training over dynamic robotic networks). Through experimental validation, we show that (i) structured networks generally result in a smaller optimality gap, and (ii) log-scale quantization leads to a smaller optimality gap compared to uniform quantization. |
| title | Log-Scale Quantization in Distributed First-Order Methods: Gradient-based Learning from Distributed Data |
| topic | Systems and Control Signal Processing Optimization and Control |
| url | https://arxiv.org/abs/2406.00621 |