Averaging Rate Scheduler for Decentralized Learning on Heterogeneous Data
Fuente:
arXiv
Guardado en:
| Autores principales: | , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866909129382035456 |
|---|---|
| author | Aketi, Sai Aparna Choudhary, Sakshi Roy, Kaushik |
| author_facet | Aketi, Sai Aparna Choudhary, Sakshi Roy, Kaushik |
| contents | State-of-the-art decentralized learning algorithms typically require the data distribution to be Independent and Identically Distributed (IID). However, in practical scenarios, the data distribution across the agents can have significant heterogeneity. In this work, we propose averaging rate scheduling as a simple yet effective way to reduce the impact of heterogeneity in decentralized learning. Our experiments illustrate the superiority of the proposed method (~3% improvement in test accuracy) compared to the conventional approach of employing a constant averaging rate. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_03292 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Averaging Rate Scheduler for Decentralized Learning on Heterogeneous Data Aketi, Sai Aparna Choudhary, Sakshi Roy, Kaushik Machine Learning Distributed, Parallel, and Cluster Computing State-of-the-art decentralized learning algorithms typically require the data distribution to be Independent and Identically Distributed (IID). However, in practical scenarios, the data distribution across the agents can have significant heterogeneity. In this work, we propose averaging rate scheduling as a simple yet effective way to reduce the impact of heterogeneity in decentralized learning. Our experiments illustrate the superiority of the proposed method (~3% improvement in test accuracy) compared to the conventional approach of employing a constant averaging rate. |
| title | Averaging Rate Scheduler for Decentralized Learning on Heterogeneous Data |
| topic | Machine Learning Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2403.03292 |