The Entrapment Problem in Random Walk Decentralized Learning
Fuente:
arXiv
Salvato in:
| Autori principali: | , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866929441798619136 |
|---|---|
| author | Liu, Zonghong Rouayheb, Salim El Dwyer, Matthew |
| author_facet | Liu, Zonghong Rouayheb, Salim El Dwyer, Matthew |
| contents | This paper explores decentralized learning in a graph-based setting, where data is distributed across nodes. We investigate a decentralized SGD algorithm that utilizes a random walk to update a global model based on local data. Our focus is on designing the transition probability matrix to speed up convergence. While importance sampling can enhance centralized learning, its decentralized counterpart, using the Metropolis-Hastings (MH) algorithm, can lead to the entrapment problem, where the random walk becomes stuck at certain nodes, slowing convergence. To address this, we propose the Metropolis-Hastings with Lévy Jumps (MHLJ) algorithm, which incorporates random perturbations (jumps) to overcome entrapment. We theoretically establish the convergence rate and error gap of MHLJ and validate our findings through numerical experiments. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_20611 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | The Entrapment Problem in Random Walk Decentralized Learning Liu, Zonghong Rouayheb, Salim El Dwyer, Matthew Machine Learning Distributed, Parallel, and Cluster Computing Information Theory This paper explores decentralized learning in a graph-based setting, where data is distributed across nodes. We investigate a decentralized SGD algorithm that utilizes a random walk to update a global model based on local data. Our focus is on designing the transition probability matrix to speed up convergence. While importance sampling can enhance centralized learning, its decentralized counterpart, using the Metropolis-Hastings (MH) algorithm, can lead to the entrapment problem, where the random walk becomes stuck at certain nodes, slowing convergence. To address this, we propose the Metropolis-Hastings with Lévy Jumps (MHLJ) algorithm, which incorporates random perturbations (jumps) to overcome entrapment. We theoretically establish the convergence rate and error gap of MHLJ and validate our findings through numerical experiments. |
| title | The Entrapment Problem in Random Walk Decentralized Learning |
| topic | Machine Learning Distributed, Parallel, and Cluster Computing Information Theory |
| url | https://arxiv.org/abs/2407.20611 |