Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2405.17471 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913367965302784 |
|---|---|
| author | Yue, Sheng Hua, Xingyuan Chen, Lili Ren, Ju |
| author_facet | Yue, Sheng Hua, Xingyuan Chen, Lili Ren, Ju |
| contents | Federated Reinforcement Learning (FRL) has garnered increasing attention recently. However, due to the intrinsic spatio-temporal non-stationarity of data distributions, the current approaches typically suffer from high interaction and communication costs. In this paper, we introduce a new FRL algorithm, named $\texttt{MFPO}$, that utilizes momentum, importance sampling, and additional server-side adjustment to control the shift of stochastic policy gradients and enhance the efficiency of data utilization. We prove that by proper selection of momentum parameters and interaction frequency, $\texttt{MFPO}$ can achieve $\tilde{\mathcal{O}}(H N^{-1}ε^{-3/2})$ and $\tilde{\mathcal{O}}(ε^{-1})$ interaction and communication complexities ($N$ represents the number of agents), where the interaction complexity achieves linear speedup with the number of agents, and the communication complexity aligns the best achievable of existing first-order FL algorithms. Extensive experiments corroborate the substantial performance gains of $\texttt{MFPO}$ over existing methods on a suite of complex and high-dimensional benchmarks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_17471 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Momentum-Based Federated Reinforcement Learning with Interaction and Communication Efficiency Yue, Sheng Hua, Xingyuan Chen, Lili Ren, Ju Machine Learning Artificial Intelligence Federated Reinforcement Learning (FRL) has garnered increasing attention recently. However, due to the intrinsic spatio-temporal non-stationarity of data distributions, the current approaches typically suffer from high interaction and communication costs. In this paper, we introduce a new FRL algorithm, named $\texttt{MFPO}$, that utilizes momentum, importance sampling, and additional server-side adjustment to control the shift of stochastic policy gradients and enhance the efficiency of data utilization. We prove that by proper selection of momentum parameters and interaction frequency, $\texttt{MFPO}$ can achieve $\tilde{\mathcal{O}}(H N^{-1}ε^{-3/2})$ and $\tilde{\mathcal{O}}(ε^{-1})$ interaction and communication complexities ($N$ represents the number of agents), where the interaction complexity achieves linear speedup with the number of agents, and the communication complexity aligns the best achievable of existing first-order FL algorithms. Extensive experiments corroborate the substantial performance gains of $\texttt{MFPO}$ over existing methods on a suite of complex and high-dimensional benchmarks. |
| title | Momentum-Based Federated Reinforcement Learning with Interaction and Communication Efficiency |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2405.17471 |