FedMuon: Federated Learning with Bias-corrected LMO-based Optimization
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866912618231365632 |
|---|---|
| author | Takezawa, Yuki Koloskova, Anastasia Jiang, Xiaowen Stich, Sebastian U. |
| author_facet | Takezawa, Yuki Koloskova, Anastasia Jiang, Xiaowen Stich, Sebastian U. |
| contents | Recently, a new optimization method based on the linear minimization oracle (LMO), called Muon, has been attracting increasing attention since it can train neural networks faster than existing adaptive optimization methods, such as Adam. In this paper, we study how Muon can be utilized in federated learning. We first show that straightforwardly using Muon as the local optimizer of FedAvg does not converge to the stationary point since the LMO is a biased operator. We then propose FedMuon which can mitigate this issue. We also analyze how solving the LMO approximately affects the convergence rate and find that, surprisingly, FedMuon can converge for any number of Newton-Schulz iterations, while it can converge faster as we solve the LMO more accurately. Through experiments, we demonstrated that FedMuon can outperform the state-of-the-art federated learning methods. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_26337 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | FedMuon: Federated Learning with Bias-corrected LMO-based Optimization Takezawa, Yuki Koloskova, Anastasia Jiang, Xiaowen Stich, Sebastian U. Machine Learning Optimization and Control Recently, a new optimization method based on the linear minimization oracle (LMO), called Muon, has been attracting increasing attention since it can train neural networks faster than existing adaptive optimization methods, such as Adam. In this paper, we study how Muon can be utilized in federated learning. We first show that straightforwardly using Muon as the local optimizer of FedAvg does not converge to the stationary point since the LMO is a biased operator. We then propose FedMuon which can mitigate this issue. We also analyze how solving the LMO approximately affects the convergence rate and find that, surprisingly, FedMuon can converge for any number of Newton-Schulz iterations, while it can converge faster as we solve the LMO more accurately. Through experiments, we demonstrated that FedMuon can outperform the state-of-the-art federated learning methods. |
| title | FedMuon: Federated Learning with Bias-corrected LMO-based Optimization |
| topic | Machine Learning Optimization and Control |
| url | https://arxiv.org/abs/2509.26337 |