A Note on the Convergence of Muon
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866913869265371136 |
|---|---|
| author | Li, Jiaxiang Hong, Mingyi |
| author_facet | Li, Jiaxiang Hong, Mingyi |
| contents | In this note, we inspect the convergence of a new optimizer for pretraining LLMs, namely the Muon optimizer. Such an optimizer is closely related to a specialized steepest descent method where the update direction is the minimizer of the quadratic approximation of the objective function under spectral norm. We provide the convergence analysis on both versions of the optimizer and discuss its implications. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_02900 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | A Note on the Convergence of Muon Li, Jiaxiang Hong, Mingyi Optimization and Control In this note, we inspect the convergence of a new optimizer for pretraining LLMs, namely the Muon optimizer. Such an optimizer is closely related to a specialized steepest descent method where the update direction is the minimizer of the quadratic approximation of the objective function under spectral norm. We provide the convergence analysis on both versions of the optimizer and discuss its implications. |
| title | A Note on the Convergence of Muon |
| topic | Optimization and Control |
| url | https://arxiv.org/abs/2502.02900 |