Effective Quantization of Muon Optimizer States
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866917287281295360 |
|---|---|
| author | Gupta, Aman Celente, Rafael Shivanna, Abhishek Braithwaite, D. T. Dexter, Gregory Tang, Shao Udagawa, Hiroto Silva, Daniel Ramanath, Rohan Keerthi, S. Sathiya |
| author_facet | Gupta, Aman Celente, Rafael Shivanna, Abhishek Braithwaite, D. T. Dexter, Gregory Tang, Shao Udagawa, Hiroto Silva, Daniel Ramanath, Rohan Keerthi, S. Sathiya |
| contents | The Muon optimizer, based on matrix orthogonalization, has recently shown faster convergence and better computational efficiency over AdamW in LLM pre-training. However, the memory overhead of maintaining high-precision optimizer states remains a challenge for large-scale deployment. In this paper, we introduce the 8-bit Muon optimizer using blockwise quantization.
In extensive Chinchilla-optimal experiments on pre-training models of up to 2.7B in size and fine-tuning them for instruction following, we demonstrate that 8-bit Muon achieves parity with Muon in terms of validation loss and downstream benchmarks, while achieving up to a 62\% reduction in optimizer state footprint. Crucially, we show that Muon's update mechanism is uniquely compatible with a simple linear quantization scheme, bypassing the complex dynamic scaling required for quantized AdamW. We supplement our empirical findings with a theoretical analysis of Muon's robustness to quantization noise. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_23106 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Effective Quantization of Muon Optimizer States Gupta, Aman Celente, Rafael Shivanna, Abhishek Braithwaite, D. T. Dexter, Gregory Tang, Shao Udagawa, Hiroto Silva, Daniel Ramanath, Rohan Keerthi, S. Sathiya Machine Learning The Muon optimizer, based on matrix orthogonalization, has recently shown faster convergence and better computational efficiency over AdamW in LLM pre-training. However, the memory overhead of maintaining high-precision optimizer states remains a challenge for large-scale deployment. In this paper, we introduce the 8-bit Muon optimizer using blockwise quantization. In extensive Chinchilla-optimal experiments on pre-training models of up to 2.7B in size and fine-tuning them for instruction following, we demonstrate that 8-bit Muon achieves parity with Muon in terms of validation loss and downstream benchmarks, while achieving up to a 62\% reduction in optimizer state footprint. Crucially, we show that Muon's update mechanism is uniquely compatible with a simple linear quantization scheme, bypassing the complex dynamic scaling required for quantized AdamW. We supplement our empirical findings with a theoretical analysis of Muon's robustness to quantization noise. |
| title | Effective Quantization of Muon Optimizer States |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2509.23106 |