LoRA meets Riemannion: Muon Optimizer for Parametrization-independent Low-Rank Adapters
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866912619871338496 |
|---|---|
| author | Bogachev, Vladimir Aletov, Vladimir Molozhavenko, Alexander Bobkov, Denis Soboleva, Vera Alanov, Aibek Rakhuba, Maxim |
| author_facet | Bogachev, Vladimir Aletov, Vladimir Molozhavenko, Alexander Bobkov, Denis Soboleva, Vera Alanov, Aibek Rakhuba, Maxim |
| contents | This work presents a novel, fully Riemannian framework for Low-Rank Adaptation (LoRA) that geometrically treats low-rank adapters by optimizing them directly on the fixed-rank manifold. This formulation eliminates the parametrization ambiguity present in standard Euclidean optimizers. Our framework integrates three key components to achieve this: (1) we derive Riemannion, a new Riemannian optimizer on the fixed-rank matrix manifold that generalizes the recently proposed Muon optimizer; (2) we develop a Riemannian gradient-informed LoRA initialization, and (3) we provide an efficient implementation without prominent overhead that uses automatic differentiation to compute arising geometric operations while adhering to best practices in numerical linear algebra. Comprehensive experimental results on both LLM and diffusion model architectures demonstrate that our approach yields consistent and noticeable improvements in convergence speed and final task performance over both standard LoRA and its state-of-the-art modifications. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_12142 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | LoRA meets Riemannion: Muon Optimizer for Parametrization-independent Low-Rank Adapters Bogachev, Vladimir Aletov, Vladimir Molozhavenko, Alexander Bobkov, Denis Soboleva, Vera Alanov, Aibek Rakhuba, Maxim Machine Learning Computation and Language Numerical Analysis Differential Geometry 68T07, 65F55, 53Z50 This work presents a novel, fully Riemannian framework for Low-Rank Adaptation (LoRA) that geometrically treats low-rank adapters by optimizing them directly on the fixed-rank manifold. This formulation eliminates the parametrization ambiguity present in standard Euclidean optimizers. Our framework integrates three key components to achieve this: (1) we derive Riemannion, a new Riemannian optimizer on the fixed-rank matrix manifold that generalizes the recently proposed Muon optimizer; (2) we develop a Riemannian gradient-informed LoRA initialization, and (3) we provide an efficient implementation without prominent overhead that uses automatic differentiation to compute arising geometric operations while adhering to best practices in numerical linear algebra. Comprehensive experimental results on both LLM and diffusion model architectures demonstrate that our approach yields consistent and noticeable improvements in convergence speed and final task performance over both standard LoRA and its state-of-the-art modifications. |
| title | LoRA meets Riemannion: Muon Optimizer for Parametrization-independent Low-Rank Adapters |
| topic | Machine Learning Computation and Language Numerical Analysis Differential Geometry 68T07, 65F55, 53Z50 |
| url | https://arxiv.org/abs/2507.12142 |