Riemannian Optimization for LoRA on the Stiefel Manifold
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914003911966720 |
|---|---|
| author | Park, Juneyoung Kang, Minjae Lee, Seongbae Lee, Haegang Kim, Seongwan Lee, Jaeho |
| author_facet | Park, Juneyoung Kang, Minjae Lee, Seongbae Lee, Haegang Kim, Seongwan Lee, Jaeho |
| contents | While powerful, large language models (LLMs) present significant fine-tuning challenges due to their size. Parameter-efficient fine-tuning (PEFT) methods like LoRA provide solutions, yet suffer from critical optimizer inefficiencies; notably basis redundancy in LoRA's $B$ matrix when using AdamW, which fundamentally limits performance. We address this by optimizing the $B$ matrix on the Stiefel manifold, imposing explicit orthogonality constraints that achieve near-perfect orthogonality and full effective rank. This geometric approach dramatically enhances parameter efficiency and representational capacity. Our Stiefel optimizer consistently outperforms AdamW across benchmarks with both LoRA and DoRA, demonstrating that geometric constraints are the key to unlocking LoRA's full potential for effective LLM fine-tuning. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_17901 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Riemannian Optimization for LoRA on the Stiefel Manifold Park, Juneyoung Kang, Minjae Lee, Seongbae Lee, Haegang Kim, Seongwan Lee, Jaeho Machine Learning Artificial Intelligence While powerful, large language models (LLMs) present significant fine-tuning challenges due to their size. Parameter-efficient fine-tuning (PEFT) methods like LoRA provide solutions, yet suffer from critical optimizer inefficiencies; notably basis redundancy in LoRA's $B$ matrix when using AdamW, which fundamentally limits performance. We address this by optimizing the $B$ matrix on the Stiefel manifold, imposing explicit orthogonality constraints that achieve near-perfect orthogonality and full effective rank. This geometric approach dramatically enhances parameter efficiency and representational capacity. Our Stiefel optimizer consistently outperforms AdamW across benchmarks with both LoRA and DoRA, demonstrating that geometric constraints are the key to unlocking LoRA's full potential for effective LLM fine-tuning. |
| title | Riemannian Optimization for LoRA on the Stiefel Manifold |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2508.17901 |