PoLAR: Polar-Decomposed Low-Rank Adapter Representation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918180094476288 |
|---|---|
| author | Lion, Kai Zhang, Liang Li, Bingcong He, Niao |
| author_facet | Lion, Kai Zhang, Liang Li, Bingcong He, Niao |
| contents | We show that low-rank adaptation of large-scale models suffers from a low stable rank that is well below the linear algebraic rank of the subspace, degrading fine-tuning performance. To mitigate the underutilization of the allocated subspace, we propose PoLAR, a parameterization inspired by the polar decomposition that factorizes the low-rank update into two direction matrices constrained to Stiefel manifolds and an unconstrained scale matrix. Our theory shows that PoLAR yields an exponentially faster convergence rate on a canonical low-rank adaptation problem. Pairing the parameterization with Riemannian optimization leads to consistent gains on three different benchmarks testing general language understanding, commonsense reasoning, and mathematical problem solving with base model sizes ranging from 350M to 27B. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_03133 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | PoLAR: Polar-Decomposed Low-Rank Adapter Representation Lion, Kai Zhang, Liang Li, Bingcong He, Niao Machine Learning Artificial Intelligence Signal Processing Optimization and Control We show that low-rank adaptation of large-scale models suffers from a low stable rank that is well below the linear algebraic rank of the subspace, degrading fine-tuning performance. To mitigate the underutilization of the allocated subspace, we propose PoLAR, a parameterization inspired by the polar decomposition that factorizes the low-rank update into two direction matrices constrained to Stiefel manifolds and an unconstrained scale matrix. Our theory shows that PoLAR yields an exponentially faster convergence rate on a canonical low-rank adaptation problem. Pairing the parameterization with Riemannian optimization leads to consistent gains on three different benchmarks testing general language understanding, commonsense reasoning, and mathematical problem solving with base model sizes ranging from 350M to 27B. |
| title | PoLAR: Polar-Decomposed Low-Rank Adapter Representation |
| topic | Machine Learning Artificial Intelligence Signal Processing Optimization and Control |
| url | https://arxiv.org/abs/2506.03133 |