GiVA: Gradient-Informed Bases for Vector-Based Adaptation
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866913058033500160 |
|---|---|
| author | Gangwar, Neeraj Deshmukh, Rishabh Shavlovsky, Michael Li, Hancao Mittal, Vivek Ying, Lexing Kani, Nickvash |
| author_facet | Gangwar, Neeraj Deshmukh, Rishabh Shavlovsky, Michael Li, Hancao Mittal, Vivek Ying, Lexing Kani, Nickvash |
| contents | As model sizes continue to grow, parameter-efficient fine-tuning has emerged as a powerful alternative to full fine-tuning. While LoRA is widely adopted among these methods, recent research has explored vector-based adaptation methods due to their extreme parameter efficiency. However, these methods typically require substantially higher ranks than LoRA to match its performance, leading to increased training costs. This work introduces GiVA, a gradient-based initialization strategy for vector-based adaptation. It achieves training times comparable to LoRA and maintains the extreme parameter efficiency of vector-based adaptation. We evaluate GiVA across diverse benchmarks, including natural language understanding, natural language generation, and image classification. Experiments show that our approach consistently outperforms or achieves performance competitive with existing vector-based adaptation methods and LoRA while reducing rank requirements by a factor of eight ($8\times$). |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_21901 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | GiVA: Gradient-Informed Bases for Vector-Based Adaptation Gangwar, Neeraj Deshmukh, Rishabh Shavlovsky, Michael Li, Hancao Mittal, Vivek Ying, Lexing Kani, Nickvash Computation and Language Artificial Intelligence As model sizes continue to grow, parameter-efficient fine-tuning has emerged as a powerful alternative to full fine-tuning. While LoRA is widely adopted among these methods, recent research has explored vector-based adaptation methods due to their extreme parameter efficiency. However, these methods typically require substantially higher ranks than LoRA to match its performance, leading to increased training costs. This work introduces GiVA, a gradient-based initialization strategy for vector-based adaptation. It achieves training times comparable to LoRA and maintains the extreme parameter efficiency of vector-based adaptation. We evaluate GiVA across diverse benchmarks, including natural language understanding, natural language generation, and image classification. Experiments show that our approach consistently outperforms or achieves performance competitive with existing vector-based adaptation methods and LoRA while reducing rank requirements by a factor of eight ($8\times$). |
| title | GiVA: Gradient-Informed Bases for Vector-Based Adaptation |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2604.21901 |