Rapid Switching and Multi-Adapter Fusion via Sparse High Rank Adapters
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866914883395649536 |
|---|---|
| author | Bhardwaj, Kartikeya Pandey, Nilesh Prasad Priyadarshi, Sweta Ganapathy, Viswanath Esteves, Rafael Kadambi, Shreya Borse, Shubhankar Whatmough, Paul Garrepalli, Risheek Van Baalen, Mart Teague, Harris Nagel, Markus |
| author_facet | Bhardwaj, Kartikeya Pandey, Nilesh Prasad Priyadarshi, Sweta Ganapathy, Viswanath Esteves, Rafael Kadambi, Shreya Borse, Shubhankar Whatmough, Paul Garrepalli, Risheek Van Baalen, Mart Teague, Harris Nagel, Markus |
| contents | In this paper, we propose Sparse High Rank Adapters (SHiRA) that directly finetune 1-2% of the base model weights while leaving others unchanged, thus, resulting in a highly sparse adapter. This high sparsity incurs no inference overhead, enables rapid switching directly in the fused mode, and significantly reduces concept-loss during multi-adapter fusion. Our extensive experiments on LVMs and LLMs demonstrate that finetuning merely 1-2% parameters in the base model is sufficient for many adapter tasks and significantly outperforms Low Rank Adaptation (LoRA). We also show that SHiRA is orthogonal to advanced LoRA methods such as DoRA and can be easily combined with existing techniques. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_16712 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Rapid Switching and Multi-Adapter Fusion via Sparse High Rank Adapters Bhardwaj, Kartikeya Pandey, Nilesh Prasad Priyadarshi, Sweta Ganapathy, Viswanath Esteves, Rafael Kadambi, Shreya Borse, Shubhankar Whatmough, Paul Garrepalli, Risheek Van Baalen, Mart Teague, Harris Nagel, Markus Machine Learning In this paper, we propose Sparse High Rank Adapters (SHiRA) that directly finetune 1-2% of the base model weights while leaving others unchanged, thus, resulting in a highly sparse adapter. This high sparsity incurs no inference overhead, enables rapid switching directly in the fused mode, and significantly reduces concept-loss during multi-adapter fusion. Our extensive experiments on LVMs and LLMs demonstrate that finetuning merely 1-2% parameters in the base model is sufficient for many adapter tasks and significantly outperforms Low Rank Adaptation (LoRA). We also show that SHiRA is orthogonal to advanced LoRA methods such as DoRA and can be easily combined with existing techniques. |
| title | Rapid Switching and Multi-Adapter Fusion via Sparse High Rank Adapters |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2407.16712 |