Serial Low-rank Adaptation of Vision Transformer
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866912289070776320 |
|---|---|
| author | Zhong, Houqiang Shen, Shaocheng Cai, Ke Wu, Zhenglong Yao, Jiangchao Cheng, Yuan Li, Xuefei Zhang, Xiaoyun Song, Li Hu, Qiang |
| author_facet | Zhong, Houqiang Shen, Shaocheng Cai, Ke Wu, Zhenglong Yao, Jiangchao Cheng, Yuan Li, Xuefei Zhang, Xiaoyun Song, Li Hu, Qiang |
| contents | Fine-tuning large pre-trained vision foundation models in a parameter-efficient manner is critical for downstream vision tasks, considering the practical constraints of computational and storage costs. Low-rank adaptation (LoRA) is a well-established technique in this domain, achieving impressive efficiency by reducing the parameter space to a low-rank form. However, developing more advanced low-rank adaptation methods to reduce parameters and memory requirements remains a significant challenge in resource-constrained application scenarios. In this study, we consider on top of the commonly used vision transformer and propose Serial LoRA, a novel LoRA variant that introduces a shared low-rank matrix serially composite with the attention mechanism. Such a design extracts the underlying commonality of parameters in adaptation, significantly reducing redundancy. Notably, Serial LoRA uses only 1/4 parameters of LoRA but achieves comparable performance in most cases. We conduct extensive experiments on a range of vision foundation models with the transformer structure, and the results confirm consistent superiority of our method. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_17750 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Serial Low-rank Adaptation of Vision Transformer Zhong, Houqiang Shen, Shaocheng Cai, Ke Wu, Zhenglong Yao, Jiangchao Cheng, Yuan Li, Xuefei Zhang, Xiaoyun Song, Li Hu, Qiang Computer Vision and Pattern Recognition Multimedia Fine-tuning large pre-trained vision foundation models in a parameter-efficient manner is critical for downstream vision tasks, considering the practical constraints of computational and storage costs. Low-rank adaptation (LoRA) is a well-established technique in this domain, achieving impressive efficiency by reducing the parameter space to a low-rank form. However, developing more advanced low-rank adaptation methods to reduce parameters and memory requirements remains a significant challenge in resource-constrained application scenarios. In this study, we consider on top of the commonly used vision transformer and propose Serial LoRA, a novel LoRA variant that introduces a shared low-rank matrix serially composite with the attention mechanism. Such a design extracts the underlying commonality of parameters in adaptation, significantly reducing redundancy. Notably, Serial LoRA uses only 1/4 parameters of LoRA but achieves comparable performance in most cases. We conduct extensive experiments on a range of vision foundation models with the transformer structure, and the results confirm consistent superiority of our method. |
| title | Serial Low-rank Adaptation of Vision Transformer |
| topic | Computer Vision and Pattern Recognition Multimedia |
| url | https://arxiv.org/abs/2503.17750 |