ASLoRA: Adaptive Sharing Low-Rank Adaptation Across Layers
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866915065216630784 |
|---|---|
| author | Hu, Junyan Xiao, Xue Zhang, Mengqi Chen, Yao Ren, Zhaochun Chen, Zhumin Ren, Pengjie |
| author_facet | Hu, Junyan Xiao, Xue Zhang, Mengqi Chen, Yao Ren, Zhaochun Chen, Zhumin Ren, Pengjie |
| contents | As large language models (LLMs) grow in size, traditional full fine-tuning becomes increasingly impractical due to its high computational and storage costs. Although popular parameter-efficient fine-tuning methods, such as LoRA, have significantly reduced the number of tunable parameters, there is still room for further optimization. In this work, we propose ASLoRA, a cross-layer parameter-sharing strategy combining global sharing with partial adaptive sharing. Specifically, we share the low-rank matrix A across all layers and adaptively merge matrix B during training. This sharing mechanism not only mitigates overfitting effectively but also captures inter-layer dependencies, significantly enhancing the model's representational capability. We conduct extensive experiments on various NLP tasks, showing that ASLoRA outperforms LoRA while using less than 25% of the parameters, highlighting its flexibility and superior parameter efficiency. Furthermore, in-depth analyses of the adaptive sharing strategy confirm its significant advantages in enhancing both model flexibility and task adaptability. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_10135 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | ASLoRA: Adaptive Sharing Low-Rank Adaptation Across Layers Hu, Junyan Xiao, Xue Zhang, Mengqi Chen, Yao Ren, Zhaochun Chen, Zhumin Ren, Pengjie Computation and Language As large language models (LLMs) grow in size, traditional full fine-tuning becomes increasingly impractical due to its high computational and storage costs. Although popular parameter-efficient fine-tuning methods, such as LoRA, have significantly reduced the number of tunable parameters, there is still room for further optimization. In this work, we propose ASLoRA, a cross-layer parameter-sharing strategy combining global sharing with partial adaptive sharing. Specifically, we share the low-rank matrix A across all layers and adaptively merge matrix B during training. This sharing mechanism not only mitigates overfitting effectively but also captures inter-layer dependencies, significantly enhancing the model's representational capability. We conduct extensive experiments on various NLP tasks, showing that ASLoRA outperforms LoRA while using less than 25% of the parameters, highlighting its flexibility and superior parameter efficiency. Furthermore, in-depth analyses of the adaptive sharing strategy confirm its significant advantages in enhancing both model flexibility and task adaptability. |
| title | ASLoRA: Adaptive Sharing Low-Rank Adaptation Across Layers |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2412.10135 |