ResLoRA: Identity Residual Mapping in Low-Rank Adaption
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929258427842560 |
|---|---|
| author | Shi, Shuhua Huang, Shaohan Song, Minghui Li, Zhoujun Zhang, Zihan Huang, Haizhen Wei, Furu Deng, Weiwei Sun, Feng Zhang, Qi |
| author_facet | Shi, Shuhua Huang, Shaohan Song, Minghui Li, Zhoujun Zhang, Zihan Huang, Haizhen Wei, Furu Deng, Weiwei Sun, Feng Zhang, Qi |
| contents | As one of the most popular parameter-efficient fine-tuning (PEFT) methods, low-rank adaptation (LoRA) is commonly applied to fine-tune large language models (LLMs). However, updating the weights of LoRA blocks effectively and expeditiously is challenging due to the long calculation path in the original model. To address this, we propose ResLoRA, an improved framework of LoRA. By adding residual paths during training and using merging approaches to eliminate these extra paths during inference, our method can achieve better results in fewer training steps without any extra trainable parameters or inference cost compared to LoRA. The experiments on NLG, NLU, and text-to-image tasks demonstrate the effectiveness of our method. To the best of our knowledge, ResLoRA is the first work that combines the residual path with LoRA. The code of our method is available at https://github.com/microsoft/LMOps/tree/main/reslora . |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_18039 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | ResLoRA: Identity Residual Mapping in Low-Rank Adaption Shi, Shuhua Huang, Shaohan Song, Minghui Li, Zhoujun Zhang, Zihan Huang, Haizhen Wei, Furu Deng, Weiwei Sun, Feng Zhang, Qi Computation and Language Artificial Intelligence As one of the most popular parameter-efficient fine-tuning (PEFT) methods, low-rank adaptation (LoRA) is commonly applied to fine-tune large language models (LLMs). However, updating the weights of LoRA blocks effectively and expeditiously is challenging due to the long calculation path in the original model. To address this, we propose ResLoRA, an improved framework of LoRA. By adding residual paths during training and using merging approaches to eliminate these extra paths during inference, our method can achieve better results in fewer training steps without any extra trainable parameters or inference cost compared to LoRA. The experiments on NLG, NLU, and text-to-image tasks demonstrate the effectiveness of our method. To the best of our knowledge, ResLoRA is the first work that combines the residual path with LoRA. The code of our method is available at https://github.com/microsoft/LMOps/tree/main/reslora . |
| title | ResLoRA: Identity Residual Mapping in Low-Rank Adaption |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2402.18039 |