QDyLoRA: Quantized Dynamic Low-Rank Adaptation for Efficient Large Language Model Tuning
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866909728577159168 |
|---|---|
| author | Rajabzadeh, Hossein Valipour, Mojtaba Zhu, Tianshu Tahaei, Marzieh Kwon, Hyock Ju Ghodsi, Ali Chen, Boxing Rezagholizadeh, Mehdi |
| author_facet | Rajabzadeh, Hossein Valipour, Mojtaba Zhu, Tianshu Tahaei, Marzieh Kwon, Hyock Ju Ghodsi, Ali Chen, Boxing Rezagholizadeh, Mehdi |
| contents | Finetuning large language models requires huge GPU memory, restricting the choice to acquire Larger models. While the quantized version of the Low-Rank Adaptation technique, named QLoRA, significantly alleviates this issue, finding the efficient LoRA rank is still challenging. Moreover, QLoRA is trained on a pre-defined rank and, therefore, cannot be reconfigured for its lower ranks without requiring further fine-tuning steps. This paper proposes QDyLoRA -Quantized Dynamic Low-Rank Adaptation-, as an efficient quantization approach for dynamic low-rank adaptation. Motivated by Dynamic LoRA, QDyLoRA is able to efficiently finetune LLMs on a set of pre-defined LoRA ranks. QDyLoRA enables fine-tuning Falcon-40b for ranks 1 to 64 on a single 32 GB V100-GPU through one round of fine-tuning. Experimental results show that QDyLoRA is competitive to QLoRA and outperforms when employing its optimal rank. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_10462 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | QDyLoRA: Quantized Dynamic Low-Rank Adaptation for Efficient Large Language Model Tuning Rajabzadeh, Hossein Valipour, Mojtaba Zhu, Tianshu Tahaei, Marzieh Kwon, Hyock Ju Ghodsi, Ali Chen, Boxing Rezagholizadeh, Mehdi Machine Learning Computation and Language Finetuning large language models requires huge GPU memory, restricting the choice to acquire Larger models. While the quantized version of the Low-Rank Adaptation technique, named QLoRA, significantly alleviates this issue, finding the efficient LoRA rank is still challenging. Moreover, QLoRA is trained on a pre-defined rank and, therefore, cannot be reconfigured for its lower ranks without requiring further fine-tuning steps. This paper proposes QDyLoRA -Quantized Dynamic Low-Rank Adaptation-, as an efficient quantization approach for dynamic low-rank adaptation. Motivated by Dynamic LoRA, QDyLoRA is able to efficiently finetune LLMs on a set of pre-defined LoRA ranks. QDyLoRA enables fine-tuning Falcon-40b for ranks 1 to 64 on a single 32 GB V100-GPU through one round of fine-tuning. Experimental results show that QDyLoRA is competitive to QLoRA and outperforms when employing its optimal rank. |
| title | QDyLoRA: Quantized Dynamic Low-Rank Adaptation for Efficient Large Language Model Tuning |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2402.10462 |