MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866929348973428736 |
|---|---|
| author | Jiang, Ting Huang, Shaohan Luo, Shengyue Zhang, Zihan Huang, Haizhen Wei, Furu Deng, Weiwei Sun, Feng Zhang, Qi Wang, Deqing Zhuang, Fuzhen |
| author_facet | Jiang, Ting Huang, Shaohan Luo, Shengyue Zhang, Zihan Huang, Haizhen Wei, Furu Deng, Weiwei Sun, Feng Zhang, Qi Wang, Deqing Zhuang, Fuzhen |
| contents | Low-rank adaptation is a popular parameter-efficient fine-tuning method for large language models. In this paper, we analyze the impact of low-rank updating, as implemented in LoRA. Our findings suggest that the low-rank updating mechanism may limit the ability of LLMs to effectively learn and memorize new knowledge. Inspired by this observation, we propose a new method called MoRA, which employs a square matrix to achieve high-rank updating while maintaining the same number of trainable parameters. To achieve it, we introduce the corresponding non-parameter operators to reduce the input dimension and increase the output dimension for the square matrix. Furthermore, these operators ensure that the weight can be merged back into LLMs, which makes our method can be deployed like LoRA. We perform a comprehensive evaluation of our method across five tasks: instruction tuning, mathematical reasoning, continual pretraining, memory and pretraining. Our method outperforms LoRA on memory-intensive tasks and achieves comparable performance on other tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_12130 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning Jiang, Ting Huang, Shaohan Luo, Shengyue Zhang, Zihan Huang, Haizhen Wei, Furu Deng, Weiwei Sun, Feng Zhang, Qi Wang, Deqing Zhuang, Fuzhen Computation and Language Machine Learning Low-rank adaptation is a popular parameter-efficient fine-tuning method for large language models. In this paper, we analyze the impact of low-rank updating, as implemented in LoRA. Our findings suggest that the low-rank updating mechanism may limit the ability of LLMs to effectively learn and memorize new knowledge. Inspired by this observation, we propose a new method called MoRA, which employs a square matrix to achieve high-rank updating while maintaining the same number of trainable parameters. To achieve it, we introduce the corresponding non-parameter operators to reduce the input dimension and increase the output dimension for the square matrix. Furthermore, these operators ensure that the weight can be merged back into LLMs, which makes our method can be deployed like LoRA. We perform a comprehensive evaluation of our method across five tasks: instruction tuning, mathematical reasoning, continual pretraining, memory and pretraining. Our method outperforms LoRA on memory-intensive tasks and achieves comparable performance on other tasks. |
| title | MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2405.12130 |