LoRA Learns Less and Forgets Less
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866913511295156224 |
|---|---|
| author | Biderman, Dan Portes, Jacob Ortiz, Jose Javier Gonzalez Paul, Mansheej Greengard, Philip Jennings, Connor King, Daniel Havens, Sam Chiley, Vitaliy Frankle, Jonathan Blakeney, Cody Cunningham, John P. |
| author_facet | Biderman, Dan Portes, Jacob Ortiz, Jose Javier Gonzalez Paul, Mansheej Greengard, Philip Jennings, Connor King, Daniel Havens, Sam Chiley, Vitaliy Frankle, Jonathan Blakeney, Cody Cunningham, John P. |
| contents | Low-Rank Adaptation (LoRA) is a widely-used parameter-efficient finetuning method for large language models. LoRA saves memory by training only low rank perturbations to selected weight matrices. In this work, we compare the performance of LoRA and full finetuning on two target domains, programming and mathematics. We consider both the instruction finetuning (approximately 100K prompt-response pairs) and continued pretraining (20B unstructured tokens) data regimes. Our results show that, in the standard low-rank settings, LoRA substantially underperforms full finetuning. Nevertheless, LoRA better maintains the base model's performance on tasks outside the target domain. We show that LoRA mitigates forgetting more than common regularization techniques such as weight decay and dropout; it also helps maintain more diverse generations. Finally, we show that full finetuning learns perturbations with a rank that is 10-100X greater than typical LoRA configurations, possibly explaining some of the reported gaps. We conclude by proposing best practices for finetuning with LoRA. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_09673 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | LoRA Learns Less and Forgets Less Biderman, Dan Portes, Jacob Ortiz, Jose Javier Gonzalez Paul, Mansheej Greengard, Philip Jennings, Connor King, Daniel Havens, Sam Chiley, Vitaliy Frankle, Jonathan Blakeney, Cody Cunningham, John P. Machine Learning Artificial Intelligence Computation and Language Low-Rank Adaptation (LoRA) is a widely-used parameter-efficient finetuning method for large language models. LoRA saves memory by training only low rank perturbations to selected weight matrices. In this work, we compare the performance of LoRA and full finetuning on two target domains, programming and mathematics. We consider both the instruction finetuning (approximately 100K prompt-response pairs) and continued pretraining (20B unstructured tokens) data regimes. Our results show that, in the standard low-rank settings, LoRA substantially underperforms full finetuning. Nevertheless, LoRA better maintains the base model's performance on tasks outside the target domain. We show that LoRA mitigates forgetting more than common regularization techniques such as weight decay and dropout; it also helps maintain more diverse generations. Finally, we show that full finetuning learns perturbations with a rank that is 10-100X greater than typical LoRA configurations, possibly explaining some of the reported gaps. We conclude by proposing best practices for finetuning with LoRA. |
| title | LoRA Learns Less and Forgets Less |
| topic | Machine Learning Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2405.09673 |