LoRA Done RITE: Robust Invariant Transformation Equilibration for LoRA Optimization
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866913945285033984 |
|---|---|
| author | Yen, Jui-Nan Si, Si Meng, Zhao Yu, Felix Duvvuri, Sai Surya Dhillon, Inderjit S. Hsieh, Cho-Jui Kumar, Sanjiv |
| author_facet | Yen, Jui-Nan Si, Si Meng, Zhao Yu, Felix Duvvuri, Sai Surya Dhillon, Inderjit S. Hsieh, Cho-Jui Kumar, Sanjiv |
| contents | Low-rank adaption (LoRA) is a widely used parameter-efficient finetuning method for LLM that reduces memory requirements. However, current LoRA optimizers lack transformation invariance, meaning the actual updates to the weights depends on how the two LoRA factors are scaled or rotated. This deficiency leads to inefficient learning and sub-optimal solutions in practice. This paper introduces LoRA-RITE, a novel adaptive matrix preconditioning method for LoRA optimization, which can achieve transformation invariance and remain computationally efficient. We provide theoretical analysis to demonstrate the benefit of our method and conduct experiments on various LLM tasks with different models including Gemma 2B, 7B, and mT5-XXL. The results demonstrate consistent improvements against existing optimizers. For example, replacing Adam with LoRA-RITE during LoRA fine-tuning of Gemma-2B yielded 4.6\% accuracy gain on Super-Natural Instructions and 3.5\% accuracy gain across other four LLM benchmarks (HellaSwag, ArcChallenge, GSM8K, OpenBookQA). |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_20625 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | LoRA Done RITE: Robust Invariant Transformation Equilibration for LoRA Optimization Yen, Jui-Nan Si, Si Meng, Zhao Yu, Felix Duvvuri, Sai Surya Dhillon, Inderjit S. Hsieh, Cho-Jui Kumar, Sanjiv Machine Learning Artificial Intelligence Computation and Language Low-rank adaption (LoRA) is a widely used parameter-efficient finetuning method for LLM that reduces memory requirements. However, current LoRA optimizers lack transformation invariance, meaning the actual updates to the weights depends on how the two LoRA factors are scaled or rotated. This deficiency leads to inefficient learning and sub-optimal solutions in practice. This paper introduces LoRA-RITE, a novel adaptive matrix preconditioning method for LoRA optimization, which can achieve transformation invariance and remain computationally efficient. We provide theoretical analysis to demonstrate the benefit of our method and conduct experiments on various LLM tasks with different models including Gemma 2B, 7B, and mT5-XXL. The results demonstrate consistent improvements against existing optimizers. For example, replacing Adam with LoRA-RITE during LoRA fine-tuning of Gemma-2B yielded 4.6\% accuracy gain on Super-Natural Instructions and 3.5\% accuracy gain across other four LLM benchmarks (HellaSwag, ArcChallenge, GSM8K, OpenBookQA). |
| title | LoRA Done RITE: Robust Invariant Transformation Equilibration for LoRA Optimization |
| topic | Machine Learning Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2410.20625 |