Run LoRA Run: Faster and Lighter LoRA Implementations
Fuente:
arXiv
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911917213220864 |
|---|---|
| author | Cherniuk, Daria Mikhalev, Aleksandr Oseledets, Ivan |
| author_facet | Cherniuk, Daria Mikhalev, Aleksandr Oseledets, Ivan |
| contents | LoRA is a technique that reduces the number of trainable parameters in a neural network by introducing low-rank adapters to linear layers. This technique is used both for fine-tuning and full training of large language models. This paper presents the RunLoRA framework for efficient implementations of LoRA that significantly improves the speed of neural network training and fine-tuning using low-rank adapters. The proposed implementation optimizes the computation of LoRA operations based on dimensions of corresponding linear layer, layer input dimensions and lora rank by choosing best forward and backward computation graph based on FLOPs and time estimations, resulting in faster training without sacrificing accuracy. The experimental results show up to 28\% speedup on language modeling networks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2312_03415 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Run LoRA Run: Faster and Lighter LoRA Implementations Cherniuk, Daria Mikhalev, Aleksandr Oseledets, Ivan Machine Learning LoRA is a technique that reduces the number of trainable parameters in a neural network by introducing low-rank adapters to linear layers. This technique is used both for fine-tuning and full training of large language models. This paper presents the RunLoRA framework for efficient implementations of LoRA that significantly improves the speed of neural network training and fine-tuning using low-rank adapters. The proposed implementation optimizes the computation of LoRA operations based on dimensions of corresponding linear layer, layer input dimensions and lora rank by choosing best forward and backward computation graph based on FLOPs and time estimations, resulting in faster training without sacrificing accuracy. The experimental results show up to 28\% speedup on language modeling networks. |
| title | Run LoRA Run: Faster and Lighter LoRA Implementations |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2312.03415 |