Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2410.21073 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909368245551104 |
|---|---|
| author | Matsutani, Hiroki Kondo, Masaaki Sunaga, Kazuki Marculescu, Radu |
| author_facet | Matsutani, Hiroki Kondo, Masaaki Sunaga, Kazuki Marculescu, Radu |
| contents | This paper proposes Skip2-LoRA as a lightweight fine-tuning method for deep neural networks to address the gap between pre-trained and deployed models. In our approach, trainable LoRA (low-rank adaptation) adapters are inserted between the last layer and every other layer to enhance the network expressive power while keeping the backward computation cost low. This architecture is well-suited to cache intermediate computation results of the forward pass and then can skip the forward computation of seen samples as training epochs progress. We implemented the combination of the proposed architecture and cache, denoted as Skip2-LoRA, and tested it on a $15 single board computer. Our results show that Skip2-LoRA reduces the fine-tuning time by 90.0% on average compared to the counterpart that has the same number of trainable parameters while preserving the accuracy, while taking only a few seconds on the microcontroller board. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_21073 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Skip2-LoRA: A Lightweight On-device DNN Fine-tuning Method for Low-cost Edge Devices Matsutani, Hiroki Kondo, Masaaki Sunaga, Kazuki Marculescu, Radu Machine Learning Artificial Intelligence This paper proposes Skip2-LoRA as a lightweight fine-tuning method for deep neural networks to address the gap between pre-trained and deployed models. In our approach, trainable LoRA (low-rank adaptation) adapters are inserted between the last layer and every other layer to enhance the network expressive power while keeping the backward computation cost low. This architecture is well-suited to cache intermediate computation results of the forward pass and then can skip the forward computation of seen samples as training epochs progress. We implemented the combination of the proposed architecture and cache, denoted as Skip2-LoRA, and tested it on a $15 single board computer. Our results show that Skip2-LoRA reduces the fine-tuning time by 90.0% on average compared to the counterpart that has the same number of trainable parameters while preserving the accuracy, while taking only a few seconds on the microcontroller board. |
| title | Skip2-LoRA: A Lightweight On-device DNN Fine-tuning Method for Low-cost Edge Devices |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2410.21073 |