FouRA: Fourier Low Rank Adaptation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916285261021184 |
|---|---|
| author | Borse, Shubhankar Kadambi, Shreya Pandey, Nilesh Prasad Bhardwaj, Kartikeya Ganapathy, Viswanath Priyadarshi, Sweta Garrepalli, Risheek Esteves, Rafael Hayat, Munawar Porikli, Fatih |
| author_facet | Borse, Shubhankar Kadambi, Shreya Pandey, Nilesh Prasad Bhardwaj, Kartikeya Ganapathy, Viswanath Priyadarshi, Sweta Garrepalli, Risheek Esteves, Rafael Hayat, Munawar Porikli, Fatih |
| contents | While Low-Rank Adaptation (LoRA) has proven beneficial for efficiently fine-tuning large models, LoRA fine-tuned text-to-image diffusion models lack diversity in the generated images, as the model tends to copy data from the observed training samples. This effect becomes more pronounced at higher values of adapter strength and for adapters with higher ranks which are fine-tuned on smaller datasets. To address these challenges, we present FouRA, a novel low-rank method that learns projections in the Fourier domain along with learning a flexible input-dependent adapter rank selection strategy. Through extensive experiments and analysis, we show that FouRA successfully solves the problems related to data copying and distribution collapse while significantly improving the generated image quality. We demonstrate that FouRA enhances the generalization of fine-tuned models thanks to its adaptive rank selection. We further show that the learned projections in the frequency domain are decorrelated and prove effective when merging multiple adapters. While FouRA is motivated for vision tasks, we also demonstrate its merits for language tasks on the GLUE benchmark. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_08798 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | FouRA: Fourier Low Rank Adaptation Borse, Shubhankar Kadambi, Shreya Pandey, Nilesh Prasad Bhardwaj, Kartikeya Ganapathy, Viswanath Priyadarshi, Sweta Garrepalli, Risheek Esteves, Rafael Hayat, Munawar Porikli, Fatih Computer Vision and Pattern Recognition While Low-Rank Adaptation (LoRA) has proven beneficial for efficiently fine-tuning large models, LoRA fine-tuned text-to-image diffusion models lack diversity in the generated images, as the model tends to copy data from the observed training samples. This effect becomes more pronounced at higher values of adapter strength and for adapters with higher ranks which are fine-tuned on smaller datasets. To address these challenges, we present FouRA, a novel low-rank method that learns projections in the Fourier domain along with learning a flexible input-dependent adapter rank selection strategy. Through extensive experiments and analysis, we show that FouRA successfully solves the problems related to data copying and distribution collapse while significantly improving the generated image quality. We demonstrate that FouRA enhances the generalization of fine-tuned models thanks to its adaptive rank selection. We further show that the learned projections in the frequency domain are decorrelated and prove effective when merging multiple adapters. While FouRA is motivated for vision tasks, we also demonstrate its merits for language tasks on the GLUE benchmark. |
| title | FouRA: Fourier Low Rank Adaptation |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2406.08798 |