Parameter-Efficient Orthogonal Finetuning via Butterfly Factorization
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2023
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866914772835893248 |
|---|---|
| author | Liu, Weiyang Qiu, Zeju Feng, Yao Xiu, Yuliang Xue, Yuxuan Yu, Longhui Feng, Haiwen Liu, Zhen Heo, Juyeon Peng, Songyou Wen, Yandong Black, Michael J. Weller, Adrian Schölkopf, Bernhard |
| author_facet | Liu, Weiyang Qiu, Zeju Feng, Yao Xiu, Yuliang Xue, Yuxuan Yu, Longhui Feng, Haiwen Liu, Zhen Heo, Juyeon Peng, Songyou Wen, Yandong Black, Michael J. Weller, Adrian Schölkopf, Bernhard |
| contents | Large foundation models are becoming ubiquitous, but training them from scratch is prohibitively expensive. Thus, efficiently adapting these powerful models to downstream tasks is increasingly important. In this paper, we study a principled finetuning paradigm -- Orthogonal Finetuning (OFT) -- for downstream task adaptation. Despite demonstrating good generalizability, OFT still uses a fairly large number of trainable parameters due to the high dimensionality of orthogonal matrices. To address this, we start by examining OFT from an information transmission perspective, and then identify a few key desiderata that enable better parameter-efficiency. Inspired by how the Cooley-Tukey fast Fourier transform algorithm enables efficient information transmission, we propose an efficient orthogonal parameterization using butterfly structures. We apply this parameterization to OFT, creating a novel parameter-efficient finetuning method, called Orthogonal Butterfly (BOFT). By subsuming OFT as a special case, BOFT introduces a generalized orthogonal finetuning framework. Finally, we conduct an extensive empirical study of adapting large vision transformers, large language models, and text-to-image diffusion models to various downstream tasks in vision and language. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2311_06243 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Parameter-Efficient Orthogonal Finetuning via Butterfly Factorization Liu, Weiyang Qiu, Zeju Feng, Yao Xiu, Yuliang Xue, Yuxuan Yu, Longhui Feng, Haiwen Liu, Zhen Heo, Juyeon Peng, Songyou Wen, Yandong Black, Michael J. Weller, Adrian Schölkopf, Bernhard Machine Learning Artificial Intelligence Computation and Language Computer Vision and Pattern Recognition Large foundation models are becoming ubiquitous, but training them from scratch is prohibitively expensive. Thus, efficiently adapting these powerful models to downstream tasks is increasingly important. In this paper, we study a principled finetuning paradigm -- Orthogonal Finetuning (OFT) -- for downstream task adaptation. Despite demonstrating good generalizability, OFT still uses a fairly large number of trainable parameters due to the high dimensionality of orthogonal matrices. To address this, we start by examining OFT from an information transmission perspective, and then identify a few key desiderata that enable better parameter-efficiency. Inspired by how the Cooley-Tukey fast Fourier transform algorithm enables efficient information transmission, we propose an efficient orthogonal parameterization using butterfly structures. We apply this parameterization to OFT, creating a novel parameter-efficient finetuning method, called Orthogonal Butterfly (BOFT). By subsuming OFT as a special case, BOFT introduces a generalized orthogonal finetuning framework. Finally, we conduct an extensive empirical study of adapting large vision transformers, large language models, and text-to-image diffusion models to various downstream tasks in vision and language. |
| title | Parameter-Efficient Orthogonal Finetuning via Butterfly Factorization |
| topic | Machine Learning Artificial Intelligence Computation and Language Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2311.06243 |