Spectrum-Aware Parameter Efficient Fine-Tuning for Diffusion Models
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866911897026035712 |
|---|---|
| author | Zhang, Xinxi Wen, Song Han, Ligong Juefei-Xu, Felix Srivastava, Akash Huang, Junzhou Wang, Hao Tao, Molei Metaxas, Dimitris N. |
| author_facet | Zhang, Xinxi Wen, Song Han, Ligong Juefei-Xu, Felix Srivastava, Akash Huang, Junzhou Wang, Hao Tao, Molei Metaxas, Dimitris N. |
| contents | Adapting large-scale pre-trained generative models in a parameter-efficient manner is gaining traction. Traditional methods like low rank adaptation achieve parameter efficiency by imposing constraints but may not be optimal for tasks requiring high representation capacity. We propose a novel spectrum-aware adaptation framework for generative models. Our method adjusts both singular values and their basis vectors of pretrained weights. Using the Kronecker product and efficient Stiefel optimizers, we achieve parameter-efficient adaptation of orthogonal matrices. We introduce Spectral Orthogonal Decomposition Adaptation (SODA), which balances computational efficiency and representation capacity. Extensive evaluations on text-to-image diffusion models demonstrate SODA's effectiveness, offering a spectrum-aware alternative to existing fine-tuning methods. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_21050 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Spectrum-Aware Parameter Efficient Fine-Tuning for Diffusion Models Zhang, Xinxi Wen, Song Han, Ligong Juefei-Xu, Felix Srivastava, Akash Huang, Junzhou Wang, Hao Tao, Molei Metaxas, Dimitris N. Computer Vision and Pattern Recognition Machine Learning Adapting large-scale pre-trained generative models in a parameter-efficient manner is gaining traction. Traditional methods like low rank adaptation achieve parameter efficiency by imposing constraints but may not be optimal for tasks requiring high representation capacity. We propose a novel spectrum-aware adaptation framework for generative models. Our method adjusts both singular values and their basis vectors of pretrained weights. Using the Kronecker product and efficient Stiefel optimizers, we achieve parameter-efficient adaptation of orthogonal matrices. We introduce Spectral Orthogonal Decomposition Adaptation (SODA), which balances computational efficiency and representation capacity. Extensive evaluations on text-to-image diffusion models demonstrate SODA's effectiveness, offering a spectrum-aware alternative to existing fine-tuning methods. |
| title | Spectrum-Aware Parameter Efficient Fine-Tuning for Diffusion Models |
| topic | Computer Vision and Pattern Recognition Machine Learning |
| url | https://arxiv.org/abs/2405.21050 |