Group and Shuffle: Efficient Structured Orthogonal Parametrization
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866914834391498752 |
|---|---|
| author | Gorbunov, Mikhail Yudin, Nikolay Soboleva, Vera Alanov, Aibek Naumov, Alexey Rakhuba, Maxim |
| author_facet | Gorbunov, Mikhail Yudin, Nikolay Soboleva, Vera Alanov, Aibek Naumov, Alexey Rakhuba, Maxim |
| contents | The increasing size of neural networks has led to a growing demand for methods of efficient fine-tuning. Recently, an orthogonal fine-tuning paradigm was introduced that uses orthogonal matrices for adapting the weights of a pretrained model. In this paper, we introduce a new class of structured matrices, which unifies and generalizes structured classes from previous works. We examine properties of this class and build a structured orthogonal parametrization upon it. We then use this parametrization to modify the orthogonal fine-tuning framework, improving parameter and computational efficiency. We empirically validate our method on different domains, including adapting of text-to-image diffusion models and downstream task fine-tuning in language modeling. Additionally, we adapt our construction for orthogonal convolutions and conduct experiments with 1-Lipschitz neural networks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_10019 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Group and Shuffle: Efficient Structured Orthogonal Parametrization Gorbunov, Mikhail Yudin, Nikolay Soboleva, Vera Alanov, Aibek Naumov, Alexey Rakhuba, Maxim Machine Learning Artificial Intelligence Computation and Language Computer Vision and Pattern Recognition Numerical Analysis The increasing size of neural networks has led to a growing demand for methods of efficient fine-tuning. Recently, an orthogonal fine-tuning paradigm was introduced that uses orthogonal matrices for adapting the weights of a pretrained model. In this paper, we introduce a new class of structured matrices, which unifies and generalizes structured classes from previous works. We examine properties of this class and build a structured orthogonal parametrization upon it. We then use this parametrization to modify the orthogonal fine-tuning framework, improving parameter and computational efficiency. We empirically validate our method on different domains, including adapting of text-to-image diffusion models and downstream task fine-tuning in language modeling. Additionally, we adapt our construction for orthogonal convolutions and conduct experiments with 1-Lipschitz neural networks. |
| title | Group and Shuffle: Efficient Structured Orthogonal Parametrization |
| topic | Machine Learning Artificial Intelligence Computation and Language Computer Vision and Pattern Recognition Numerical Analysis |
| url | https://arxiv.org/abs/2406.10019 |