ProcrustesGPT: Compressing LLMs with Structured Matrices and Orthogonal Transformations
Fuente:
arXiv
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910983284326400 |
|---|---|
| author | Grishina, Ekaterina Gorbunov, Mikhail Rakhuba, Maxim |
| author_facet | Grishina, Ekaterina Gorbunov, Mikhail Rakhuba, Maxim |
| contents | Large language models (LLMs) demonstrate impressive results in natural language processing tasks but require a significant amount of computational and memory resources. Structured matrix representations are a promising way for reducing the number of parameters of these models. However, it seems unrealistic to expect that weight matrices of pretrained models can be accurately represented by structured matrices without any fine-tuning. To overcome this issue, we utilize the fact that LLM output is invariant under certain orthogonal transformations of weight matrices. This insight can be leveraged to identify transformations that significantly improve the compressibility of weights within structured classes. The proposed approach is applicable to various types of structured matrices that support efficient projection operations. Code is available at https://github.com/GrishKate/ProcrustesGPT |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_02818 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | ProcrustesGPT: Compressing LLMs with Structured Matrices and Orthogonal Transformations Grishina, Ekaterina Gorbunov, Mikhail Rakhuba, Maxim Computation and Language Machine Learning Large language models (LLMs) demonstrate impressive results in natural language processing tasks but require a significant amount of computational and memory resources. Structured matrix representations are a promising way for reducing the number of parameters of these models. However, it seems unrealistic to expect that weight matrices of pretrained models can be accurately represented by structured matrices without any fine-tuning. To overcome this issue, we utilize the fact that LLM output is invariant under certain orthogonal transformations of weight matrices. This insight can be leveraged to identify transformations that significantly improve the compressibility of weights within structured classes. The proposed approach is applicable to various types of structured matrices that support efficient projection operations. Code is available at https://github.com/GrishKate/ProcrustesGPT |
| title | ProcrustesGPT: Compressing LLMs with Structured Matrices and Orthogonal Transformations |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2506.02818 |