Quantization of Large Language Models with an Overdetermined Basis
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866916206436417536 |
|---|---|
| author | Merkulov, Daniil Cherniuk, Daria Rudikov, Alexander Oseledets, Ivan Muravleva, Ekaterina Mikhalev, Aleksandr Kashin, Boris |
| author_facet | Merkulov, Daniil Cherniuk, Daria Rudikov, Alexander Oseledets, Ivan Muravleva, Ekaterina Mikhalev, Aleksandr Kashin, Boris |
| contents | In this paper, we introduce an algorithm for data quantization based on the principles of Kashin representation. This approach hinges on decomposing any given vector, matrix, or tensor into two factors. The first factor maintains a small infinity norm, while the second exhibits a similarly constrained norm when multiplied by an orthogonal matrix. Surprisingly, the entries of factors after decomposition are well-concentrated around several peaks, which allows us to efficiently replace them with corresponding centroids for quantization purposes. We study the theoretical properties of the proposed approach and rigorously evaluate our compression algorithm in the context of next-word prediction tasks and on a set of downstream tasks for text classification. Our findings demonstrate that Kashin Quantization achieves competitive or superior quality in model performance while ensuring data compression, marking a significant advancement in the field of data quantization. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_09737 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Quantization of Large Language Models with an Overdetermined Basis Merkulov, Daniil Cherniuk, Daria Rudikov, Alexander Oseledets, Ivan Muravleva, Ekaterina Mikhalev, Aleksandr Kashin, Boris Machine Learning Computation and Language In this paper, we introduce an algorithm for data quantization based on the principles of Kashin representation. This approach hinges on decomposing any given vector, matrix, or tensor into two factors. The first factor maintains a small infinity norm, while the second exhibits a similarly constrained norm when multiplied by an orthogonal matrix. Surprisingly, the entries of factors after decomposition are well-concentrated around several peaks, which allows us to efficiently replace them with corresponding centroids for quantization purposes. We study the theoretical properties of the proposed approach and rigorously evaluate our compression algorithm in the context of next-word prediction tasks and on a set of downstream tasks for text classification. Our findings demonstrate that Kashin Quantization achieves competitive or superior quality in model performance while ensuring data compression, marking a significant advancement in the field of data quantization. |
| title | Quantization of Large Language Models with an Overdetermined Basis |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2404.09737 |