Getting Free Bits Back from Rotational Symmetries in LLMs
Fuente:
arXiv
Guardado en:
| Autores principales: | , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866914962609274880 |
|---|---|
| author | He, Jiajun Flamich, Gergely Hernández-Lobato, José Miguel |
| author_facet | He, Jiajun Flamich, Gergely Hernández-Lobato, José Miguel |
| contents | Current methods for compressing neural network weights, such as decomposition, pruning, quantization, and channel simulation, often overlook the inherent symmetries within these networks and thus waste bits on encoding redundant information. In this paper, we propose a format based on bits-back coding for storing rotationally symmetric Transformer weights more efficiently than the usual array layout at the same floating-point precision. We evaluate our method on Large Language Models (LLMs) pruned by SliceGPT (Ashkboos et al., 2024) and achieve a 3-5% reduction in total bit usage for free across different model sizes and architectures without impacting model performance within a certain numerical precision. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_01309 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Getting Free Bits Back from Rotational Symmetries in LLMs He, Jiajun Flamich, Gergely Hernández-Lobato, José Miguel Information Theory Machine Learning Current methods for compressing neural network weights, such as decomposition, pruning, quantization, and channel simulation, often overlook the inherent symmetries within these networks and thus waste bits on encoding redundant information. In this paper, we propose a format based on bits-back coding for storing rotationally symmetric Transformer weights more efficiently than the usual array layout at the same floating-point precision. We evaluate our method on Large Language Models (LLMs) pruned by SliceGPT (Ashkboos et al., 2024) and achieve a 3-5% reduction in total bit usage for free across different model sizes and architectures without impacting model performance within a certain numerical precision. |
| title | Getting Free Bits Back from Rotational Symmetries in LLMs |
| topic | Information Theory Machine Learning |
| url | https://arxiv.org/abs/2410.01309 |