From 2:4 to 8:16 sparsity patterns in LLMs for Outliers and Weights with Variance Correction
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866913041279352832 |
|---|---|
| author | Maximov, Egor Kuzkina, Yulia Kanametov, Azamat Prutko, Alexander Goncharov, Aleksei Zhelnin, Maxim Shvetsov, Egor |
| author_facet | Maximov, Egor Kuzkina, Yulia Kanametov, Azamat Prutko, Alexander Goncharov, Aleksei Zhelnin, Maxim Shvetsov, Egor |
| contents | As large language models (LLMs) grow in size, efficient compression techniques like quantization and sparsification are critical. While quantization maintains performance with reduced precision, structured sparsity methods, such as N:M sparsification, often fall short due to limited flexibility, and sensitivity to outlier weights. We explore 8:16 semi-structured sparsity, demonstrating its ability to surpass the Performance Threshold-where a compressed model matches the accuracy of its uncompressed or smaller counterpart under equivalent memory constraints. Compared to 2:4 sparsity, 8:16 offers greater flexibility with minimal storage overhead (0.875 vs. 0.75 bits/element). We also apply sparse structured patterns for salient weights, showing that structured sparsity for outliers is competitive with unstructured approaches leading to equivalent or better results. Finally, we demonstrate that simple techniques such as variance correction and SmoothQuant like weight equalization improve sparse models performance. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_03052 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | From 2:4 to 8:16 sparsity patterns in LLMs for Outliers and Weights with Variance Correction Maximov, Egor Kuzkina, Yulia Kanametov, Azamat Prutko, Alexander Goncharov, Aleksei Zhelnin, Maxim Shvetsov, Egor Machine Learning Artificial Intelligence As large language models (LLMs) grow in size, efficient compression techniques like quantization and sparsification are critical. While quantization maintains performance with reduced precision, structured sparsity methods, such as N:M sparsification, often fall short due to limited flexibility, and sensitivity to outlier weights. We explore 8:16 semi-structured sparsity, demonstrating its ability to surpass the Performance Threshold-where a compressed model matches the accuracy of its uncompressed or smaller counterpart under equivalent memory constraints. Compared to 2:4 sparsity, 8:16 offers greater flexibility with minimal storage overhead (0.875 vs. 0.75 bits/element). We also apply sparse structured patterns for salient weights, showing that structured sparsity for outliers is competitive with unstructured approaches leading to equivalent or better results. Finally, we demonstrate that simple techniques such as variance correction and SmoothQuant like weight equalization improve sparse models performance. |
| title | From 2:4 to 8:16 sparsity patterns in LLMs for Outliers and Weights with Variance Correction |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2507.03052 |