Compression Scaling Laws:Unifying Sparsity and Quantization
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866929732679892992 |
|---|---|
| author | Frantar, Elias Evci, Utku Park, Wonpyo Houlsby, Neil Alistarh, Dan |
| author_facet | Frantar, Elias Evci, Utku Park, Wonpyo Houlsby, Neil Alistarh, Dan |
| contents | We investigate how different compression techniques -- such as weight and activation quantization, and weight sparsity -- affect the scaling behavior of large language models (LLMs) during pretraining. Building on previous work showing that weight sparsity acts as a constant multiplier on model size in scaling laws, we demonstrate that this "effective parameter" scaling pattern extends to quantization as well. Specifically, we establish that weight-only quantization achieves strong parameter efficiency multipliers, while full quantization of both weights and activations shows diminishing returns at lower bitwidths. Our results suggest that different compression techniques can be unified under a common scaling law framework, enabling principled comparison and combination of these methods. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_16440 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Compression Scaling Laws:Unifying Sparsity and Quantization Frantar, Elias Evci, Utku Park, Wonpyo Houlsby, Neil Alistarh, Dan Machine Learning Computation and Language We investigate how different compression techniques -- such as weight and activation quantization, and weight sparsity -- affect the scaling behavior of large language models (LLMs) during pretraining. Building on previous work showing that weight sparsity acts as a constant multiplier on model size in scaling laws, we demonstrate that this "effective parameter" scaling pattern extends to quantization as well. Specifically, we establish that weight-only quantization achieves strong parameter efficiency multipliers, while full quantization of both weights and activations shows diminishing returns at lower bitwidths. Our results suggest that different compression techniques can be unified under a common scaling law framework, enabling principled comparison and combination of these methods. |
| title | Compression Scaling Laws:Unifying Sparsity and Quantization |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2502.16440 |