Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2507.16933 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912497868472320 |
|---|---|
| author | Esser, Steven K. McKinstry, Jeffrey L. Bablani, Deepika Appuswamy, Rathinakumar Modha, Dharmendra S. |
| author_facet | Esser, Steven K. McKinstry, Jeffrey L. Bablani, Deepika Appuswamy, Rathinakumar Modha, Dharmendra S. |
| contents | Large language models can be quantized to reduce inference time latency, model size, and energy consumption, thereby delivering a better user experience at lower cost. A challenge exists to deliver quantized models with minimal loss of accuracy in reasonable time, and in particular to do so without requiring mechanisms incompatible with specialized inference accelerators. Here, we demonstrate a simple, end-to-end quantization-aware training approach that, with an increase in total model training budget of less than 0.1%, outperforms the leading published quantization methods by large margins on several modern benchmarks, with both base and instruct model variants. The approach easily generalizes across different model architectures, can be applied to activations, cache, and weights, and requires the introduction of no additional operations to the model other than the quantization itself. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_16933 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | SiLQ: Simple Large Language Model Quantization-Aware Training Esser, Steven K. McKinstry, Jeffrey L. Bablani, Deepika Appuswamy, Rathinakumar Modha, Dharmendra S. Machine Learning Artificial Intelligence Computation and Language Large language models can be quantized to reduce inference time latency, model size, and energy consumption, thereby delivering a better user experience at lower cost. A challenge exists to deliver quantized models with minimal loss of accuracy in reasonable time, and in particular to do so without requiring mechanisms incompatible with specialized inference accelerators. Here, we demonstrate a simple, end-to-end quantization-aware training approach that, with an increase in total model training budget of less than 0.1%, outperforms the leading published quantization methods by large margins on several modern benchmarks, with both base and instruct model variants. The approach easily generalizes across different model architectures, can be applied to activations, cache, and weights, and requires the introduction of no additional operations to the model other than the quantization itself. |
| title | SiLQ: Simple Large Language Model Quantization-Aware Training |
| topic | Machine Learning Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2507.16933 |