Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2505.07531 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908534133751808 |
|---|---|
| author | Ahmad, Muhammad Mazher, Khurram Akram, Saqib Tameem, Ahmad Nasir, Saad Bin |
| author_facet | Ahmad, Muhammad Mazher, Khurram Akram, Saqib Tameem, Ahmad Nasir, Saad Bin |
| contents | We present QuantX: a tailored suite of recipes for LLM and VLM quantization. It is capable of quantizing down to 3-bit resolutions with minimal loss in performance. The quantization strategies in QuantX take into account hardware-specific constraints to achieve efficient dequantization during inference ensuring flexible trade-off between runtime speed, memory requirement and model accuracy. Our results demonstrate that QuantX achieves performance within 6% of the unquantized model for LlaVa-v1.6 quantized down to 3-bits for multiple end user tasks and outperforms recently published state-of-the-art quantization techniques. We further integrate one particular technique from QuantX into the popular Llama.cpp framework and show its feasibility in terms of runtime compared to the mainstream quantization techniques from Llama.cpp. Lastly, this manuscript provides insights into the LLM quantization process that motivated the range of recipes and options that are incorporated in QuantX. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_07531 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | QuantX: A Framework for Hardware-Aware Quantization of Generative AI Workloads Ahmad, Muhammad Mazher, Khurram Akram, Saqib Tameem, Ahmad Nasir, Saad Bin Artificial Intelligence Signal Processing We present QuantX: a tailored suite of recipes for LLM and VLM quantization. It is capable of quantizing down to 3-bit resolutions with minimal loss in performance. The quantization strategies in QuantX take into account hardware-specific constraints to achieve efficient dequantization during inference ensuring flexible trade-off between runtime speed, memory requirement and model accuracy. Our results demonstrate that QuantX achieves performance within 6% of the unquantized model for LlaVa-v1.6 quantized down to 3-bits for multiple end user tasks and outperforms recently published state-of-the-art quantization techniques. We further integrate one particular technique from QuantX into the popular Llama.cpp framework and show its feasibility in terms of runtime compared to the mainstream quantization techniques from Llama.cpp. Lastly, this manuscript provides insights into the LLM quantization process that motivated the range of recipes and options that are incorporated in QuantX. |
| title | QuantX: A Framework for Hardware-Aware Quantization of Generative AI Workloads |
| topic | Artificial Intelligence Signal Processing |
| url | https://arxiv.org/abs/2505.07531 |