Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2505.02380 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910184042921984 |
|---|---|
| author | Sanyal, Arnab Datta, Gourav Mukherjee, Prithwish Chinchali, Sandeep P. Orshansky, Michael |
| author_facet | Sanyal, Arnab Datta, Gourav Mukherjee, Prithwish Chinchali, Sandeep P. Orshansky, Michael |
| contents | Large Language Models (LLMs) achieve strong performance across tasks, but face storage and compute challenges on edge devices. We propose EntroLLM, a compression framework combining mixed quantization and entropy coding to reduce storage while preserving accuracy. We use a combination of unsigned and asymmetric quantization. Tensor-level quantization produces an entropy-reducing effect, increasing weight compressibility, and improving downstream Huffman encoding by $7\times$ (8-bit) and $11.3\times$ (4-bit) over state-of-the-art methods. Huffman coding further reduces memory bandwidth demands, while a parallel decoding strategy enables efficient weight retrieval with minimal latency. Experiments on edge-scale LLMs (smolLM-1.7B, phi3-mini-4k, mistral-7B) show up to $30\%$ storage savings over uint8 and $65\%$ over uint4 models, with $31.9-146.6\%$ faster inference on memory-limited devices like the NVIDIA JETSON P3450. EntroLLM requires no retraining and is compatible with existing post-training quantization pipelines, making it practical for edge LLM deployment. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_02380 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | EntroLLM: Entropy Encoded Weight Compression for Efficient Large Language Model Inference on Edge Devices Sanyal, Arnab Datta, Gourav Mukherjee, Prithwish Chinchali, Sandeep P. Orshansky, Michael Machine Learning Large Language Models (LLMs) achieve strong performance across tasks, but face storage and compute challenges on edge devices. We propose EntroLLM, a compression framework combining mixed quantization and entropy coding to reduce storage while preserving accuracy. We use a combination of unsigned and asymmetric quantization. Tensor-level quantization produces an entropy-reducing effect, increasing weight compressibility, and improving downstream Huffman encoding by $7\times$ (8-bit) and $11.3\times$ (4-bit) over state-of-the-art methods. Huffman coding further reduces memory bandwidth demands, while a parallel decoding strategy enables efficient weight retrieval with minimal latency. Experiments on edge-scale LLMs (smolLM-1.7B, phi3-mini-4k, mistral-7B) show up to $30\%$ storage savings over uint8 and $65\%$ over uint4 models, with $31.9-146.6\%$ faster inference on memory-limited devices like the NVIDIA JETSON P3450. EntroLLM requires no retraining and is compatible with existing post-training quantization pipelines, making it practical for edge LLM deployment. |
| title | EntroLLM: Entropy Encoded Weight Compression for Efficient Large Language Model Inference on Edge Devices |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2505.02380 |