Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2410.10714 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929544540192768 |
|---|---|
| author | Shafipour, Rasoul Harrison, David Horton, Maxwell Marker, Jeffrey Bedayat, Houman Mehta, Sachin Rastegari, Mohammad Najibi, Mahyar Naderiparizi, Saman |
| author_facet | Shafipour, Rasoul Harrison, David Horton, Maxwell Marker, Jeffrey Bedayat, Houman Mehta, Sachin Rastegari, Mohammad Najibi, Mahyar Naderiparizi, Saman |
| contents | Large Language Models (LLMs) have transformed natural language processing, but face significant challenges in widespread deployment due to their high runtime cost. In this paper, we introduce SeedLM, a novel post-training compression method that uses seeds of pseudo-random generators to encode and compress model weights. Specifically, for each block of weights, we find a seed that is fed into a Linear Feedback Shift Register (LFSR) during inference to efficiently generate a random matrix. This matrix is then linearly combined with compressed coefficients to reconstruct the weight block. SeedLM reduces memory access and leverages idle compute cycles during inference, effectively speeding up memory-bound tasks by trading compute for fewer memory accesses. Unlike state-of-the-art compression methods that rely on calibration data, our approach is data-free and generalizes well across diverse tasks. Our experiments with Llama 3 70B, which is particularly challenging to compress, show that SeedLM achieves significantly better zero-shot accuracy retention at 4- and 3-bit than state-of-the-art techniques, while maintaining performance comparable to FP16 baselines. Additionally, FPGA-based tests demonstrate that 4-bit SeedLM, as model size increases to 70B, approaches a 4x speed-up over an FP16 Llama 2/3 baseline. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_10714 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | SeedLM: Compressing LLM Weights into Seeds of Pseudo-Random Generators Shafipour, Rasoul Harrison, David Horton, Maxwell Marker, Jeffrey Bedayat, Houman Mehta, Sachin Rastegari, Mohammad Najibi, Mahyar Naderiparizi, Saman Machine Learning Artificial Intelligence Large Language Models (LLMs) have transformed natural language processing, but face significant challenges in widespread deployment due to their high runtime cost. In this paper, we introduce SeedLM, a novel post-training compression method that uses seeds of pseudo-random generators to encode and compress model weights. Specifically, for each block of weights, we find a seed that is fed into a Linear Feedback Shift Register (LFSR) during inference to efficiently generate a random matrix. This matrix is then linearly combined with compressed coefficients to reconstruct the weight block. SeedLM reduces memory access and leverages idle compute cycles during inference, effectively speeding up memory-bound tasks by trading compute for fewer memory accesses. Unlike state-of-the-art compression methods that rely on calibration data, our approach is data-free and generalizes well across diverse tasks. Our experiments with Llama 3 70B, which is particularly challenging to compress, show that SeedLM achieves significantly better zero-shot accuracy retention at 4- and 3-bit than state-of-the-art techniques, while maintaining performance comparable to FP16 baselines. Additionally, FPGA-based tests demonstrate that 4-bit SeedLM, as model size increases to 70B, approaches a 4x speed-up over an FP16 Llama 2/3 baseline. |
| title | SeedLM: Compressing LLM Weights into Seeds of Pseudo-Random Generators |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2410.10714 |