EoRA: Fine-tuning-free Compensation for Compressed LLM with Eigenspace Low-Rank Approximation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910055447658496 |
|---|---|
| author | Liu, Shih-Yang Khadkevich, Maksim Fung, Nai Chit Sakr, Charbel Yang, Chao-Han Huck Wang, Chien-Yi Muralidharan, Saurav Yin, Hongxu Cheng, Kwang-Ting Kautz, Jan Wang, Yu-Chiang Frank Molchanov, Pavlo Chen, Min-Hung |
| author_facet | Liu, Shih-Yang Khadkevich, Maksim Fung, Nai Chit Sakr, Charbel Yang, Chao-Han Huck Wang, Chien-Yi Muralidharan, Saurav Yin, Hongxu Cheng, Kwang-Ting Kautz, Jan Wang, Yu-Chiang Frank Molchanov, Pavlo Chen, Min-Hung |
| contents | While post-training compression techniques effectively reduce the memory footprint, latency, and power consumption of Large Language Models (LLMs), they often result in noticeable accuracy degradation and remain limited by hardware and kernel constraints that restrict supported compression formats - ultimately reducing flexibility across a wide range of deployment scenarios. In this work, we propose EoRA - a novel $\textbf{fine-tuning-free}$ method that augments compressed LLMs with low-rank matrices, allowing users to rapidly enhance task-specific performance and freely balance the trade-off between accuracy and computational overhead beyond the constraints of compression formats. EoRA consistently outperforms prior fine-tuning-free low rank methods in recovering the accuracy of compressed LLMs, achieving notable accuracy improvements (e.g., $\mathbf{10.84\%}$ on ARC-Challenge, $\mathbf{6.74\%}$ on MathQA, and $\mathbf{11.45\%}$ on GSM8K for LLaMA3-8B compressed to 3-bit). We also introduce an optimized CUDA kernel, accelerating inference by up to 1.4x and reducing memory overhead through quantizing EoRA. Overall, EoRA offers a prompt solution for improving the accuracy of compressed models under varying user requirements, enabling more efficient and flexible deployment of LLMs. Code is available at https://github.com/NVlabs/EoRA. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_21271 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | EoRA: Fine-tuning-free Compensation for Compressed LLM with Eigenspace Low-Rank Approximation Liu, Shih-Yang Khadkevich, Maksim Fung, Nai Chit Sakr, Charbel Yang, Chao-Han Huck Wang, Chien-Yi Muralidharan, Saurav Yin, Hongxu Cheng, Kwang-Ting Kautz, Jan Wang, Yu-Chiang Frank Molchanov, Pavlo Chen, Min-Hung Computation and Language Artificial Intelligence While post-training compression techniques effectively reduce the memory footprint, latency, and power consumption of Large Language Models (LLMs), they often result in noticeable accuracy degradation and remain limited by hardware and kernel constraints that restrict supported compression formats - ultimately reducing flexibility across a wide range of deployment scenarios. In this work, we propose EoRA - a novel $\textbf{fine-tuning-free}$ method that augments compressed LLMs with low-rank matrices, allowing users to rapidly enhance task-specific performance and freely balance the trade-off between accuracy and computational overhead beyond the constraints of compression formats. EoRA consistently outperforms prior fine-tuning-free low rank methods in recovering the accuracy of compressed LLMs, achieving notable accuracy improvements (e.g., $\mathbf{10.84\%}$ on ARC-Challenge, $\mathbf{6.74\%}$ on MathQA, and $\mathbf{11.45\%}$ on GSM8K for LLaMA3-8B compressed to 3-bit). We also introduce an optimized CUDA kernel, accelerating inference by up to 1.4x and reducing memory overhead through quantizing EoRA. Overall, EoRA offers a prompt solution for improving the accuracy of compressed models under varying user requirements, enabling more efficient and flexible deployment of LLMs. Code is available at https://github.com/NVlabs/EoRA. |
| title | EoRA: Fine-tuning-free Compensation for Compressed LLM with Eigenspace Low-Rank Approximation |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2410.21271 |