Delta-CoMe: Training-Free Delta-Compression with Mixed-Precision for Large Language Models
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866909404950953984 |
|---|---|
| author | Ping, Bowen Wang, Shuo Wang, Hanqing Han, Xu Xu, Yuzhuang Yan, Yukun Chen, Yun Chang, Baobao Liu, Zhiyuan Sun, Maosong |
| author_facet | Ping, Bowen Wang, Shuo Wang, Hanqing Han, Xu Xu, Yuzhuang Yan, Yukun Chen, Yun Chang, Baobao Liu, Zhiyuan Sun, Maosong |
| contents | Fine-tuning is a crucial process for adapting large language models (LLMs) to diverse applications. In certain scenarios, such as multi-tenant serving, deploying multiple LLMs becomes necessary to meet complex demands. Recent studies suggest decomposing a fine-tuned LLM into a base model and corresponding delta weights, which are then compressed using low-rank or low-bit approaches to reduce costs. In this work, we observe that existing low-rank and low-bit compression methods can significantly harm the model performance for task-specific fine-tuned LLMs (e.g., WizardMath for math problems). Motivated by the long-tail distribution of singular values in the delta weights, we propose a delta quantization approach using mixed-precision. This method employs higher-bit representation for singular vectors corresponding to larger singular values. We evaluate our approach on various fine-tuned LLMs, including math LLMs, code LLMs, chat LLMs, and even VLMs. Experimental results demonstrate that our approach performs comparably to full fine-tuned LLMs, surpassing both low-rank and low-bit baselines by a considerable margin. Additionally, we show that our method is compatible with various backbone LLMs, such as Llama-2, Llama-3, and Mistral, highlighting its generalizability. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_08903 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Delta-CoMe: Training-Free Delta-Compression with Mixed-Precision for Large Language Models Ping, Bowen Wang, Shuo Wang, Hanqing Han, Xu Xu, Yuzhuang Yan, Yukun Chen, Yun Chang, Baobao Liu, Zhiyuan Sun, Maosong Computation and Language Fine-tuning is a crucial process for adapting large language models (LLMs) to diverse applications. In certain scenarios, such as multi-tenant serving, deploying multiple LLMs becomes necessary to meet complex demands. Recent studies suggest decomposing a fine-tuned LLM into a base model and corresponding delta weights, which are then compressed using low-rank or low-bit approaches to reduce costs. In this work, we observe that existing low-rank and low-bit compression methods can significantly harm the model performance for task-specific fine-tuned LLMs (e.g., WizardMath for math problems). Motivated by the long-tail distribution of singular values in the delta weights, we propose a delta quantization approach using mixed-precision. This method employs higher-bit representation for singular vectors corresponding to larger singular values. We evaluate our approach on various fine-tuned LLMs, including math LLMs, code LLMs, chat LLMs, and even VLMs. Experimental results demonstrate that our approach performs comparably to full fine-tuned LLMs, surpassing both low-rank and low-bit baselines by a considerable margin. Additionally, we show that our method is compatible with various backbone LLMs, such as Llama-2, Llama-3, and Mistral, highlighting its generalizability. |
| title | Delta-CoMe: Training-Free Delta-Compression with Mixed-Precision for Large Language Models |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2406.08903 |