1bit-Merging: Dynamic Quantized Merging for Large Language Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910963783958528 |
|---|---|
| author | Liu, Shuqi Yao, Yuxuan He, Bowei Liu, Zehua Han, Xiongwei Yuan, Mingxuan Wu, Han Song, Linqi |
| author_facet | Liu, Shuqi Yao, Yuxuan He, Bowei Liu, Zehua Han, Xiongwei Yuan, Mingxuan Wu, Han Song, Linqi |
| contents | Recent advances in large language models have led to specialized models excelling in specific domains, creating a need for efficient model merging techniques. While traditional merging approaches combine parameters into a single static model, they often compromise task-specific performance. However, task-specific routing methods maintain accuracy but introduce substantial storage overhead. We present \texttt{1bit}-Merging, a novel framework that integrates task-specific routing with 1-bit quantized task vectors to balance performance and storage efficiency. Our approach leverages the observation that different task-specific models store knowledge in distinct layers-chat models primarily in attention layers and math/code models in MLP layers, enabling targeted compression strategies. Through extensive experiments with LLaMA2 and Mistral model families across chat, mathematical reasoning, and code generation tasks, we demonstrate that 1bit-Merging achieves comparable or superior performance to existing methods while significantly reducing storage requirements. Our framework offers a practical solution for combining specialized models while maintaining their individual strengths and addressing the storage challenges of current approaches. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_10743 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | 1bit-Merging: Dynamic Quantized Merging for Large Language Models Liu, Shuqi Yao, Yuxuan He, Bowei Liu, Zehua Han, Xiongwei Yuan, Mingxuan Wu, Han Song, Linqi Computation and Language Recent advances in large language models have led to specialized models excelling in specific domains, creating a need for efficient model merging techniques. While traditional merging approaches combine parameters into a single static model, they often compromise task-specific performance. However, task-specific routing methods maintain accuracy but introduce substantial storage overhead. We present \texttt{1bit}-Merging, a novel framework that integrates task-specific routing with 1-bit quantized task vectors to balance performance and storage efficiency. Our approach leverages the observation that different task-specific models store knowledge in distinct layers-chat models primarily in attention layers and math/code models in MLP layers, enabling targeted compression strategies. Through extensive experiments with LLaMA2 and Mistral model families across chat, mathematical reasoning, and code generation tasks, we demonstrate that 1bit-Merging achieves comparable or superior performance to existing methods while significantly reducing storage requirements. Our framework offers a practical solution for combining specialized models while maintaining their individual strengths and addressing the storage challenges of current approaches. |
| title | 1bit-Merging: Dynamic Quantized Merging for Large Language Models |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2502.10743 |