Rethinking 1-bit Optimization Leveraging Pre-trained Large Language Models
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866910230495887360 |
|---|---|
| author | Tu, Zhijun Li, Jian Xi, Yuanyuan Liu, Siqi Liu, Chuanjian Chen, Hanting Hu, Jie Wang, Yunhe |
| author_facet | Tu, Zhijun Li, Jian Xi, Yuanyuan Liu, Siqi Liu, Chuanjian Chen, Hanting Hu, Jie Wang, Yunhe |
| contents | 1-bit LLM quantization offers significant advantages in reducing storage and computational costs. However, existing methods typically train 1-bit LLMs from scratch, failing to fully leverage pre-trained models. This results in high training costs and notable accuracy degradation. We identify that the large gap between full precision and 1-bit representations makes naive adaptation difficult. In this paper, we introduce a consistent progressive training for both forward and backward, smoothly converting the full-precision weights into the binarized ones. Additionally, we incorporate binary-aware initialization and dual-scaling compensation to reduce the difficulty of progressive training and improve the performance. Experimental results on LLMs of various sizes demonstrate that our method outperforms existing approaches. Our results show that high-performance 1-bit LLMs can be achieved using pre-trained models, eliminating the need for expensive training from scratch. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_06974 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Rethinking 1-bit Optimization Leveraging Pre-trained Large Language Models Tu, Zhijun Li, Jian Xi, Yuanyuan Liu, Siqi Liu, Chuanjian Chen, Hanting Hu, Jie Wang, Yunhe Computation and Language 1-bit LLM quantization offers significant advantages in reducing storage and computational costs. However, existing methods typically train 1-bit LLMs from scratch, failing to fully leverage pre-trained models. This results in high training costs and notable accuracy degradation. We identify that the large gap between full precision and 1-bit representations makes naive adaptation difficult. In this paper, we introduce a consistent progressive training for both forward and backward, smoothly converting the full-precision weights into the binarized ones. Additionally, we incorporate binary-aware initialization and dual-scaling compensation to reduce the difficulty of progressive training and improve the performance. Experimental results on LLMs of various sizes demonstrate that our method outperforms existing approaches. Our results show that high-performance 1-bit LLMs can be achieved using pre-trained models, eliminating the need for expensive training from scratch. |
| title | Rethinking 1-bit Optimization Leveraging Pre-trained Large Language Models |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2508.06974 |