pQuant: Towards Effective Low-Bit Language Models via Decoupled Linear Quantization-Aware Training
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866915818148724736 |
|---|---|
| author | Zhang, Wenzheng Liu, Bingzheng Hu, Yang Bai, Xiaoying Zhang, Wentao Cui, Bin |
| author_facet | Zhang, Wenzheng Liu, Bingzheng Hu, Yang Bai, Xiaoying Zhang, Wentao Cui, Bin |
| contents | Quantization-Aware Training from scratch has emerged as a promising approach for building efficient large language models (LLMs) with extremely low-bit weights (sub 2-bit), which can offer substantial advantages for edge deployment. However, existing methods still fail to achieve satisfactory accuracy and scalability. In this work, we identify a parameter democratization effect as a key bottleneck: the sensitivity of all parameters becomes homogenized, severely limiting expressivity. To address this, we propose pQuant, a method that decouples parameters by splitting linear layers into two specialized branches: a dominant 1-bit branch for efficient computation and a compact high-precision branch dedicated to preserving the most sensitive parameters. Through tailored feature scaling, we explicitly guide the model to allocate sensitive parameters to the high-precision branch. Furthermore, we extend this branch into multiple, sparsely-activated experts, enabling efficient capacity scaling. Extensive experiments indicate our pQuant achieves state-of-the-art performance in extremely low-bit quantization. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_22592 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | pQuant: Towards Effective Low-Bit Language Models via Decoupled Linear Quantization-Aware Training Zhang, Wenzheng Liu, Bingzheng Hu, Yang Bai, Xiaoying Zhang, Wentao Cui, Bin Machine Learning Computation and Language Quantization-Aware Training from scratch has emerged as a promising approach for building efficient large language models (LLMs) with extremely low-bit weights (sub 2-bit), which can offer substantial advantages for edge deployment. However, existing methods still fail to achieve satisfactory accuracy and scalability. In this work, we identify a parameter democratization effect as a key bottleneck: the sensitivity of all parameters becomes homogenized, severely limiting expressivity. To address this, we propose pQuant, a method that decouples parameters by splitting linear layers into two specialized branches: a dominant 1-bit branch for efficient computation and a compact high-precision branch dedicated to preserving the most sensitive parameters. Through tailored feature scaling, we explicitly guide the model to allocate sensitive parameters to the high-precision branch. Furthermore, we extend this branch into multiple, sparsely-activated experts, enabling efficient capacity scaling. Extensive experiments indicate our pQuant achieves state-of-the-art performance in extremely low-bit quantization. |
| title | pQuant: Towards Effective Low-Bit Language Models via Decoupled Linear Quantization-Aware Training |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2602.22592 |