Quantization Meets dLLMs: A Systematic Study of Post-training Quantization for Diffusion LLMs
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914392004624384 |
|---|---|
| author | Lin, Haokun Xu, Haobo Wu, Yichen Guo, Ziyu Zhang, Renrui Lu, Zhichao Wei, Ying Zhang, Qingfu Sun, Zhenan |
| author_facet | Lin, Haokun Xu, Haobo Wu, Yichen Guo, Ziyu Zhang, Renrui Lu, Zhichao Wei, Ying Zhang, Qingfu Sun, Zhenan |
| contents | Recent advances in diffusion large language models (dLLMs) have introduced a promising alternative to autoregressive (AR) LLMs for natural language generation tasks, leveraging full attention and denoising-based decoding strategies. However, the deployment of these models on edge devices remains challenging due to their massive parameter scale and high resource demands. While post-training quantization (PTQ) has emerged as a widely adopted technique for compressing AR LLMs, its applicability to dLLMs remains largely unexplored. In this work, we present the first systematic study on quantizing diffusion-based language models. We begin by identifying the presence of activation outliers, characterized by abnormally large activation values that dominate the dynamic range. These outliers pose a key challenge to low-bit quantization, as they make it difficult to preserve precision for the majority of values. More importantly, we implement state-of-the-art PTQ methods and conduct a comprehensive evaluation across multiple task types and model variants. Our analysis is structured along four key dimensions: bit-width, quantization method, task category, and model type. Through this multi-perspective evaluation, we offer practical insights into the quantization behavior of dLLMs under different configurations. We hope our findings provide a foundation for future research in efficient dLLM deployment. Our code is publicly available at https://github.com/FelixMessi/QDLM. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_14896 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Quantization Meets dLLMs: A Systematic Study of Post-training Quantization for Diffusion LLMs Lin, Haokun Xu, Haobo Wu, Yichen Guo, Ziyu Zhang, Renrui Lu, Zhichao Wei, Ying Zhang, Qingfu Sun, Zhenan Computation and Language Artificial Intelligence Recent advances in diffusion large language models (dLLMs) have introduced a promising alternative to autoregressive (AR) LLMs for natural language generation tasks, leveraging full attention and denoising-based decoding strategies. However, the deployment of these models on edge devices remains challenging due to their massive parameter scale and high resource demands. While post-training quantization (PTQ) has emerged as a widely adopted technique for compressing AR LLMs, its applicability to dLLMs remains largely unexplored. In this work, we present the first systematic study on quantizing diffusion-based language models. We begin by identifying the presence of activation outliers, characterized by abnormally large activation values that dominate the dynamic range. These outliers pose a key challenge to low-bit quantization, as they make it difficult to preserve precision for the majority of values. More importantly, we implement state-of-the-art PTQ methods and conduct a comprehensive evaluation across multiple task types and model variants. Our analysis is structured along four key dimensions: bit-width, quantization method, task category, and model type. Through this multi-perspective evaluation, we offer practical insights into the quantization behavior of dLLMs under different configurations. We hope our findings provide a foundation for future research in efficient dLLM deployment. Our code is publicly available at https://github.com/FelixMessi/QDLM. |
| title | Quantization Meets dLLMs: A Systematic Study of Post-training Quantization for Diffusion LLMs |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2508.14896 |