Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2601.23182 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910006297755648 |
|---|---|
| author | He, Siyang Wang, Qiqi Liu, Xiaoran Ma, Hongnan Shi, Yiwei Song, Yuerong Zhu, Ying Liang, Tianyi Huang, Zengfeng He, Ziwei Qiu, Xipeng |
| author_facet | He, Siyang Wang, Qiqi Liu, Xiaoran Ma, Hongnan Shi, Yiwei Song, Yuerong Zhu, Ying Liang, Tianyi Huang, Zengfeng He, Ziwei Qiu, Xipeng |
| contents | Despite the non-autoregressive potential of diffusion language models (dLLMs), existing decoding strategies demonstrate positional bias, failing to fully unlock the potential of arbitrary generation. In this work, we delve into the inherent spectral characteristics of dLLMs and present the first frequency-domain analysis showing that low-frequency components in hidden states primarily encode global structural information and long-range dependencies, while high-frequency components are responsible for characterizing local details. Based on this observation, we propose FourierSampler, which leverages a frequency-domain sliding window mechanism to dynamically guide the model to achieve a "structure-to-detail" generation. FourierSampler outperforms other inference enhancement strategies on LLADA and SDAR, achieving relative improvements of 20.4% on LLaDA1.5-8B and 16.0% on LLaDA-8B-Instruct. It notably surpasses similarly sized autoregressive models like Llama3.1-8B-Instruct. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_23182 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | FourierSampler: Unlocking Non-Autoregressive Potential in Diffusion Language Models via Frequency-Guided Generation He, Siyang Wang, Qiqi Liu, Xiaoran Ma, Hongnan Shi, Yiwei Song, Yuerong Zhu, Ying Liang, Tianyi Huang, Zengfeng He, Ziwei Qiu, Xipeng Computation and Language Despite the non-autoregressive potential of diffusion language models (dLLMs), existing decoding strategies demonstrate positional bias, failing to fully unlock the potential of arbitrary generation. In this work, we delve into the inherent spectral characteristics of dLLMs and present the first frequency-domain analysis showing that low-frequency components in hidden states primarily encode global structural information and long-range dependencies, while high-frequency components are responsible for characterizing local details. Based on this observation, we propose FourierSampler, which leverages a frequency-domain sliding window mechanism to dynamically guide the model to achieve a "structure-to-detail" generation. FourierSampler outperforms other inference enhancement strategies on LLADA and SDAR, achieving relative improvements of 20.4% on LLaDA1.5-8B and 16.0% on LLaDA-8B-Instruct. It notably surpasses similarly sized autoregressive models like Llama3.1-8B-Instruct. |
| title | FourierSampler: Unlocking Non-Autoregressive Potential in Diffusion Language Models via Frequency-Guided Generation |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2601.23182 |