Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2601.15593 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917399492558848 |
|---|---|
| author | Zhong, Yangyang Gu, Yanmei Zang, Zhengqing Li, Xiaomeng Ding, Yuqi Jia, Xibei Shen, Yuting Lan, Zhenzhong Zhu, Liwang Liu, Weiping Zhou, Junlin Liu, Haisheng Yu, Zhong Xin Luo, Pengxin Qi, Donglian Yan, Yunfeng Zhao, Junbo |
| author_facet | Zhong, Yangyang Gu, Yanmei Zang, Zhengqing Li, Xiaomeng Ding, Yuqi Jia, Xibei Shen, Yuting Lan, Zhenzhong Zhu, Liwang Liu, Weiping Zhou, Junlin Liu, Haisheng Yu, Zhong Xin Luo, Pengxin Qi, Donglian Yan, Yunfeng Zhao, Junbo |
| contents | Masked Diffusion Language Models (MDLMs) promise parallel token generation and arbitrary-order decoding, yet it remains unclear to what extent current models truly realize these capabilities. We characterize MDLM behavior along two dimensions -- parallelism strength and generation order -- using Average Finalization Parallelism (AFP) and Kendall's tau. We evaluate eight mainstream MDLMs (up to 100B parameters) on 58 benchmarks spanning knowledge, reasoning, and programming. The results show that MDLMs still lag behind comparably sized autoregressive models, mainly because parallel probabilistic modeling weakens inter-token dependencies. Meanwhile, MDLMs exhibit adaptive decoding behavior: their parallelism and generation order vary significantly with the task domain, the stage of reasoning, and whether the output is correct. On tasks that require "backward information" (e.g., Sudoku), MDLMs adopt a solution order that tends to fill easier Sudoku blanks first, highlighting their advantages. Finally, we provide theoretical motivation and design insights supporting a Generate-then-Edit paradigm, which mitigates dependency loss while retaining the efficiency of parallel decoding. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_15593 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Parallelism and Generation Order in Masked Diffusion Language Models: Limits Today, Potential Tomorrow Zhong, Yangyang Gu, Yanmei Zang, Zhengqing Li, Xiaomeng Ding, Yuqi Jia, Xibei Shen, Yuting Lan, Zhenzhong Zhu, Liwang Liu, Weiping Zhou, Junlin Liu, Haisheng Yu, Zhong Xin Luo, Pengxin Qi, Donglian Yan, Yunfeng Zhao, Junbo Computation and Language Artificial Intelligence Machine Learning Masked Diffusion Language Models (MDLMs) promise parallel token generation and arbitrary-order decoding, yet it remains unclear to what extent current models truly realize these capabilities. We characterize MDLM behavior along two dimensions -- parallelism strength and generation order -- using Average Finalization Parallelism (AFP) and Kendall's tau. We evaluate eight mainstream MDLMs (up to 100B parameters) on 58 benchmarks spanning knowledge, reasoning, and programming. The results show that MDLMs still lag behind comparably sized autoregressive models, mainly because parallel probabilistic modeling weakens inter-token dependencies. Meanwhile, MDLMs exhibit adaptive decoding behavior: their parallelism and generation order vary significantly with the task domain, the stage of reasoning, and whether the output is correct. On tasks that require "backward information" (e.g., Sudoku), MDLMs adopt a solution order that tends to fill easier Sudoku blanks first, highlighting their advantages. Finally, we provide theoretical motivation and design insights supporting a Generate-then-Edit paradigm, which mitigates dependency loss while retaining the efficiency of parallel decoding. |
| title | Parallelism and Generation Order in Masked Diffusion Language Models: Limits Today, Potential Tomorrow |
| topic | Computation and Language Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2601.15593 |