Introspective Diffusion Language Models
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866908958010114048 |
|---|---|
| author | Yu, Yifan Jian, Yuqing Wang, Junxiong Zhou, Zhongzhu Zhuang, Donglin Fang, Xinyu Yanamandra, Sri Wu, Xiaoxia Wu, Qingyang Song, Shuaiwen Leon Dao, Tri Athiwaratkun, Ben Zou, James Lai, Fan Xu, Chenfeng |
| author_facet | Yu, Yifan Jian, Yuqing Wang, Junxiong Zhou, Zhongzhu Zhuang, Donglin Fang, Xinyu Yanamandra, Sri Wu, Xiaoxia Wu, Qingyang Song, Shuaiwen Leon Dao, Tri Athiwaratkun, Ben Zou, James Lai, Fan Xu, Chenfeng |
| contents | Diffusion language models promise parallel generation, yet still lag behind autoregressive (AR) models in quality. We stem this gap to a failure of introspective consistency: AR models agree with their own generations, while DLMs often do not. We define the introspective acceptance rate, which measures whether a model accepts its previously generated tokens. This reveals why AR training has a structural advantage: causal masking and logit shifting implicitly enforce introspective consistency. Motivated by this observation, we introduce Introspective Diffusion Language Model (I-DLM), a paradigm that retains diffusion-style parallel decoding while inheriting the introspective consistency of AR training. I-DLM uses a novel introspective strided decoding (ISD) algorithm, which enables the model to verify previously generated tokens while advancing new ones in the same forward pass. From a systems standpoint, we build I-DLM inference engine on AR-inherited optimizations and further customize it with a stationary-batch scheduler. To the best of our knowledge, I-DLM is the first DLM to match the quality of its same-scale AR counterpart while outperforming prior DLMs in both model quality and practical serving efficiency across 15 benchmarks. It reaches 69.6 on AIME-24 and 45.7 on LiveCodeBench-v6, exceeding LLaDA-2.1-mini (16B) by more than 26 and 15 points, respectively. Beyond quality, I-DLM is designed for the growing demand of large-concurrency serving, delivering about 3x higher throughput than prior state-of-the-art DLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_11035 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Introspective Diffusion Language Models Yu, Yifan Jian, Yuqing Wang, Junxiong Zhou, Zhongzhu Zhuang, Donglin Fang, Xinyu Yanamandra, Sri Wu, Xiaoxia Wu, Qingyang Song, Shuaiwen Leon Dao, Tri Athiwaratkun, Ben Zou, James Lai, Fan Xu, Chenfeng Artificial Intelligence Diffusion language models promise parallel generation, yet still lag behind autoregressive (AR) models in quality. We stem this gap to a failure of introspective consistency: AR models agree with their own generations, while DLMs often do not. We define the introspective acceptance rate, which measures whether a model accepts its previously generated tokens. This reveals why AR training has a structural advantage: causal masking and logit shifting implicitly enforce introspective consistency. Motivated by this observation, we introduce Introspective Diffusion Language Model (I-DLM), a paradigm that retains diffusion-style parallel decoding while inheriting the introspective consistency of AR training. I-DLM uses a novel introspective strided decoding (ISD) algorithm, which enables the model to verify previously generated tokens while advancing new ones in the same forward pass. From a systems standpoint, we build I-DLM inference engine on AR-inherited optimizations and further customize it with a stationary-batch scheduler. To the best of our knowledge, I-DLM is the first DLM to match the quality of its same-scale AR counterpart while outperforming prior DLMs in both model quality and practical serving efficiency across 15 benchmarks. It reaches 69.6 on AIME-24 and 45.7 on LiveCodeBench-v6, exceeding LLaDA-2.1-mini (16B) by more than 26 and 15 points, respectively. Beyond quality, I-DLM is designed for the growing demand of large-concurrency serving, delivering about 3x higher throughput than prior state-of-the-art DLMs. |
| title | Introspective Diffusion Language Models |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2604.11035 |