Causal Autoregressive Diffusion Language Model
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866914291824721920 |
|---|---|
| author | Ruan, Junhao Li, Bei Yin, Yongjing Huang, Pengcheng Chen, Xin Wang, Jingang Cai, Xunliang Xiao, Tong Zhu, JingBo |
| author_facet | Ruan, Junhao Li, Bei Yin, Yongjing Huang, Pengcheng Chen, Xin Wang, Jingang Cai, Xunliang Xiao, Tong Zhu, JingBo |
| contents | In this work, we propose Causal Autoregressive Diffusion (CARD), a novel framework that unifies the training efficiency of ARMs with the high-throughput inference of diffusion models. CARD reformulates the diffusion process within a strictly causal attention mask, enabling dense, per-token supervision in a single forward pass. To address the optimization instability of causal diffusion, we introduce a soft-tailed masking schema to preserve local context and a context-aware reweighting mechanism derived from signal-to-noise principles. This design enables dynamic parallel decoding, where the model leverages KV-caching to adaptively generate variable-length token sequences based on confidence. Empirically, CARD outperforms existing discrete diffusion baselines while reducing training latency by 3 $\times$ compared to block diffusion methods. Our results demonstrate that CARD achieves ARM-level data efficiency while unlocking the latency benefits of parallel generation, establishing a robust paradigm for next-generation efficient LLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_22031 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Causal Autoregressive Diffusion Language Model Ruan, Junhao Li, Bei Yin, Yongjing Huang, Pengcheng Chen, Xin Wang, Jingang Cai, Xunliang Xiao, Tong Zhu, JingBo Computation and Language In this work, we propose Causal Autoregressive Diffusion (CARD), a novel framework that unifies the training efficiency of ARMs with the high-throughput inference of diffusion models. CARD reformulates the diffusion process within a strictly causal attention mask, enabling dense, per-token supervision in a single forward pass. To address the optimization instability of causal diffusion, we introduce a soft-tailed masking schema to preserve local context and a context-aware reweighting mechanism derived from signal-to-noise principles. This design enables dynamic parallel decoding, where the model leverages KV-caching to adaptively generate variable-length token sequences based on confidence. Empirically, CARD outperforms existing discrete diffusion baselines while reducing training latency by 3 $\times$ compared to block diffusion methods. Our results demonstrate that CARD achieves ARM-level data efficiency while unlocking the latency benefits of parallel generation, establishing a robust paradigm for next-generation efficient LLMs. |
| title | Causal Autoregressive Diffusion Language Model |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2601.22031 |