Continuous Latent Diffusion Language Model
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909022704107520 |
|---|---|
| author | Guo, Hongcan Zhao, Qinyu Zhao, Yian Nie, Shen Zhu, Rui Guo, Qiushan Wang, Feng Yang, Tao Zhao, Hengshuang Wei, Guoqiang Zeng, Yan |
| author_facet | Guo, Hongcan Zhao, Qinyu Zhao, Yian Nie, Shen Zhu, Rui Guo, Qiushan Wang, Feng Yang, Tao Zhao, Hengshuang Wei, Guoqiang Zeng, Yan |
| contents | Large language models have achieved remarkable success under the autoregressive paradigm, yet high-quality text generation need not be tied to a fixed left-to-right order. Existing alternatives still struggle to jointly achieve generation efficiency, scalable representation learning, and effective global semantic modeling. We propose Cola DLM, a hierarchical latent diffusion language model that frames text generation through hierarchical information decomposition. Cola DLM first learns a stable text-to-latent mapping with a Text VAE, then models a global semantic prior in continuous latent space with a block-causal DiT, and finally generates text through conditional decoding. From a unified Markov-path perspective, its diffusion process performs latent prior transport rather than token-level observation recovery, thereby separating global semantic organization from local textual realization. This design yields a more flexible non-autoregressive inductive bias, supports semantic compression and prior fitting in continuous space, and naturally extends to other continuous modalities. Through experiments spanning 4 research questions, 8 benchmarks, strictly matched ~2B-parameter autoregressive and LLaDA baselines, and scaling curves up to about 2000 EFLOPs, we identify an effective overall configuration of Cola DLM and verify its strong scaling behavior for text generation. Taken together, the results establish hierarchical continuous latent prior modeling as a principled alternative to strictly token-level language modeling, where generation quality and scaling behavior may better reflect model capability than likelihood, while also suggesting a concrete path toward unified modeling across discrete text and continuous modalities. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_06548 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Continuous Latent Diffusion Language Model Guo, Hongcan Zhao, Qinyu Zhao, Yian Nie, Shen Zhu, Rui Guo, Qiushan Wang, Feng Yang, Tao Zhao, Hengshuang Wei, Guoqiang Zeng, Yan Computation and Language Artificial Intelligence Computer Vision and Pattern Recognition 68T50, 68T07 I.2.7; I.2.6 Large language models have achieved remarkable success under the autoregressive paradigm, yet high-quality text generation need not be tied to a fixed left-to-right order. Existing alternatives still struggle to jointly achieve generation efficiency, scalable representation learning, and effective global semantic modeling. We propose Cola DLM, a hierarchical latent diffusion language model that frames text generation through hierarchical information decomposition. Cola DLM first learns a stable text-to-latent mapping with a Text VAE, then models a global semantic prior in continuous latent space with a block-causal DiT, and finally generates text through conditional decoding. From a unified Markov-path perspective, its diffusion process performs latent prior transport rather than token-level observation recovery, thereby separating global semantic organization from local textual realization. This design yields a more flexible non-autoregressive inductive bias, supports semantic compression and prior fitting in continuous space, and naturally extends to other continuous modalities. Through experiments spanning 4 research questions, 8 benchmarks, strictly matched ~2B-parameter autoregressive and LLaDA baselines, and scaling curves up to about 2000 EFLOPs, we identify an effective overall configuration of Cola DLM and verify its strong scaling behavior for text generation. Taken together, the results establish hierarchical continuous latent prior modeling as a principled alternative to strictly token-level language modeling, where generation quality and scaling behavior may better reflect model capability than likelihood, while also suggesting a concrete path toward unified modeling across discrete text and continuous modalities. |
| title | Continuous Latent Diffusion Language Model |
| topic | Computation and Language Artificial Intelligence Computer Vision and Pattern Recognition 68T50, 68T07 I.2.7; I.2.6 |
| url | https://arxiv.org/abs/2605.06548 |