TextLDM: Language Modeling with Continuous Latent Diffusion
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917473066942464 |
|---|---|
| author | Jiang, Jiaxiu Ren, Jingjing Li, Wenbo Wang, Bo Sun, Haoze Yang, Yijun Liu, Jianhui Zhang, Yanbing Zheng, Shenghe Zhang, Yuan Huang, Haoyang Duan, Nan Zuo, Wangmeng |
| author_facet | Jiang, Jiaxiu Ren, Jingjing Li, Wenbo Wang, Bo Sun, Haoze Yang, Yijun Liu, Jianhui Zhang, Yanbing Zheng, Shenghe Zhang, Yuan Huang, Haoyang Duan, Nan Zuo, Wangmeng |
| contents | Diffusion Transformers (DiT) trained with flow matching in a VAE latent space have unified visual generation across images and videos. A natural next step toward a single architecture for both generation (visual synthesis) and understanding (text generation) is to apply this framework to language modeling. We propose TextLDM, which transfers the visual latent diffusion recipe to text generation with minimal architectural modification. A Transformer-based VAE maps discrete tokens to continuous latents, enhanced by Representation Alignment (REPA) with a frozen pretrained language model to produce representations effective for conditional denoising. A standard DiT then performs flow matching in this latent space, identical in architecture to its visual counterpart. The central challenge we address is obtaining high-quality continuous text representations: we find that reconstruction fidelity alone is insufficient, and that aligning latent features with a pretrained language model via REPA is critical for downstream generation quality. Trained from scratch on OpenWebText2, TextLDM substantially outperforms prior diffusion language models and matches GPT-2 under the same settings. Our results establish that the visual DiT recipe transfers effectively to language, taking a concrete step toward unified diffusion architectures for multimodal generation and understanding. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_07748 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | TextLDM: Language Modeling with Continuous Latent Diffusion Jiang, Jiaxiu Ren, Jingjing Li, Wenbo Wang, Bo Sun, Haoze Yang, Yijun Liu, Jianhui Zhang, Yanbing Zheng, Shenghe Zhang, Yuan Huang, Haoyang Duan, Nan Zuo, Wangmeng Computation and Language Diffusion Transformers (DiT) trained with flow matching in a VAE latent space have unified visual generation across images and videos. A natural next step toward a single architecture for both generation (visual synthesis) and understanding (text generation) is to apply this framework to language modeling. We propose TextLDM, which transfers the visual latent diffusion recipe to text generation with minimal architectural modification. A Transformer-based VAE maps discrete tokens to continuous latents, enhanced by Representation Alignment (REPA) with a frozen pretrained language model to produce representations effective for conditional denoising. A standard DiT then performs flow matching in this latent space, identical in architecture to its visual counterpart. The central challenge we address is obtaining high-quality continuous text representations: we find that reconstruction fidelity alone is insufficient, and that aligning latent features with a pretrained language model via REPA is critical for downstream generation quality. Trained from scratch on OpenWebText2, TextLDM substantially outperforms prior diffusion language models and matches GPT-2 under the same settings. Our results establish that the visual DiT recipe transfers effectively to language, taking a concrete step toward unified diffusion architectures for multimodal generation and understanding. |
| title | TextLDM: Language Modeling with Continuous Latent Diffusion |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2605.07748 |