Stop-Think-AutoRegress: Language Modeling with Latent Diffusion Planning
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917290935582720 |
|---|---|
| author | Lovelace, Justin Belardi, Christian Zalouk, Sofian Polavaram, Adhitya Kundurthy, Srivatsa Weinberger, Kilian Q. |
| author_facet | Lovelace, Justin Belardi, Christian Zalouk, Sofian Polavaram, Adhitya Kundurthy, Srivatsa Weinberger, Kilian Q. |
| contents | The Stop-Think-AutoRegress Language Diffusion Model (STAR-LDM) integrates latent diffusion planning with autoregressive generation. Unlike conventional autoregressive language models limited to token-by-token decisions, STAR-LDM incorporates a "thinking" phase that pauses generation to refine a semantic plan through diffusion before continuing. This enables global planning in continuous space prior to committing to discrete tokens. Evaluations show STAR-LDM significantly outperforms similar-sized models on language understanding benchmarks and achieves $>70\%$ win rates in LLM-as-judge comparisons for narrative coherence and commonsense reasoning. The architecture also allows straightforward control through lightweight classifiers, enabling fine-grained steering of attributes without model retraining while maintaining better fluency-control trade-offs than specialized approaches. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_20528 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Stop-Think-AutoRegress: Language Modeling with Latent Diffusion Planning Lovelace, Justin Belardi, Christian Zalouk, Sofian Polavaram, Adhitya Kundurthy, Srivatsa Weinberger, Kilian Q. Computation and Language Machine Learning The Stop-Think-AutoRegress Language Diffusion Model (STAR-LDM) integrates latent diffusion planning with autoregressive generation. Unlike conventional autoregressive language models limited to token-by-token decisions, STAR-LDM incorporates a "thinking" phase that pauses generation to refine a semantic plan through diffusion before continuing. This enables global planning in continuous space prior to committing to discrete tokens. Evaluations show STAR-LDM significantly outperforms similar-sized models on language understanding benchmarks and achieves $>70\%$ win rates in LLM-as-judge comparisons for narrative coherence and commonsense reasoning. The architecture also allows straightforward control through lightweight classifiers, enabling fine-grained steering of attributes without model retraining while maintaining better fluency-control trade-offs than specialized approaches. |
| title | Stop-Think-AutoRegress: Language Modeling with Latent Diffusion Planning |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2602.20528 |