Latent Refinement Decoding: Enhancing Diffusion-Based Language Models by Refining Belief States
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866909846781034496 |
|---|---|
| author | Zhu, Qinglin Yao, Yizhen Zhao, Runcong Xiang, Yanzheng Saseendran, Amrutha Jin, Chen Teare, Philip Liang, Bin He, Yulan Gui, Lin |
| author_facet | Zhu, Qinglin Yao, Yizhen Zhao, Runcong Xiang, Yanzheng Saseendran, Amrutha Jin, Chen Teare, Philip Liang, Bin He, Yulan Gui, Lin |
| contents | Autoregressive (AR) models remain the standard for natural language generation but still suffer from high latency due to strictly sequential decoding. Recent diffusion-inspired approaches, such as LlaDA and Dream, mitigate this by generating in parallel, yet they suffer from two core limitations: information loss, as predictive distributions for non-finalized tokens are discarded at each step, and premature commitment, where local decisions are made without sufficient global coordination. We introduce Latent Refinement Decoding (LRD), a two-stage framework with Latent Refinement and a Predictive Feedback Loop. The first stage maintains masked positions as distributional mixtures of predicted tokens and the mask embedding, allowing the model to establish more globally consistent beliefs. The second stage progressively finalizes confident tokens while retaining uncertain ones for iterative feedback. KL-divergence dynamics provide a principled and reliable criterion for convergence and early stopping. Experiments across coding (HumanEval +6.3, MBPP +2.6) and reasoning (GSM8K +2.9, MATH500 +3.8) show that LRD improves accuracy while delivering speedups of up to 10.6x, making it a strong and versatile alternative for parallel sequence generation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_11052 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Latent Refinement Decoding: Enhancing Diffusion-Based Language Models by Refining Belief States Zhu, Qinglin Yao, Yizhen Zhao, Runcong Xiang, Yanzheng Saseendran, Amrutha Jin, Chen Teare, Philip Liang, Bin He, Yulan Gui, Lin Computation and Language Autoregressive (AR) models remain the standard for natural language generation but still suffer from high latency due to strictly sequential decoding. Recent diffusion-inspired approaches, such as LlaDA and Dream, mitigate this by generating in parallel, yet they suffer from two core limitations: information loss, as predictive distributions for non-finalized tokens are discarded at each step, and premature commitment, where local decisions are made without sufficient global coordination. We introduce Latent Refinement Decoding (LRD), a two-stage framework with Latent Refinement and a Predictive Feedback Loop. The first stage maintains masked positions as distributional mixtures of predicted tokens and the mask embedding, allowing the model to establish more globally consistent beliefs. The second stage progressively finalizes confident tokens while retaining uncertain ones for iterative feedback. KL-divergence dynamics provide a principled and reliable criterion for convergence and early stopping. Experiments across coding (HumanEval +6.3, MBPP +2.6) and reasoning (GSM8K +2.9, MATH500 +3.8) show that LRD improves accuracy while delivering speedups of up to 10.6x, making it a strong and versatile alternative for parallel sequence generation. |
| title | Latent Refinement Decoding: Enhancing Diffusion-Based Language Models by Refining Belief States |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2510.11052 |