Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | https://arxiv.org/abs/2510.09541 |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866910129703616512 |
|---|---|
| author | Wang, Chenyu Rashidinejad, Paria Su, DiJia Jiang, Song Wang, Sid Zhao, Siyan Zhou, Cai Shen, Shannon Zejiang Chen, Feiyu Jaakkola, Tommi Tian, Yuandong Liu, Bo |
| author_facet | Wang, Chenyu Rashidinejad, Paria Su, DiJia Jiang, Song Wang, Sid Zhao, Siyan Zhou, Cai Shen, Shannon Zejiang Chen, Feiyu Jaakkola, Tommi Tian, Yuandong Liu, Bo |
| contents | Diffusion large language models (dLLMs) are emerging as an efficient alternative to autoregressive models due to their ability to decode multiple tokens in parallel. However, aligning dLLMs with human preferences or task-specific rewards via reinforcement learning (RL) is challenging because their intractable log-likelihood precludes the direct application of standard policy gradient methods. While prior work uses surrogates like the evidence lower bound (ELBO), these one-sided approximations can introduce significant policy gradient bias. To address this, we propose the Sandwiched Policy Gradient (SPG) that leverages both an upper and a lower bound of the true log-likelihood. Experiments show that SPG significantly outperforms baselines based on ELBO or one-step estimation. Specifically, SPG improves the accuracy over state-of-the-art RL methods for dLLMs by 3.6% in GSM8K, 2.6% in MATH500, 18.4% in Countdown and 27.0% in Sudoku. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_09541 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | SPG: Sandwiched Policy Gradient for Masked Diffusion Language Models Wang, Chenyu Rashidinejad, Paria Su, DiJia Jiang, Song Wang, Sid Zhao, Siyan Zhou, Cai Shen, Shannon Zejiang Chen, Feiyu Jaakkola, Tommi Tian, Yuandong Liu, Bo Computation and Language Artificial Intelligence Diffusion large language models (dLLMs) are emerging as an efficient alternative to autoregressive models due to their ability to decode multiple tokens in parallel. However, aligning dLLMs with human preferences or task-specific rewards via reinforcement learning (RL) is challenging because their intractable log-likelihood precludes the direct application of standard policy gradient methods. While prior work uses surrogates like the evidence lower bound (ELBO), these one-sided approximations can introduce significant policy gradient bias. To address this, we propose the Sandwiched Policy Gradient (SPG) that leverages both an upper and a lower bound of the true log-likelihood. Experiments show that SPG significantly outperforms baselines based on ELBO or one-step estimation. Specifically, SPG improves the accuracy over state-of-the-art RL methods for dLLMs by 3.6% in GSM8K, 2.6% in MATH500, 18.4% in Countdown and 27.0% in Sudoku. |
| title | SPG: Sandwiched Policy Gradient for Masked Diffusion Language Models |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2510.09541 |