Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2603.06743 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911493962858496 |
|---|---|
| author | Zhong, Jianyuan Wang, Kaibo Ding, Ding Feng, Zijin Bai, Haoli Xiang, Yang Sun, Jiacheng Xu, Qiang |
| author_facet | Zhong, Jianyuan Wang, Kaibo Ding, Ding Feng, Zijin Bai, Haoli Xiang, Yang Sun, Jiacheng Xu, Qiang |
| contents | Group Relative Policy Optimization (GRPO) is highly effective for post-training autoregressive (AR) language models, yet its direct application to diffusion large language models (dLLMs) often triggers reward collapse. We identify two sources of incompatibility. First, GRPO relies on importance ratios defined by sequence probabilities, which are intractable in dLLMs and must be estimated (e.g., via ELBO-based or mean-field likelihood proxies), yielding inherently noisy ratios. Second, standard GRPO's formulation is not designed for estimated ratios: its conditional clipping can be anomalously bypassed by model-agnostic estimation noise, producing gradient spikes, while its fixed group-size normalization amplifies gradient-magnitude fluctuations under high-variance ratio estimates. We show these effects form a self-reinforcing instability loop that drives policy drift and further increases ratio variance. To break this loop, we propose StableDRL, a reformulation of GRPO tailored for dLLMs that uses (i) unconditional clipping to suppress outlier-induced spikes and (ii) self-normalization to constrain updates within the convex hull of per-sample gradients. We further extend StableDRL to block-wise diffusion models via a staircase attention mechanism. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_06743 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Stabilizing Reinforcement Learning for Diffusion Language Models Zhong, Jianyuan Wang, Kaibo Ding, Ding Feng, Zijin Bai, Haoli Xiang, Yang Sun, Jiacheng Xu, Qiang Machine Learning Artificial Intelligence Group Relative Policy Optimization (GRPO) is highly effective for post-training autoregressive (AR) language models, yet its direct application to diffusion large language models (dLLMs) often triggers reward collapse. We identify two sources of incompatibility. First, GRPO relies on importance ratios defined by sequence probabilities, which are intractable in dLLMs and must be estimated (e.g., via ELBO-based or mean-field likelihood proxies), yielding inherently noisy ratios. Second, standard GRPO's formulation is not designed for estimated ratios: its conditional clipping can be anomalously bypassed by model-agnostic estimation noise, producing gradient spikes, while its fixed group-size normalization amplifies gradient-magnitude fluctuations under high-variance ratio estimates. We show these effects form a self-reinforcing instability loop that drives policy drift and further increases ratio variance. To break this loop, we propose StableDRL, a reformulation of GRPO tailored for dLLMs that uses (i) unconditional clipping to suppress outlier-induced spikes and (ii) self-normalization to constrain updates within the convex hull of per-sample gradients. We further extend StableDRL to block-wise diffusion models via a staircase attention mechanism. |
| title | Stabilizing Reinforcement Learning for Diffusion Language Models |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2603.06743 |