Guardado en:
| Autores principales: | , , , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | https://arxiv.org/abs/2603.18806 |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866917402725318656 |
|---|---|
| author | Zhang, Wenxuan Wu, Lemeng Zhao, Changsheng Chang, Ernie Zhuge, Mingchen Liu, Zechun Su, Andy Huang, Hanxian Chen, Jun Zhou, Chong Krishnamoorthi, Raghuraman Chandra, Vikas Elhoseiny, Mohamed Wen, Wei |
| author_facet | Zhang, Wenxuan Wu, Lemeng Zhao, Changsheng Chang, Ernie Zhuge, Mingchen Liu, Zechun Su, Andy Huang, Hanxian Chen, Jun Zhou, Chong Krishnamoorthi, Raghuraman Chandra, Vikas Elhoseiny, Mohamed Wen, Wei |
| contents | Diffusion Large Language Models (dLLMs) introduce a new paradigm for language generation, which in turn presents new challenges for aligning them with human preferences. In this work, we aim to improve the policy optimization for dLLMs by reducing the cost of the trajectory probability calculation, thereby enabling scaled-up offline policy training. We prove that: (i) under reference policy regularization, the probability ratio of the newly unmasked tokens is an unbiased estimate of that of intermediate diffusion states, and (ii) the probability of the full trajectory can be effectively estimated with a single forward pass of a re-masked final state. By integrating these two trajectory reduction strategies into a policy optimization objective, we propose Trajectory Reduction Policy Optimization (dTRPO). We evaluate dTRPO on 7B dLLMs across instruction-following and reasoning benchmarks. Results show that it substantially improves the core performance of state-of-the-art dLLMs, achieving gains of up to 9.6% on STEM tasks, up to 4.3% on coding tasks, and up to 3.0% on instruction-following tasks. Moreover, dTRPO exhibits strong training efficiency due to its offline, single-forward nature, and achieves improved generation efficiency through high-quality outputs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_18806 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | dTRPO: Trajectory Reduction in Policy Optimization of Diffusion Large Language Models Zhang, Wenxuan Wu, Lemeng Zhao, Changsheng Chang, Ernie Zhuge, Mingchen Liu, Zechun Su, Andy Huang, Hanxian Chen, Jun Zhou, Chong Krishnamoorthi, Raghuraman Chandra, Vikas Elhoseiny, Mohamed Wen, Wei Artificial Intelligence Diffusion Large Language Models (dLLMs) introduce a new paradigm for language generation, which in turn presents new challenges for aligning them with human preferences. In this work, we aim to improve the policy optimization for dLLMs by reducing the cost of the trajectory probability calculation, thereby enabling scaled-up offline policy training. We prove that: (i) under reference policy regularization, the probability ratio of the newly unmasked tokens is an unbiased estimate of that of intermediate diffusion states, and (ii) the probability of the full trajectory can be effectively estimated with a single forward pass of a re-masked final state. By integrating these two trajectory reduction strategies into a policy optimization objective, we propose Trajectory Reduction Policy Optimization (dTRPO). We evaluate dTRPO on 7B dLLMs across instruction-following and reasoning benchmarks. Results show that it substantially improves the core performance of state-of-the-art dLLMs, achieving gains of up to 9.6% on STEM tasks, up to 4.3% on coding tasks, and up to 3.0% on instruction-following tasks. Moreover, dTRPO exhibits strong training efficiency due to its offline, single-forward nature, and achieves improved generation efficiency through high-quality outputs. |
| title | dTRPO: Trajectory Reduction in Policy Optimization of Diffusion Large Language Models |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2603.18806 |