LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Models
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866917006288093184 |
|---|---|
| author | Zhu, Fengqi Wang, Rongzhen Nie, Shen Zhang, Xiaolu Wu, Chunwei Hu, Jun Zhou, Jun Chen, Jianfei Lin, Yankai Wen, Ji-Rong Li, Chongxuan |
| author_facet | Zhu, Fengqi Wang, Rongzhen Nie, Shen Zhang, Xiaolu Wu, Chunwei Hu, Jun Zhou, Jun Chen, Jianfei Lin, Yankai Wen, Ji-Rong Li, Chongxuan |
| contents | While Masked Diffusion Models (MDMs), such as LLaDA, present a promising paradigm for language modeling, there has been relatively little effort in aligning these models with human preferences via reinforcement learning. The challenge primarily arises from the high variance in Evidence Lower Bound (ELBO)-based likelihood estimates required for preference optimization. To address this issue, we propose Variance-Reduced Preference Optimization (VRPO), a framework that formally analyzes the variance of ELBO estimators and derives bounds on both the bias and variance of preference optimization gradients. Building on this theoretical foundation, we introduce unbiased variance reduction strategies, including optimal Monte Carlo budget allocation and antithetic sampling, that significantly improve the performance of MDM alignment. We demonstrate the effectiveness of VRPO by applying it to LLaDA, and the resulting model, LLaDA 1.5, outperforms its SFT-only predecessor consistently and significantly across mathematical (GSM8K +4.7), code (HumanEval +3.0, MBPP +1.8), and alignment benchmarks (IFEval +4.0, Arena-Hard +4.3). Furthermore, LLaDA 1.5 demonstrates a highly competitive mathematical performance compared to strong language MDMs and ARMs. Project page: https://ml-gsai.github.io/LLaDA-1.5-Demo/. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_19223 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Models Zhu, Fengqi Wang, Rongzhen Nie, Shen Zhang, Xiaolu Wu, Chunwei Hu, Jun Zhou, Jun Chen, Jianfei Lin, Yankai Wen, Ji-Rong Li, Chongxuan Machine Learning While Masked Diffusion Models (MDMs), such as LLaDA, present a promising paradigm for language modeling, there has been relatively little effort in aligning these models with human preferences via reinforcement learning. The challenge primarily arises from the high variance in Evidence Lower Bound (ELBO)-based likelihood estimates required for preference optimization. To address this issue, we propose Variance-Reduced Preference Optimization (VRPO), a framework that formally analyzes the variance of ELBO estimators and derives bounds on both the bias and variance of preference optimization gradients. Building on this theoretical foundation, we introduce unbiased variance reduction strategies, including optimal Monte Carlo budget allocation and antithetic sampling, that significantly improve the performance of MDM alignment. We demonstrate the effectiveness of VRPO by applying it to LLaDA, and the resulting model, LLaDA 1.5, outperforms its SFT-only predecessor consistently and significantly across mathematical (GSM8K +4.7), code (HumanEval +3.0, MBPP +1.8), and alignment benchmarks (IFEval +4.0, Arena-Hard +4.3). Furthermore, LLaDA 1.5 demonstrates a highly competitive mathematical performance compared to strong language MDMs and ARMs. Project page: https://ml-gsai.github.io/LLaDA-1.5-Demo/. |
| title | LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Models |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2505.19223 |