VAR RL Done Right: Tackling Asynchronous Policy Conflicts in Visual Autoregressive Generation
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866915708817899520 |
|---|---|
| author | Sun, Shikun Qu, Liao Zhang, Huichao Liu, Yiheng Song, Yangyang Li, Xian Wang, Xu Jiang, Yi Du, Daniel K. Wu, Xinglong Jia, Jia |
| author_facet | Sun, Shikun Qu, Liao Zhang, Huichao Liu, Yiheng Song, Yangyang Li, Xian Wang, Xu Jiang, Yi Du, Daniel K. Wu, Xinglong Jia, Jia |
| contents | Visual generation is dominated by three paradigms: AutoRegressive (AR), diffusion, and Visual AutoRegressive (VAR) models. Unlike AR and diffusion, VARs operate on heterogeneous input structures across their generation steps, which creates severe asynchronous policy conflicts. This issue becomes particularly acute in reinforcement learning (RL) scenarios, leading to unstable training and suboptimal alignment. To resolve this, we propose a novel framework to enhance Group Relative Policy Optimization (GRPO) by explicitly managing these conflicts. Our method integrates three synergistic components: 1) a stabilizing intermediate reward to guide early-stage generation; 2) a dynamic time-step reweighting scheme for precise credit assignment; and 3) a novel mask propagation algorithm, derived from principles of Reward Feedback Learning (ReFL), designed to isolate optimization effects both spatially and temporally. Our approach demonstrates significant improvements in sample quality and objective alignment over the vanilla GRPO baseline, enabling robust and effective optimization for VAR models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_02256 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | VAR RL Done Right: Tackling Asynchronous Policy Conflicts in Visual Autoregressive Generation Sun, Shikun Qu, Liao Zhang, Huichao Liu, Yiheng Song, Yangyang Li, Xian Wang, Xu Jiang, Yi Du, Daniel K. Wu, Xinglong Jia, Jia Computer Vision and Pattern Recognition Machine Learning Visual generation is dominated by three paradigms: AutoRegressive (AR), diffusion, and Visual AutoRegressive (VAR) models. Unlike AR and diffusion, VARs operate on heterogeneous input structures across their generation steps, which creates severe asynchronous policy conflicts. This issue becomes particularly acute in reinforcement learning (RL) scenarios, leading to unstable training and suboptimal alignment. To resolve this, we propose a novel framework to enhance Group Relative Policy Optimization (GRPO) by explicitly managing these conflicts. Our method integrates three synergistic components: 1) a stabilizing intermediate reward to guide early-stage generation; 2) a dynamic time-step reweighting scheme for precise credit assignment; and 3) a novel mask propagation algorithm, derived from principles of Reward Feedback Learning (ReFL), designed to isolate optimization effects both spatially and temporally. Our approach demonstrates significant improvements in sample quality and objective alignment over the vanilla GRPO baseline, enabling robust and effective optimization for VAR models. |
| title | VAR RL Done Right: Tackling Asynchronous Policy Conflicts in Visual Autoregressive Generation |
| topic | Computer Vision and Pattern Recognition Machine Learning |
| url | https://arxiv.org/abs/2601.02256 |