MAR-GRPO: Stabilized GRPO for AR-diffusion Hybrid Image Generation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910112684179456 |
|---|---|
| author | Ma, Xiaoxiao Lei, Jiachen Ren, Tianfei Huang, Jie Fu, Siming Hao, Aiming Wu, Jiahong Chu, Xiangxiang Zhao, Feng |
| author_facet | Ma, Xiaoxiao Lei, Jiachen Ren, Tianfei Huang, Jie Fu, Siming Hao, Aiming Wu, Jiahong Chu, Xiangxiang Zhao, Feng |
| contents | Reinforcement learning (RL) has been successfully applied to autoregressive (AR) and diffusion models. However, extending RL to hybrid AR-diffusion frameworks remains challenging due to interleaved inference and noisy log-probability estimation. In this work, we study masked autoregressive models (MAR) and show that the diffusion head plays a critical role in training dynamics, often introducing noisy gradients that lead to instability and early performance saturation. To address this issue, we propose a stabilized RL framework for MAR. We introduce multi-trajectory expectation (MTE), which estimates the optimization direction by averaging over multiple diffusion trajectories, thereby reducing diffusion-induced gradient noise. To avoid over-smoothing, we further estimate token-wise uncertainty from multiple trajectories and apply multi-trajectory optimization only to the top-k% uncertain tokens. In addition, we introduce a consistency-aware token selection strategy that filters out AR tokens that are less aligned with the final generated content. Extensive experiments across multiple benchmarks demonstrate that our method consistently improves visual quality, training stability, and spatial structure understanding over baseline GRPO and pre-RL models. Code is available at: https://github.com/AMAP-ML/mar-grpo. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_06966 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | MAR-GRPO: Stabilized GRPO for AR-diffusion Hybrid Image Generation Ma, Xiaoxiao Lei, Jiachen Ren, Tianfei Huang, Jie Fu, Siming Hao, Aiming Wu, Jiahong Chu, Xiangxiang Zhao, Feng Computer Vision and Pattern Recognition Reinforcement learning (RL) has been successfully applied to autoregressive (AR) and diffusion models. However, extending RL to hybrid AR-diffusion frameworks remains challenging due to interleaved inference and noisy log-probability estimation. In this work, we study masked autoregressive models (MAR) and show that the diffusion head plays a critical role in training dynamics, often introducing noisy gradients that lead to instability and early performance saturation. To address this issue, we propose a stabilized RL framework for MAR. We introduce multi-trajectory expectation (MTE), which estimates the optimization direction by averaging over multiple diffusion trajectories, thereby reducing diffusion-induced gradient noise. To avoid over-smoothing, we further estimate token-wise uncertainty from multiple trajectories and apply multi-trajectory optimization only to the top-k% uncertain tokens. In addition, we introduce a consistency-aware token selection strategy that filters out AR tokens that are less aligned with the final generated content. Extensive experiments across multiple benchmarks demonstrate that our method consistently improves visual quality, training stability, and spatial structure understanding over baseline GRPO and pre-RL models. Code is available at: https://github.com/AMAP-ML/mar-grpo. |
| title | MAR-GRPO: Stabilized GRPO for AR-diffusion Hybrid Image Generation |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2604.06966 |