Power Reinforcement Post-Training of Text-to-Image Models with Super-Linear Advantage Shaping
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914554670219264 |
|---|---|
| author | Sun, Haoyuan Wang, Jing Song, Yuxin Lu, Yu Fang, Bo Luo, Yifu Yin, Jun Zeng, Pengyu Zhang, Miao Zhang, Tiantian Wang, Xueqian Lu, Shijian |
| author_facet | Sun, Haoyuan Wang, Jing Song, Yuxin Lu, Yu Fang, Bo Luo, Yifu Yin, Jun Zeng, Pengyu Zhang, Miao Zhang, Tiantian Wang, Xueqian Lu, Shijian |
| contents | Recently, post-training methods based on reinforcement learning, with a particular focus on Group Relative Policy Optimization (GRPO), have emerged as the robust paradigm for further advancement of text-to-image (T2I) models. However, these methods are often prone to reward hacking, wherein models exploit biases in imperfect reward functions rather than yielding genuine performance gains. In this work, we identify that normalization could lead to miscalibration and directly removing the prompt-level standard deviation term yields an optimal policy ascent direction that is linear in the advantage but still limits the separation of genuine signals from noise. To mitigate the above issues, we propose Super-Linear Advantage Shaping (SLAS) by revisiting the functional update from an information geometry perspective. By extending the Fisher-Rao information metric with advantage-dependent weighting, SLAS introduces a non-linear geometric structure that reshapes the local policy space. This design relaxes constraints along high-advantage directions to amplify informative updates, while tightening those in low-advantage regions to suppress illusory gradients. In addition, batch-level normalization is applied to stabilize training under varying reward scales. Extensive evaluations demonstrate that SLAS consistently surpasses the DanceGRPO baseline across multiple backbones and benchmarks. In particular, it yields faster training dynamics, improved out-of-domain performance on GenEval and UniGenBench++, and enhanced robustness to model scaling, while mitigating reward hacking and preserving semantic and compositional fidelity in generations. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_10937 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Power Reinforcement Post-Training of Text-to-Image Models with Super-Linear Advantage Shaping Sun, Haoyuan Wang, Jing Song, Yuxin Lu, Yu Fang, Bo Luo, Yifu Yin, Jun Zeng, Pengyu Zhang, Miao Zhang, Tiantian Wang, Xueqian Lu, Shijian Computer Vision and Pattern Recognition Recently, post-training methods based on reinforcement learning, with a particular focus on Group Relative Policy Optimization (GRPO), have emerged as the robust paradigm for further advancement of text-to-image (T2I) models. However, these methods are often prone to reward hacking, wherein models exploit biases in imperfect reward functions rather than yielding genuine performance gains. In this work, we identify that normalization could lead to miscalibration and directly removing the prompt-level standard deviation term yields an optimal policy ascent direction that is linear in the advantage but still limits the separation of genuine signals from noise. To mitigate the above issues, we propose Super-Linear Advantage Shaping (SLAS) by revisiting the functional update from an information geometry perspective. By extending the Fisher-Rao information metric with advantage-dependent weighting, SLAS introduces a non-linear geometric structure that reshapes the local policy space. This design relaxes constraints along high-advantage directions to amplify informative updates, while tightening those in low-advantage regions to suppress illusory gradients. In addition, batch-level normalization is applied to stabilize training under varying reward scales. Extensive evaluations demonstrate that SLAS consistently surpasses the DanceGRPO baseline across multiple backbones and benchmarks. In particular, it yields faster training dynamics, improved out-of-domain performance on GenEval and UniGenBench++, and enhanced robustness to model scaling, while mitigating reward hacking and preserving semantic and compositional fidelity in generations. |
| title | Power Reinforcement Post-Training of Text-to-Image Models with Super-Linear Advantage Shaping |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2605.10937 |