Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | https://arxiv.org/abs/2412.21059 |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866909980454551552 |
|---|---|
| author | Xu, Jiazheng Huang, Yu Cheng, Jiale Yang, Yuanming Xu, Jiajun Wang, Yuan Duan, Wenbo Yang, Shen Jin, Qunlin Li, Shurun Teng, Jiayan Yang, Zhuoyi Zheng, Wendi Liu, Xiao Zhang, Dan Ding, Ming Zhang, Xiaohan Gu, Xiaotao Huang, Shiyu Huang, Minlie Tang, Jie Dong, Yuxiao |
| author_facet | Xu, Jiazheng Huang, Yu Cheng, Jiale Yang, Yuanming Xu, Jiajun Wang, Yuan Duan, Wenbo Yang, Shen Jin, Qunlin Li, Shurun Teng, Jiayan Yang, Zhuoyi Zheng, Wendi Liu, Xiao Zhang, Dan Ding, Ming Zhang, Xiaohan Gu, Xiaotao Huang, Shiyu Huang, Minlie Tang, Jie Dong, Yuxiao |
| contents | Visual generative models have achieved remarkable progress in synthesizing photorealistic images and videos, yet aligning their outputs with human preferences across critical dimensions remains a persistent challenge. Though reinforcement learning from human feedback offers promise for preference alignment, existing reward models for visual generation face limitations, including black-box scoring without interpretability and potentially resultant unexpected biases. We present VisionReward, a general framework for learning human visual preferences in both image and video generation. Specifically, we employ a hierarchical visual assessment framework to capture fine-grained human preferences, and leverages linear weighting to enable interpretable preference learning. Furthermore, we propose a multi-dimensional consistent strategy when using VisionReward as a reward model during preference optimization for visual generation. Experiments show that VisionReward can significantly outperform existing image and video reward models on both machine metrics and human evaluation. Notably, VisionReward surpasses VideoScore by 17.2% in preference prediction accuracy, and text-to-video models with VisionReward achieve a 31.6% higher pairwise win rate compared to the same models using VideoScore. All code and datasets are provided at https://github.com/THUDM/VisionReward. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_21059 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | VisionReward: Fine-Grained Multi-Dimensional Human Preference Learning for Image and Video Generation Xu, Jiazheng Huang, Yu Cheng, Jiale Yang, Yuanming Xu, Jiajun Wang, Yuan Duan, Wenbo Yang, Shen Jin, Qunlin Li, Shurun Teng, Jiayan Yang, Zhuoyi Zheng, Wendi Liu, Xiao Zhang, Dan Ding, Ming Zhang, Xiaohan Gu, Xiaotao Huang, Shiyu Huang, Minlie Tang, Jie Dong, Yuxiao Computer Vision and Pattern Recognition Visual generative models have achieved remarkable progress in synthesizing photorealistic images and videos, yet aligning their outputs with human preferences across critical dimensions remains a persistent challenge. Though reinforcement learning from human feedback offers promise for preference alignment, existing reward models for visual generation face limitations, including black-box scoring without interpretability and potentially resultant unexpected biases. We present VisionReward, a general framework for learning human visual preferences in both image and video generation. Specifically, we employ a hierarchical visual assessment framework to capture fine-grained human preferences, and leverages linear weighting to enable interpretable preference learning. Furthermore, we propose a multi-dimensional consistent strategy when using VisionReward as a reward model during preference optimization for visual generation. Experiments show that VisionReward can significantly outperform existing image and video reward models on both machine metrics and human evaluation. Notably, VisionReward surpasses VideoScore by 17.2% in preference prediction accuracy, and text-to-video models with VisionReward achieve a 31.6% higher pairwise win rate compared to the same models using VideoScore. All code and datasets are provided at https://github.com/THUDM/VisionReward. |
| title | VisionReward: Fine-Grained Multi-Dimensional Human Preference Learning for Image and Video Generation |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2412.21059 |