Improving Video Generation with Human Feedback
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866914115651371008 |
|---|---|
| author | Liu, Jie Liu, Gongye Liang, Jiajun Yuan, Ziyang Liu, Xiaokun Zheng, Mingwu Wu, Xiele Wang, Qiulin Xia, Menghan Wang, Xintao Liu, Xiaohong Yang, Fei Wan, Pengfei Zhang, Di Gai, Kun Yang, Yujiu Ouyang, Wanli |
| author_facet | Liu, Jie Liu, Gongye Liang, Jiajun Yuan, Ziyang Liu, Xiaokun Zheng, Mingwu Wu, Xiele Wang, Qiulin Xia, Menghan Wang, Xintao Liu, Xiaohong Yang, Fei Wan, Pengfei Zhang, Di Gai, Kun Yang, Yujiu Ouyang, Wanli |
| contents | Video generation has achieved significant advances through rectified flow techniques, but issues like unsmooth motion and misalignment between videos and prompts persist. In this work, we develop a systematic pipeline that harnesses human feedback to mitigate these problems and refine the video generation model. Specifically, we begin by constructing a large-scale human preference dataset focused on modern video generation models, incorporating pairwise annotations across multi-dimensions. We then introduce VideoReward, a multi-dimensional video reward model, and examine how annotations and various design choices impact its rewarding efficacy. From a unified reinforcement learning perspective aimed at maximizing reward with KL regularization, we introduce three alignment algorithms for flow-based models. These include two training-time strategies: direct preference optimization for flow (Flow-DPO) and reward weighted regression for flow (Flow-RWR), and an inference-time technique, Flow-NRG, which applies reward guidance directly to noisy videos. Experimental results indicate that VideoReward significantly outperforms existing reward models, and Flow-DPO demonstrates superior performance compared to both Flow-RWR and supervised fine-tuning methods. Additionally, Flow-NRG lets users assign custom weights to multiple objectives during inference, meeting personalized video quality needs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_13918 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Improving Video Generation with Human Feedback Liu, Jie Liu, Gongye Liang, Jiajun Yuan, Ziyang Liu, Xiaokun Zheng, Mingwu Wu, Xiele Wang, Qiulin Xia, Menghan Wang, Xintao Liu, Xiaohong Yang, Fei Wan, Pengfei Zhang, Di Gai, Kun Yang, Yujiu Ouyang, Wanli Computer Vision and Pattern Recognition Artificial Intelligence Graphics Machine Learning Video generation has achieved significant advances through rectified flow techniques, but issues like unsmooth motion and misalignment between videos and prompts persist. In this work, we develop a systematic pipeline that harnesses human feedback to mitigate these problems and refine the video generation model. Specifically, we begin by constructing a large-scale human preference dataset focused on modern video generation models, incorporating pairwise annotations across multi-dimensions. We then introduce VideoReward, a multi-dimensional video reward model, and examine how annotations and various design choices impact its rewarding efficacy. From a unified reinforcement learning perspective aimed at maximizing reward with KL regularization, we introduce three alignment algorithms for flow-based models. These include two training-time strategies: direct preference optimization for flow (Flow-DPO) and reward weighted regression for flow (Flow-RWR), and an inference-time technique, Flow-NRG, which applies reward guidance directly to noisy videos. Experimental results indicate that VideoReward significantly outperforms existing reward models, and Flow-DPO demonstrates superior performance compared to both Flow-RWR and supervised fine-tuning methods. Additionally, Flow-NRG lets users assign custom weights to multiple objectives during inference, meeting personalized video quality needs. |
| title | Improving Video Generation with Human Feedback |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Graphics Machine Learning |
| url | https://arxiv.org/abs/2501.13918 |