FlowPortrait: Reinforcement Learning for Audio-Driven Portrait Video Generation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910035778469888 |
|---|---|
| author | Tan, Weiting Liu, Andy T. Tu, Ming Qu, Xinghua Koehn, Philipp Lu, Lu |
| author_facet | Tan, Weiting Liu, Andy T. Tu, Ming Qu, Xinghua Koehn, Philipp Lu, Lu |
| contents | Generating realistic talking-head videos remains challenging due to persistent issues such as imperfect lip synchronization, unnatural motion, and evaluation metrics that correlate poorly with human perception. We propose FlowPortrait, a reinforcement-learning framework for audio-driven portrait animation built on a multimodal backbone for autoregressive audio-to-video generation. FlowPortrait introduces a human-aligned evaluation system based on Multimodal Large Language Models (MLLMs) to assess lip-sync accuracy, expressiveness, and motion quality. These signals are combined with perceptual and temporal consistency regularizers to form a stable composite reward, which is used to post-train the generator via Group Relative Policy Optimization (GRPO). Extensive experiments, including both automatic evaluations and human preference studies, demonstrate that FlowPortrait consistently produces higher-quality talking-head videos, highlighting the effectiveness of reinforcement learning for portrait animation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_00159 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | FlowPortrait: Reinforcement Learning for Audio-Driven Portrait Video Generation Tan, Weiting Liu, Andy T. Tu, Ming Qu, Xinghua Koehn, Philipp Lu, Lu Computer Vision and Pattern Recognition Artificial Intelligence Multimedia Sound Generating realistic talking-head videos remains challenging due to persistent issues such as imperfect lip synchronization, unnatural motion, and evaluation metrics that correlate poorly with human perception. We propose FlowPortrait, a reinforcement-learning framework for audio-driven portrait animation built on a multimodal backbone for autoregressive audio-to-video generation. FlowPortrait introduces a human-aligned evaluation system based on Multimodal Large Language Models (MLLMs) to assess lip-sync accuracy, expressiveness, and motion quality. These signals are combined with perceptual and temporal consistency regularizers to form a stable composite reward, which is used to post-train the generator via Group Relative Policy Optimization (GRPO). Extensive experiments, including both automatic evaluations and human preference studies, demonstrate that FlowPortrait consistently produces higher-quality talking-head videos, highlighting the effectiveness of reinforcement learning for portrait animation. |
| title | FlowPortrait: Reinforcement Learning for Audio-Driven Portrait Video Generation |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Multimedia Sound |
| url | https://arxiv.org/abs/2603.00159 |