Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2503.09158 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909028597104640 |
|---|---|
| author | Zhao, Fufangchen Tan, Songbai Qiu, Xuerui Xun, Linrui Jiang, Wenhao Zheng, Jinkai Fan, Hehe Gao, Jian Yan, Danfeng Li, Ming |
| author_facet | Zhao, Fufangchen Tan, Songbai Qiu, Xuerui Xun, Linrui Jiang, Wenhao Zheng, Jinkai Fan, Hehe Gao, Jian Yan, Danfeng Li, Ming |
| contents | Existing video large language models (VLLMs) primarily leverage prompt agnostic visual encoders, which extract untargeted facial representations without awareness of the queried information, leading to the loss of task critical cues. To address this challenge, we propose FaVChat, the first VLLM designed for reasoning over subtle visual and dynamic facial cues. FaVChat introduces a hierarchical, prompt guided visual feature extraction framework that emphasizes question relevant information at three complementary levels. These multi level features are dynamically fused and injected into the LLM, enabling more accurate facial details reasoning To further improve learning efficiency under data scarcity, we propose Data Efficient GRPO, a reinforcement learning strategy that iteratively identifies high utility samples and maximizes the contribution of each instance via per instance utility estimation, substantially enhancing performance gains under limited supervision. We construct a large scale benchmark dataset FaVChat 170K, comprising approximately 60K high quality facial videos and 170K question answer pairs focusing on fine grained facial details. Extensive experiments, including zero shot evaluations on four facial understanding tasks, demonstrate that FaVChat consistently outperforms existing VLLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_09158 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | FaVChat: Hierarchical Prompt-Query Guided Facial Video Understanding with Data-Efficient GRPO Zhao, Fufangchen Tan, Songbai Qiu, Xuerui Xun, Linrui Jiang, Wenhao Zheng, Jinkai Fan, Hehe Gao, Jian Yan, Danfeng Li, Ming Computer Vision and Pattern Recognition Existing video large language models (VLLMs) primarily leverage prompt agnostic visual encoders, which extract untargeted facial representations without awareness of the queried information, leading to the loss of task critical cues. To address this challenge, we propose FaVChat, the first VLLM designed for reasoning over subtle visual and dynamic facial cues. FaVChat introduces a hierarchical, prompt guided visual feature extraction framework that emphasizes question relevant information at three complementary levels. These multi level features are dynamically fused and injected into the LLM, enabling more accurate facial details reasoning To further improve learning efficiency under data scarcity, we propose Data Efficient GRPO, a reinforcement learning strategy that iteratively identifies high utility samples and maximizes the contribution of each instance via per instance utility estimation, substantially enhancing performance gains under limited supervision. We construct a large scale benchmark dataset FaVChat 170K, comprising approximately 60K high quality facial videos and 170K question answer pairs focusing on fine grained facial details. Extensive experiments, including zero shot evaluations on four facial understanding tasks, demonstrate that FaVChat consistently outperforms existing VLLMs. |
| title | FaVChat: Hierarchical Prompt-Query Guided Facial Video Understanding with Data-Efficient GRPO |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2503.09158 |