Collaborative Face Experts Fusion in Video Generation: Boosting Identity Consistency Across Large Face Poses
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914181219876864 |
|---|---|
| author | Wang, Yuji Li, Moran Hu, Xiaobin Yi, Ran Zhang, Jiangning Xu, Chengming Cao, Weijian Wang, Yabiao Wang, Chengjie Ma, Lizhuang |
| author_facet | Wang, Yuji Li, Moran Hu, Xiaobin Yi, Ran Zhang, Jiangning Xu, Chengming Cao, Weijian Wang, Yabiao Wang, Chengjie Ma, Lizhuang |
| contents | Current video generation models struggle with identity preservation under large face poses, primarily facing two challenges: the difficulty in exploring an effective mechanism to integrate identity features into DiT architectures, and the lack of targeted coverage of large face poses in existing open-source video datasets. To address these, we present two key innovations. First, we propose Collaborative Face Experts Fusion (CoFE), which dynamically fuses complementary signals from three specialized experts within the DiT backbone: an identity expert that captures cross-pose invariant features, a semantic expert that encodes high-level visual context, and a detail expert that preserves pixel-level attributes such as skin texture and color gradients. Second, we introduce a data curation pipeline comprising three key components: Face Constraints to ensure diverse large-pose coverage, Identity Consistency to maintain stable identity across frames, and Speech Disambiguation to align textual captions with actual speaking behavior. This pipeline yields LaFID-180K, a large-scale dataset of pose-annotated video clips designed for identity-preserving video generation. Experimental results on several benchmarks demonstrate that our approach significantly outperforms state-of-the-art methods in face similarity, FID, and CLIP semantic alignment. Project page: https://rain152.github.io/CoFE/. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_09476 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Collaborative Face Experts Fusion in Video Generation: Boosting Identity Consistency Across Large Face Poses Wang, Yuji Li, Moran Hu, Xiaobin Yi, Ran Zhang, Jiangning Xu, Chengming Cao, Weijian Wang, Yabiao Wang, Chengjie Ma, Lizhuang Computer Vision and Pattern Recognition Current video generation models struggle with identity preservation under large face poses, primarily facing two challenges: the difficulty in exploring an effective mechanism to integrate identity features into DiT architectures, and the lack of targeted coverage of large face poses in existing open-source video datasets. To address these, we present two key innovations. First, we propose Collaborative Face Experts Fusion (CoFE), which dynamically fuses complementary signals from three specialized experts within the DiT backbone: an identity expert that captures cross-pose invariant features, a semantic expert that encodes high-level visual context, and a detail expert that preserves pixel-level attributes such as skin texture and color gradients. Second, we introduce a data curation pipeline comprising three key components: Face Constraints to ensure diverse large-pose coverage, Identity Consistency to maintain stable identity across frames, and Speech Disambiguation to align textual captions with actual speaking behavior. This pipeline yields LaFID-180K, a large-scale dataset of pose-annotated video clips designed for identity-preserving video generation. Experimental results on several benchmarks demonstrate that our approach significantly outperforms state-of-the-art methods in face similarity, FID, and CLIP semantic alignment. Project page: https://rain152.github.io/CoFE/. |
| title | Collaborative Face Experts Fusion in Video Generation: Boosting Identity Consistency Across Large Face Poses |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2508.09476 |