CFSynthesis: Controllable and Free-view 3D Human Video Synthesis
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915069305028608 |
|---|---|
| author | Cui, Liyuan Xu, Xiaogang Dong, Wenqi Yang, Zesong Bao, Hujun Cui, Zhaopeng |
| author_facet | Cui, Liyuan Xu, Xiaogang Dong, Wenqi Yang, Zesong Bao, Hujun Cui, Zhaopeng |
| contents | Human video synthesis aims to create lifelike characters in various environments, with wide applications in VR, storytelling, and content creation. While 2D diffusion-based methods have made significant progress, they struggle to generalize to complex 3D poses and varying scene backgrounds. To address these limitations, we introduce CFSynthesis, a novel framework for generating high-quality human videos with customizable attributes, including identity, motion, and scene configurations. Our method leverages a texture-SMPL-based representation to ensure consistent and stable character appearances across free viewpoints. Additionally, we introduce a novel foreground-background separation strategy that effectively decomposes the scene as foreground and background, enabling seamless integration of user-defined backgrounds. Experimental results on multiple datasets show that CFSynthesis not only achieves state-of-the-art performance in complex human animations but also adapts effectively to 3D motions in free-view and user-specified scenarios. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_11067 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | CFSynthesis: Controllable and Free-view 3D Human Video Synthesis Cui, Liyuan Xu, Xiaogang Dong, Wenqi Yang, Zesong Bao, Hujun Cui, Zhaopeng Computer Vision and Pattern Recognition Human video synthesis aims to create lifelike characters in various environments, with wide applications in VR, storytelling, and content creation. While 2D diffusion-based methods have made significant progress, they struggle to generalize to complex 3D poses and varying scene backgrounds. To address these limitations, we introduce CFSynthesis, a novel framework for generating high-quality human videos with customizable attributes, including identity, motion, and scene configurations. Our method leverages a texture-SMPL-based representation to ensure consistent and stable character appearances across free viewpoints. Additionally, we introduce a novel foreground-background separation strategy that effectively decomposes the scene as foreground and background, enabling seamless integration of user-defined backgrounds. Experimental results on multiple datasets show that CFSynthesis not only achieves state-of-the-art performance in complex human animations but also adapts effectively to 3D motions in free-view and user-specified scenarios. |
| title | CFSynthesis: Controllable and Free-view 3D Human Video Synthesis |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2412.11067 |