Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2412.07760 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916517141020672 |
|---|---|
| author | Bai, Jianhong Xia, Menghan Wang, Xintao Yuan, Ziyang Fu, Xiao Liu, Zuozhu Hu, Haoji Wan, Pengfei Zhang, Di |
| author_facet | Bai, Jianhong Xia, Menghan Wang, Xintao Yuan, Ziyang Fu, Xiao Liu, Zuozhu Hu, Haoji Wan, Pengfei Zhang, Di |
| contents | Recent advancements in video diffusion models have shown exceptional abilities in simulating real-world dynamics and maintaining 3D consistency. This progress inspires us to investigate the potential of these models to ensure dynamic consistency across various viewpoints, a highly desirable feature for applications such as virtual filming. Unlike existing methods focused on multi-view generation of single objects for 4D reconstruction, our interest lies in generating open-world videos from arbitrary viewpoints, incorporating 6 DoF camera poses. To achieve this, we propose a plug-and-play module that enhances a pre-trained text-to-video model for multi-camera video generation, ensuring consistent content across different viewpoints. Specifically, we introduce a multi-view synchronization module to maintain appearance and geometry consistency across these viewpoints. Given the scarcity of high-quality training data, we design a hybrid training scheme that leverages multi-camera images and monocular videos to supplement Unreal Engine-rendered multi-camera videos. Furthermore, our method enables intriguing extensions, such as re-rendering a video from novel viewpoints. We also release a multi-view synchronized video dataset, named SynCamVideo-Dataset. Project page: https://jianhongbai.github.io/SynCamMaster/. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_07760 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | SynCamMaster: Synchronizing Multi-Camera Video Generation from Diverse Viewpoints Bai, Jianhong Xia, Menghan Wang, Xintao Yuan, Ziyang Fu, Xiao Liu, Zuozhu Hu, Haoji Wan, Pengfei Zhang, Di Computer Vision and Pattern Recognition Recent advancements in video diffusion models have shown exceptional abilities in simulating real-world dynamics and maintaining 3D consistency. This progress inspires us to investigate the potential of these models to ensure dynamic consistency across various viewpoints, a highly desirable feature for applications such as virtual filming. Unlike existing methods focused on multi-view generation of single objects for 4D reconstruction, our interest lies in generating open-world videos from arbitrary viewpoints, incorporating 6 DoF camera poses. To achieve this, we propose a plug-and-play module that enhances a pre-trained text-to-video model for multi-camera video generation, ensuring consistent content across different viewpoints. Specifically, we introduce a multi-view synchronization module to maintain appearance and geometry consistency across these viewpoints. Given the scarcity of high-quality training data, we design a hybrid training scheme that leverages multi-camera images and monocular videos to supplement Unreal Engine-rendered multi-camera videos. Furthermore, our method enables intriguing extensions, such as re-rendering a video from novel viewpoints. We also release a multi-view synchronized video dataset, named SynCamVideo-Dataset. Project page: https://jianhongbai.github.io/SynCamMaster/. |
| title | SynCamMaster: Synchronizing Multi-Camera Video Generation from Diverse Viewpoints |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2412.07760 |