Salvato in:
| Autori principali: | , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | https://arxiv.org/abs/2510.14179 |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866914096179314688 |
|---|---|
| author | Xu, Yuancheng Xian, Wenqi Ma, Li Philip, Julien Taşel, Ahmet Levent Zhao, Yiwei Burgert, Ryan He, Mingming Hermann, Oliver Pilarski, Oliver Garg, Rahul Debevec, Paul Yu, Ning |
| author_facet | Xu, Yuancheng Xian, Wenqi Ma, Li Philip, Julien Taşel, Ahmet Levent Zhao, Yiwei Burgert, Ryan He, Mingming Hermann, Oliver Pilarski, Oliver Garg, Rahul Debevec, Paul Yu, Ning |
| contents | We introduce a framework that enables both multi-view character consistency and 3D camera control in video diffusion models through a novel customization data pipeline. We train the character consistency component with recorded volumetric capture performances re-rendered with diverse camera trajectories via 4D Gaussian Splatting (4DGS), lighting variability obtained with a video relighting model. We fine-tune state-of-the-art open-source video diffusion models on this data to provide strong multi-view identity preservation, precise camera control, and lighting adaptability. Our framework also supports core capabilities for virtual production, including multi-subject generation using two approaches: joint training and noise blending, the latter enabling efficient composition of independently customized models at inference time; it also achieves scene and real-life video customization as well as control over motion and spatial layout during customization. Extensive experiments show improved video quality, higher personalization accuracy, and enhanced camera control and lighting adaptability, advancing the integration of video generation into virtual production. Our project page is available at: https://eyeline-labs.github.io/Virtually-Being. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_14179 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Virtually Being: Customizing Camera-Controllable Video Diffusion Models with Multi-View Performance Captures Xu, Yuancheng Xian, Wenqi Ma, Li Philip, Julien Taşel, Ahmet Levent Zhao, Yiwei Burgert, Ryan He, Mingming Hermann, Oliver Pilarski, Oliver Garg, Rahul Debevec, Paul Yu, Ning Computer Vision and Pattern Recognition Artificial Intelligence We introduce a framework that enables both multi-view character consistency and 3D camera control in video diffusion models through a novel customization data pipeline. We train the character consistency component with recorded volumetric capture performances re-rendered with diverse camera trajectories via 4D Gaussian Splatting (4DGS), lighting variability obtained with a video relighting model. We fine-tune state-of-the-art open-source video diffusion models on this data to provide strong multi-view identity preservation, precise camera control, and lighting adaptability. Our framework also supports core capabilities for virtual production, including multi-subject generation using two approaches: joint training and noise blending, the latter enabling efficient composition of independently customized models at inference time; it also achieves scene and real-life video customization as well as control over motion and spatial layout during customization. Extensive experiments show improved video quality, higher personalization accuracy, and enhanced camera control and lighting adaptability, advancing the integration of video generation into virtual production. Our project page is available at: https://eyeline-labs.github.io/Virtually-Being. |
| title | Virtually Being: Customizing Camera-Controllable Video Diffusion Models with Multi-View Performance Captures |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2510.14179 |