Generative Camera Dolly: Extreme Monocular Dynamic Novel View Synthesis
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866909242223493120 |
|---|---|
| author | Van Hoorick, Basile Wu, Rundi Ozguroglu, Ege Sargent, Kyle Liu, Ruoshi Tokmakov, Pavel Dave, Achal Zheng, Changxi Vondrick, Carl |
| author_facet | Van Hoorick, Basile Wu, Rundi Ozguroglu, Ege Sargent, Kyle Liu, Ruoshi Tokmakov, Pavel Dave, Achal Zheng, Changxi Vondrick, Carl |
| contents | Accurate reconstruction of complex dynamic scenes from just a single viewpoint continues to be a challenging task in computer vision. Current dynamic novel view synthesis methods typically require videos from many different camera viewpoints, necessitating careful recording setups, and significantly restricting their utility in the wild as well as in terms of embodied AI applications. In this paper, we propose $\textbf{GCD}$, a controllable monocular dynamic view synthesis pipeline that leverages large-scale diffusion priors to, given a video of any scene, generate a synchronous video from any other chosen perspective, conditioned on a set of relative camera pose parameters. Our model does not require depth as input, and does not explicitly model 3D scene geometry, instead performing end-to-end video-to-video translation in order to achieve its goal efficiently. Despite being trained on synthetic multi-view video data only, zero-shot real-world generalization experiments show promising results in multiple domains, including robotics, object permanence, and driving environments. We believe our framework can potentially unlock powerful applications in rich dynamic scene understanding, perception for robotics, and interactive 3D video viewing experiences for virtual reality. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_14868 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Generative Camera Dolly: Extreme Monocular Dynamic Novel View Synthesis Van Hoorick, Basile Wu, Rundi Ozguroglu, Ege Sargent, Kyle Liu, Ruoshi Tokmakov, Pavel Dave, Achal Zheng, Changxi Vondrick, Carl Computer Vision and Pattern Recognition Artificial Intelligence Machine Learning Robotics Accurate reconstruction of complex dynamic scenes from just a single viewpoint continues to be a challenging task in computer vision. Current dynamic novel view synthesis methods typically require videos from many different camera viewpoints, necessitating careful recording setups, and significantly restricting their utility in the wild as well as in terms of embodied AI applications. In this paper, we propose $\textbf{GCD}$, a controllable monocular dynamic view synthesis pipeline that leverages large-scale diffusion priors to, given a video of any scene, generate a synchronous video from any other chosen perspective, conditioned on a set of relative camera pose parameters. Our model does not require depth as input, and does not explicitly model 3D scene geometry, instead performing end-to-end video-to-video translation in order to achieve its goal efficiently. Despite being trained on synthetic multi-view video data only, zero-shot real-world generalization experiments show promising results in multiple domains, including robotics, object permanence, and driving environments. We believe our framework can potentially unlock powerful applications in rich dynamic scene understanding, perception for robotics, and interactive 3D video viewing experiences for virtual reality. |
| title | Generative Camera Dolly: Extreme Monocular Dynamic Novel View Synthesis |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Machine Learning Robotics |
| url | https://arxiv.org/abs/2405.14868 |