VividDream: Generating 3D Scene with Ambient Dynamics
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929366480453632 |
|---|---|
| author | Lee, Yao-Chih Chen, Yi-Ting Wang, Andrew Liao, Ting-Hsuan Feng, Brandon Y. Huang, Jia-Bin |
| author_facet | Lee, Yao-Chih Chen, Yi-Ting Wang, Andrew Liao, Ting-Hsuan Feng, Brandon Y. Huang, Jia-Bin |
| contents | We introduce VividDream, a method for generating explorable 4D scenes with ambient dynamics from a single input image or text prompt. VividDream first expands an input image into a static 3D point cloud through iterative inpainting and geometry merging. An ensemble of animated videos is then generated using video diffusion models with quality refinement techniques and conditioned on renderings of the static 3D scene from the sampled camera trajectories. We then optimize a canonical 4D scene representation using an animated video ensemble, with per-video motion embeddings and visibility masks to mitigate inconsistencies. The resulting 4D scene enables free-view exploration of a 3D scene with plausible ambient scene dynamics. Experiments demonstrate that VividDream can provide human viewers with compelling 4D experiences generated based on diverse real images and text prompts. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_20334 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | VividDream: Generating 3D Scene with Ambient Dynamics Lee, Yao-Chih Chen, Yi-Ting Wang, Andrew Liao, Ting-Hsuan Feng, Brandon Y. Huang, Jia-Bin Computer Vision and Pattern Recognition Graphics We introduce VividDream, a method for generating explorable 4D scenes with ambient dynamics from a single input image or text prompt. VividDream first expands an input image into a static 3D point cloud through iterative inpainting and geometry merging. An ensemble of animated videos is then generated using video diffusion models with quality refinement techniques and conditioned on renderings of the static 3D scene from the sampled camera trajectories. We then optimize a canonical 4D scene representation using an animated video ensemble, with per-video motion embeddings and visibility masks to mitigate inconsistencies. The resulting 4D scene enables free-view exploration of a 3D scene with plausible ambient scene dynamics. Experiments demonstrate that VividDream can provide human viewers with compelling 4D experiences generated based on diverse real images and text prompts. |
| title | VividDream: Generating 3D Scene with Ambient Dynamics |
| topic | Computer Vision and Pattern Recognition Graphics |
| url | https://arxiv.org/abs/2405.20334 |