SimVS: Simulating World Inconsistencies for Robust View Synthesis
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866909423591489536 |
|---|---|
| author | Trevithick, Alex Paiss, Roni Henzler, Philipp Verbin, Dor Wu, Rundi Alzayer, Hadi Gao, Ruiqi Poole, Ben Barron, Jonathan T. Holynski, Aleksander Ramamoorthi, Ravi Srinivasan, Pratul P. |
| author_facet | Trevithick, Alex Paiss, Roni Henzler, Philipp Verbin, Dor Wu, Rundi Alzayer, Hadi Gao, Ruiqi Poole, Ben Barron, Jonathan T. Holynski, Aleksander Ramamoorthi, Ravi Srinivasan, Pratul P. |
| contents | Novel-view synthesis techniques achieve impressive results for static scenes but struggle when faced with the inconsistencies inherent to casual capture settings: varying illumination, scene motion, and other unintended effects that are difficult to model explicitly. We present an approach for leveraging generative video models to simulate the inconsistencies in the world that can occur during capture. We use this process, along with existing multi-view datasets, to create synthetic data for training a multi-view harmonization network that is able to reconcile inconsistent observations into a consistent 3D scene. We demonstrate that our world-simulation strategy significantly outperforms traditional augmentation methods in handling real-world scene variations, thereby enabling highly accurate static 3D reconstructions in the presence of a variety of challenging inconsistencies. Project page: https://alextrevithick.github.io/simvs |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_07696 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | SimVS: Simulating World Inconsistencies for Robust View Synthesis Trevithick, Alex Paiss, Roni Henzler, Philipp Verbin, Dor Wu, Rundi Alzayer, Hadi Gao, Ruiqi Poole, Ben Barron, Jonathan T. Holynski, Aleksander Ramamoorthi, Ravi Srinivasan, Pratul P. Computer Vision and Pattern Recognition Artificial Intelligence Graphics Machine Learning Novel-view synthesis techniques achieve impressive results for static scenes but struggle when faced with the inconsistencies inherent to casual capture settings: varying illumination, scene motion, and other unintended effects that are difficult to model explicitly. We present an approach for leveraging generative video models to simulate the inconsistencies in the world that can occur during capture. We use this process, along with existing multi-view datasets, to create synthetic data for training a multi-view harmonization network that is able to reconcile inconsistent observations into a consistent 3D scene. We demonstrate that our world-simulation strategy significantly outperforms traditional augmentation methods in handling real-world scene variations, thereby enabling highly accurate static 3D reconstructions in the presence of a variety of challenging inconsistencies. Project page: https://alextrevithick.github.io/simvs |
| title | SimVS: Simulating World Inconsistencies for Robust View Synthesis |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Graphics Machine Learning |
| url | https://arxiv.org/abs/2412.07696 |