KFC-W: Generating 3D-Consistent Videos from Unposed Internet Photos
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911653339070464 |
|---|---|
| author | Chou, Gene Zhang, Kai Bi, Sai Tan, Hao Xu, Zexiang Luan, Fujun Hariharan, Bharath Snavely, Noah |
| author_facet | Chou, Gene Zhang, Kai Bi, Sai Tan, Hao Xu, Zexiang Luan, Fujun Hariharan, Bharath Snavely, Noah |
| contents | We address the problem of generating videos from unposed internet photos. A handful of input images serve as keyframes, and our model interpolates between them to simulate a path moving between the cameras. Given random images, a model's ability to capture underlying geometry, recognize scene identity, and relate frames in terms of camera position and orientation reflects a fundamental understanding of 3D structure and scene layout. However, existing video models such as Luma Dream Machine fail at this task. We design a self-supervised method that takes advantage of the consistency of videos and variability of multiview internet photos to train a scalable, 3D-aware video model without any 3D annotations such as camera parameters. We validate that our method outperforms all baselines in terms of geometric and appearance consistency. We also show our model benefits applications that enable camera control, such as 3D Gaussian Splatting. Our results suggest that we can scale up scene-level 3D learning using only 2D data such as videos and multiview internet photos. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_13549 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | KFC-W: Generating 3D-Consistent Videos from Unposed Internet Photos Chou, Gene Zhang, Kai Bi, Sai Tan, Hao Xu, Zexiang Luan, Fujun Hariharan, Bharath Snavely, Noah Computer Vision and Pattern Recognition We address the problem of generating videos from unposed internet photos. A handful of input images serve as keyframes, and our model interpolates between them to simulate a path moving between the cameras. Given random images, a model's ability to capture underlying geometry, recognize scene identity, and relate frames in terms of camera position and orientation reflects a fundamental understanding of 3D structure and scene layout. However, existing video models such as Luma Dream Machine fail at this task. We design a self-supervised method that takes advantage of the consistency of videos and variability of multiview internet photos to train a scalable, 3D-aware video model without any 3D annotations such as camera parameters. We validate that our method outperforms all baselines in terms of geometric and appearance consistency. We also show our model benefits applications that enable camera control, such as 3D Gaussian Splatting. Our results suggest that we can scale up scene-level 3D learning using only 2D data such as videos and multiview internet photos. |
| title | KFC-W: Generating 3D-Consistent Videos from Unposed Internet Photos |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2411.13549 |