GeometryCrafter: Consistent Geometry Estimation for Open-world Videos with Diffusion Priors
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866912304037101568 |
|---|---|
| author | Xu, Tian-Xing Gao, Xiangjun Hu, Wenbo Li, Xiaoyu Zhang, Song-Hai Shan, Ying |
| author_facet | Xu, Tian-Xing Gao, Xiangjun Hu, Wenbo Li, Xiaoyu Zhang, Song-Hai Shan, Ying |
| contents | Despite remarkable advancements in video depth estimation, existing methods exhibit inherent limitations in achieving geometric fidelity through the affine-invariant predictions, limiting their applicability in reconstruction and other metrically grounded downstream tasks. We propose GeometryCrafter, a novel framework that recovers high-fidelity point map sequences with temporal coherence from open-world videos, enabling accurate 3D/4D reconstruction, camera parameter estimation, and other depth-based applications. At the core of our approach lies a point map Variational Autoencoder (VAE) that learns a latent space agnostic to video latent distributions for effective point map encoding and decoding. Leveraging the VAE, we train a video diffusion model to model the distribution of point map sequences conditioned on the input videos. Extensive evaluations on diverse datasets demonstrate that GeometryCrafter achieves state-of-the-art 3D accuracy, temporal consistency, and generalization capability. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_01016 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | GeometryCrafter: Consistent Geometry Estimation for Open-world Videos with Diffusion Priors Xu, Tian-Xing Gao, Xiangjun Hu, Wenbo Li, Xiaoyu Zhang, Song-Hai Shan, Ying Graphics Artificial Intelligence Computer Vision and Pattern Recognition Despite remarkable advancements in video depth estimation, existing methods exhibit inherent limitations in achieving geometric fidelity through the affine-invariant predictions, limiting their applicability in reconstruction and other metrically grounded downstream tasks. We propose GeometryCrafter, a novel framework that recovers high-fidelity point map sequences with temporal coherence from open-world videos, enabling accurate 3D/4D reconstruction, camera parameter estimation, and other depth-based applications. At the core of our approach lies a point map Variational Autoencoder (VAE) that learns a latent space agnostic to video latent distributions for effective point map encoding and decoding. Leveraging the VAE, we train a video diffusion model to model the distribution of point map sequences conditioned on the input videos. Extensive evaluations on diverse datasets demonstrate that GeometryCrafter achieves state-of-the-art 3D accuracy, temporal consistency, and generalization capability. |
| title | GeometryCrafter: Consistent Geometry Estimation for Open-world Videos with Diffusion Priors |
| topic | Graphics Artificial Intelligence Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2504.01016 |