3D Congealing: 3D-Aware Image Alignment in the Wild
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866914738579963904 |
|---|---|
| author | Zhang, Yunzhi Li, Zizhang Raj, Amit Engelhardt, Andreas Li, Yuanzhen Hou, Tingbo Wu, Jiajun Jampani, Varun |
| author_facet | Zhang, Yunzhi Li, Zizhang Raj, Amit Engelhardt, Andreas Li, Yuanzhen Hou, Tingbo Wu, Jiajun Jampani, Varun |
| contents | We propose 3D Congealing, a novel problem of 3D-aware alignment for 2D images capturing semantically similar objects. Given a collection of unlabeled Internet images, our goal is to associate the shared semantic parts from the inputs and aggregate the knowledge from 2D images to a shared 3D canonical space. We introduce a general framework that tackles the task without assuming shape templates, poses, or any camera parameters. At its core is a canonical 3D representation that encapsulates geometric and semantic information. The framework optimizes for the canonical representation together with the pose for each input image, and a per-image coordinate map that warps 2D pixel coordinates to the 3D canonical frame to account for the shape matching. The optimization procedure fuses prior knowledge from a pre-trained image generative model and semantic information from input images. The former provides strong knowledge guidance for this under-constraint task, while the latter provides the necessary information to mitigate the training data bias from the pre-trained model. Our framework can be used for various tasks such as correspondence matching, pose estimation, and image editing, achieving strong results on real-world image datasets under challenging illumination conditions and on in-the-wild online image collections. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_02125 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | 3D Congealing: 3D-Aware Image Alignment in the Wild Zhang, Yunzhi Li, Zizhang Raj, Amit Engelhardt, Andreas Li, Yuanzhen Hou, Tingbo Wu, Jiajun Jampani, Varun Computer Vision and Pattern Recognition We propose 3D Congealing, a novel problem of 3D-aware alignment for 2D images capturing semantically similar objects. Given a collection of unlabeled Internet images, our goal is to associate the shared semantic parts from the inputs and aggregate the knowledge from 2D images to a shared 3D canonical space. We introduce a general framework that tackles the task without assuming shape templates, poses, or any camera parameters. At its core is a canonical 3D representation that encapsulates geometric and semantic information. The framework optimizes for the canonical representation together with the pose for each input image, and a per-image coordinate map that warps 2D pixel coordinates to the 3D canonical frame to account for the shape matching. The optimization procedure fuses prior knowledge from a pre-trained image generative model and semantic information from input images. The former provides strong knowledge guidance for this under-constraint task, while the latter provides the necessary information to mitigate the training data bias from the pre-trained model. Our framework can be used for various tasks such as correspondence matching, pose estimation, and image editing, achieving strong results on real-world image datasets under challenging illumination conditions and on in-the-wild online image collections. |
| title | 3D Congealing: 3D-Aware Image Alignment in the Wild |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2404.02125 |