InFusion: Inpainting 3D Gaussians via Learning Depth Completion from Diffusion Prior
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866917642681450496 |
|---|---|
| author | Liu, Zhiheng Ouyang, Hao Wang, Qiuyu Cheng, Ka Leong Xiao, Jie Zhu, Kai Xue, Nan Liu, Yu Shen, Yujun Cao, Yang |
| author_facet | Liu, Zhiheng Ouyang, Hao Wang, Qiuyu Cheng, Ka Leong Xiao, Jie Zhu, Kai Xue, Nan Liu, Yu Shen, Yujun Cao, Yang |
| contents | 3D Gaussians have recently emerged as an efficient representation for novel view synthesis. This work studies its editability with a particular focus on the inpainting task, which aims to supplement an incomplete set of 3D Gaussians with additional points for visually harmonious rendering. Compared to 2D inpainting, the crux of inpainting 3D Gaussians is to figure out the rendering-relevant properties of the introduced points, whose optimization largely benefits from their initial 3D positions. To this end, we propose to guide the point initialization with an image-conditioned depth completion model, which learns to directly restore the depth map based on the observed image. Such a design allows our model to fill in depth values at an aligned scale with the original depth, and also to harness strong generalizability from largescale diffusion prior. Thanks to the more accurate depth completion, our approach, dubbed InFusion, surpasses existing alternatives with sufficiently better fidelity and efficiency under various complex scenarios. We further demonstrate the effectiveness of InFusion with several practical applications, such as inpainting with user-specific texture or with novel object insertion. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_11613 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | InFusion: Inpainting 3D Gaussians via Learning Depth Completion from Diffusion Prior Liu, Zhiheng Ouyang, Hao Wang, Qiuyu Cheng, Ka Leong Xiao, Jie Zhu, Kai Xue, Nan Liu, Yu Shen, Yujun Cao, Yang Computer Vision and Pattern Recognition 3D Gaussians have recently emerged as an efficient representation for novel view synthesis. This work studies its editability with a particular focus on the inpainting task, which aims to supplement an incomplete set of 3D Gaussians with additional points for visually harmonious rendering. Compared to 2D inpainting, the crux of inpainting 3D Gaussians is to figure out the rendering-relevant properties of the introduced points, whose optimization largely benefits from their initial 3D positions. To this end, we propose to guide the point initialization with an image-conditioned depth completion model, which learns to directly restore the depth map based on the observed image. Such a design allows our model to fill in depth values at an aligned scale with the original depth, and also to harness strong generalizability from largescale diffusion prior. Thanks to the more accurate depth completion, our approach, dubbed InFusion, surpasses existing alternatives with sufficiently better fidelity and efficiency under various complex scenarios. We further demonstrate the effectiveness of InFusion with several practical applications, such as inpainting with user-specific texture or with novel object insertion. |
| title | InFusion: Inpainting 3D Gaussians via Learning Depth Completion from Diffusion Prior |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2404.11613 |