Taming Latent Diffusion Model for Neural Radiance Field Inpainting
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917835153866752 |
|---|---|
| author | Lin, Chieh Hubert Kim, Changil Huang, Jia-Bin Li, Qinbo Ma, Chih-Yao Kopf, Johannes Yang, Ming-Hsuan Tseng, Hung-Yu |
| author_facet | Lin, Chieh Hubert Kim, Changil Huang, Jia-Bin Li, Qinbo Ma, Chih-Yao Kopf, Johannes Yang, Ming-Hsuan Tseng, Hung-Yu |
| contents | Neural Radiance Field (NeRF) is a representation for 3D reconstruction from multi-view images. Despite some recent work showing preliminary success in editing a reconstructed NeRF with diffusion prior, they remain struggling to synthesize reasonable geometry in completely uncovered regions. One major reason is the high diversity of synthetic contents from the diffusion model, which hinders the radiance field from converging to a crisp and deterministic geometry. Moreover, applying latent diffusion models on real data often yields a textural shift incoherent to the image condition due to auto-encoding errors. These two problems are further reinforced with the use of pixel-distance losses. To address these issues, we propose tempering the diffusion model's stochasticity with per-scene customization and mitigating the textural shift with masked adversarial training. During the analyses, we also found the commonly used pixel and perceptual losses are harmful in the NeRF inpainting task. Through rigorous experiments, our framework yields state-of-the-art NeRF inpainting results on various real-world scenes. Project page: https://hubert0527.github.io/MALD-NeRF |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_09995 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Taming Latent Diffusion Model for Neural Radiance Field Inpainting Lin, Chieh Hubert Kim, Changil Huang, Jia-Bin Li, Qinbo Ma, Chih-Yao Kopf, Johannes Yang, Ming-Hsuan Tseng, Hung-Yu Computer Vision and Pattern Recognition Artificial Intelligence Machine Learning Neural Radiance Field (NeRF) is a representation for 3D reconstruction from multi-view images. Despite some recent work showing preliminary success in editing a reconstructed NeRF with diffusion prior, they remain struggling to synthesize reasonable geometry in completely uncovered regions. One major reason is the high diversity of synthetic contents from the diffusion model, which hinders the radiance field from converging to a crisp and deterministic geometry. Moreover, applying latent diffusion models on real data often yields a textural shift incoherent to the image condition due to auto-encoding errors. These two problems are further reinforced with the use of pixel-distance losses. To address these issues, we propose tempering the diffusion model's stochasticity with per-scene customization and mitigating the textural shift with masked adversarial training. During the analyses, we also found the commonly used pixel and perceptual losses are harmful in the NeRF inpainting task. Through rigorous experiments, our framework yields state-of-the-art NeRF inpainting results on various real-world scenes. Project page: https://hubert0527.github.io/MALD-NeRF |
| title | Taming Latent Diffusion Model for Neural Radiance Field Inpainting |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2404.09995 |