Template-Free Single-View 3D Human Digitalization with Diffusion-Guided LRM
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866911797255077888 |
|---|---|
| author | Weng, Zhenzhen Liu, Jingyuan Tan, Hao Xu, Zhan Zhou, Yang Yeung-Levy, Serena Yang, Jimei |
| author_facet | Weng, Zhenzhen Liu, Jingyuan Tan, Hao Xu, Zhan Zhou, Yang Yeung-Levy, Serena Yang, Jimei |
| contents | Reconstructing 3D humans from a single image has been extensively investigated. However, existing approaches often fall short on capturing fine geometry and appearance details, hallucinating occluded parts with plausible details, and achieving generalization across unseen and in-the-wild datasets. We present Human-LRM, a diffusion-guided feed-forward model that predicts the implicit field of a human from a single image. Leveraging the power of the state-of-the-art reconstruction model (i.e., LRM) and generative model (i.e Stable Diffusion), our method is able to capture human without any template prior, e.g., SMPL, and effectively enhance occluded parts with rich and realistic details. Our approach first uses a single-view LRM model with an enhanced geometry decoder to get the triplane NeRF representation. The novel view renderings from the triplane NeRF provide strong geometry and color prior, from which we generate photo-realistic details for the occluded parts using a diffusion model. The generated multiple views then enable reconstruction with high-quality geometry and appearance, leading to superior overall performance comparing to all existing human reconstruction methods. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_12175 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Template-Free Single-View 3D Human Digitalization with Diffusion-Guided LRM Weng, Zhenzhen Liu, Jingyuan Tan, Hao Xu, Zhan Zhou, Yang Yeung-Levy, Serena Yang, Jimei Computer Vision and Pattern Recognition Reconstructing 3D humans from a single image has been extensively investigated. However, existing approaches often fall short on capturing fine geometry and appearance details, hallucinating occluded parts with plausible details, and achieving generalization across unseen and in-the-wild datasets. We present Human-LRM, a diffusion-guided feed-forward model that predicts the implicit field of a human from a single image. Leveraging the power of the state-of-the-art reconstruction model (i.e., LRM) and generative model (i.e Stable Diffusion), our method is able to capture human without any template prior, e.g., SMPL, and effectively enhance occluded parts with rich and realistic details. Our approach first uses a single-view LRM model with an enhanced geometry decoder to get the triplane NeRF representation. The novel view renderings from the triplane NeRF provide strong geometry and color prior, from which we generate photo-realistic details for the occluded parts using a diffusion model. The generated multiple views then enable reconstruction with high-quality geometry and appearance, leading to superior overall performance comparing to all existing human reconstruction methods. |
| title | Template-Free Single-View 3D Human Digitalization with Diffusion-Guided LRM |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2401.12175 |