DPA-Net: Structured 3D Abstraction from Sparse Views via Differentiable Primitive Assembly
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866916349001859072 |
|---|---|
| author | Yu, Fenggen Qian, Yiming Zhang, Xu Gil-Ureta, Francisca Jackson, Brian Bennett, Eric Zhang, Hao |
| author_facet | Yu, Fenggen Qian, Yiming Zhang, Xu Gil-Ureta, Francisca Jackson, Brian Bennett, Eric Zhang, Hao |
| contents | We present a differentiable rendering framework to learn structured 3D abstractions in the form of primitive assemblies from sparse RGB images capturing a 3D object. By leveraging differentiable volume rendering, our method does not require 3D supervision. Architecturally, our network follows the general pipeline of an image-conditioned neural radiance field (NeRF) exemplified by pixelNeRF for color prediction. As our core contribution, we introduce differential primitive assembly (DPA) into NeRF to output a 3D occupancy field in place of density prediction, where the predicted occupancies serve as opacity values for volume rendering. Our network, coined DPA-Net, produces a union of convexes, each as an intersection of convex quadric primitives, to approximate the target 3D object, subject to an abstraction loss and a masking loss, both defined in the image space upon volume rendering. With test-time adaptation and additional sampling and loss designs aimed at improving the accuracy and compactness of the obtained assemblies, our method demonstrates superior performance over state-of-the-art alternatives for 3D primitive abstraction from sparse views. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_00875 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | DPA-Net: Structured 3D Abstraction from Sparse Views via Differentiable Primitive Assembly Yu, Fenggen Qian, Yiming Zhang, Xu Gil-Ureta, Francisca Jackson, Brian Bennett, Eric Zhang, Hao Computer Vision and Pattern Recognition We present a differentiable rendering framework to learn structured 3D abstractions in the form of primitive assemblies from sparse RGB images capturing a 3D object. By leveraging differentiable volume rendering, our method does not require 3D supervision. Architecturally, our network follows the general pipeline of an image-conditioned neural radiance field (NeRF) exemplified by pixelNeRF for color prediction. As our core contribution, we introduce differential primitive assembly (DPA) into NeRF to output a 3D occupancy field in place of density prediction, where the predicted occupancies serve as opacity values for volume rendering. Our network, coined DPA-Net, produces a union of convexes, each as an intersection of convex quadric primitives, to approximate the target 3D object, subject to an abstraction loss and a masking loss, both defined in the image space upon volume rendering. With test-time adaptation and additional sampling and loss designs aimed at improving the accuracy and compactness of the obtained assemblies, our method demonstrates superior performance over state-of-the-art alternatives for 3D primitive abstraction from sparse views. |
| title | DPA-Net: Structured 3D Abstraction from Sparse Views via Differentiable Primitive Assembly |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2404.00875 |