3DGUT: Enabling Distorted Cameras and Secondary Rays in Gaussian Splatting
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909550859255808 |
|---|---|
| author | Wu, Qi Esturo, Janick Martinez Mirzaei, Ashkan Moenne-Loccoz, Nicolas Gojcic, Zan |
| author_facet | Wu, Qi Esturo, Janick Martinez Mirzaei, Ashkan Moenne-Loccoz, Nicolas Gojcic, Zan |
| contents | 3D Gaussian Splatting (3DGS) enables efficient reconstruction and high-fidelity real-time rendering of complex scenes on consumer hardware. However, due to its rasterization-based formulation, 3DGS is constrained to ideal pinhole cameras and lacks support for secondary lighting effects. Recent methods address these limitations by tracing the particles instead, but, this comes at the cost of significantly slower rendering. In this work, we propose 3D Gaussian Unscented Transform (3DGUT), replacing the EWA splatting formulation with the Unscented Transform that approximates the particles through sigma points, which can be projected exactly under any nonlinear projection function. This modification enables trivial support of distorted cameras with time dependent effects such as rolling shutter, while retaining the efficiency of rasterization. Additionally, we align our rendering formulation with that of tracing-based methods, enabling secondary ray tracing required to represent phenomena such as reflections and refraction within the same 3D representation. The source code is available at: https://github.com/nv-tlabs/3dgrut. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_12507 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | 3DGUT: Enabling Distorted Cameras and Secondary Rays in Gaussian Splatting Wu, Qi Esturo, Janick Martinez Mirzaei, Ashkan Moenne-Loccoz, Nicolas Gojcic, Zan Graphics Computer Vision and Pattern Recognition 3D Gaussian Splatting (3DGS) enables efficient reconstruction and high-fidelity real-time rendering of complex scenes on consumer hardware. However, due to its rasterization-based formulation, 3DGS is constrained to ideal pinhole cameras and lacks support for secondary lighting effects. Recent methods address these limitations by tracing the particles instead, but, this comes at the cost of significantly slower rendering. In this work, we propose 3D Gaussian Unscented Transform (3DGUT), replacing the EWA splatting formulation with the Unscented Transform that approximates the particles through sigma points, which can be projected exactly under any nonlinear projection function. This modification enables trivial support of distorted cameras with time dependent effects such as rolling shutter, while retaining the efficiency of rasterization. Additionally, we align our rendering formulation with that of tracing-based methods, enabling secondary ray tracing required to represent phenomena such as reflections and refraction within the same 3D representation. The source code is available at: https://github.com/nv-tlabs/3dgrut. |
| title | 3DGUT: Enabling Distorted Cameras and Secondary Rays in Gaussian Splatting |
| topic | Graphics Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2412.12507 |