SGD: Street View Synthesis with Gaussian Splatting and Diffusion Prior
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866913290371727360 |
|---|---|
| author | Yu, Zhongrui Wang, Haoran Yang, Jinze Wang, Hanzhang Xie, Zeke Cai, Yunfeng Cao, Jiale Ji, Zhong Sun, Mingming |
| author_facet | Yu, Zhongrui Wang, Haoran Yang, Jinze Wang, Hanzhang Xie, Zeke Cai, Yunfeng Cao, Jiale Ji, Zhong Sun, Mingming |
| contents | Novel View Synthesis (NVS) for street scenes play a critical role in the autonomous driving simulation. The current mainstream technique to achieve it is neural rendering, such as Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS). Although thrilling progress has been made, when handling street scenes, current methods struggle to maintain rendering quality at the viewpoint that deviates significantly from the training viewpoints. This issue stems from the sparse training views captured by a fixed camera on a moving vehicle. To tackle this problem, we propose a novel approach that enhances the capacity of 3DGS by leveraging prior from a Diffusion Model along with complementary multi-modal data. Specifically, we first fine-tune a Diffusion Model by adding images from adjacent frames as condition, meanwhile exploiting depth data from LiDAR point clouds to supply additional spatial information. Then we apply the Diffusion Model to regularize the 3DGS at unseen views during training. Experimental results validate the effectiveness of our method compared with current state-of-the-art models, and demonstrate its advance in rendering images from broader views. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_20079 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | SGD: Street View Synthesis with Gaussian Splatting and Diffusion Prior Yu, Zhongrui Wang, Haoran Yang, Jinze Wang, Hanzhang Xie, Zeke Cai, Yunfeng Cao, Jiale Ji, Zhong Sun, Mingming Computer Vision and Pattern Recognition Novel View Synthesis (NVS) for street scenes play a critical role in the autonomous driving simulation. The current mainstream technique to achieve it is neural rendering, such as Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS). Although thrilling progress has been made, when handling street scenes, current methods struggle to maintain rendering quality at the viewpoint that deviates significantly from the training viewpoints. This issue stems from the sparse training views captured by a fixed camera on a moving vehicle. To tackle this problem, we propose a novel approach that enhances the capacity of 3DGS by leveraging prior from a Diffusion Model along with complementary multi-modal data. Specifically, we first fine-tune a Diffusion Model by adding images from adjacent frames as condition, meanwhile exploiting depth data from LiDAR point clouds to supply additional spatial information. Then we apply the Diffusion Model to regularize the 3DGS at unseen views during training. Experimental results validate the effectiveness of our method compared with current state-of-the-art models, and demonstrate its advance in rendering images from broader views. |
| title | SGD: Street View Synthesis with Gaussian Splatting and Diffusion Prior |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2403.20079 |