MuDG: Taming Multi-modal Diffusion with Gaussian Splatting for Urban Scene Reconstruction
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916651926028288 |
|---|---|
| author | Zou, Yingshuang Ding, Yikang Zhang, Chuanrui Guo, Jiazhe Li, Bohan Lyu, Xiaoyang Tan, Feiyang Qi, Xiaojuan Wang, Haoqian |
| author_facet | Zou, Yingshuang Ding, Yikang Zhang, Chuanrui Guo, Jiazhe Li, Bohan Lyu, Xiaoyang Tan, Feiyang Qi, Xiaojuan Wang, Haoqian |
| contents | Recent breakthroughs in radiance fields have significantly advanced 3D scene reconstruction and novel view synthesis (NVS) in autonomous driving. Nevertheless, critical limitations persist: reconstruction-based methods exhibit substantial performance deterioration under significant viewpoint deviations from training trajectories, while generation-based techniques struggle with temporal coherence and precise scene controllability. To overcome these challenges, we present MuDG, an innovative framework that integrates Multi-modal Diffusion model with Gaussian Splatting (GS) for Urban Scene Reconstruction. MuDG leverages aggregated LiDAR point clouds with RGB and geometric priors to condition a multi-modal video diffusion model, synthesizing photorealistic RGB, depth, and semantic outputs for novel viewpoints. This synthesis pipeline enables feed-forward NVS without computationally intensive per-scene optimization, providing comprehensive supervision signals to refine 3DGS representations for rendering robustness enhancement under extreme viewpoint changes. Experiments on the Open Waymo Dataset demonstrate that MuDG outperforms existing methods in both reconstruction and synthesis quality. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_10604 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | MuDG: Taming Multi-modal Diffusion with Gaussian Splatting for Urban Scene Reconstruction Zou, Yingshuang Ding, Yikang Zhang, Chuanrui Guo, Jiazhe Li, Bohan Lyu, Xiaoyang Tan, Feiyang Qi, Xiaojuan Wang, Haoqian Computer Vision and Pattern Recognition Recent breakthroughs in radiance fields have significantly advanced 3D scene reconstruction and novel view synthesis (NVS) in autonomous driving. Nevertheless, critical limitations persist: reconstruction-based methods exhibit substantial performance deterioration under significant viewpoint deviations from training trajectories, while generation-based techniques struggle with temporal coherence and precise scene controllability. To overcome these challenges, we present MuDG, an innovative framework that integrates Multi-modal Diffusion model with Gaussian Splatting (GS) for Urban Scene Reconstruction. MuDG leverages aggregated LiDAR point clouds with RGB and geometric priors to condition a multi-modal video diffusion model, synthesizing photorealistic RGB, depth, and semantic outputs for novel viewpoints. This synthesis pipeline enables feed-forward NVS without computationally intensive per-scene optimization, providing comprehensive supervision signals to refine 3DGS representations for rendering robustness enhancement under extreme viewpoint changes. Experiments on the Open Waymo Dataset demonstrate that MuDG outperforms existing methods in both reconstruction and synthesis quality. |
| title | MuDG: Taming Multi-modal Diffusion with Gaussian Splatting for Urban Scene Reconstruction |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2503.10604 |