ReconDreamer: Crafting World Models for Driving Scene Reconstruction via Online Restoration
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917851852439552 |
|---|---|
| author | Ni, Chaojun Zhao, Guosheng Wang, Xiaofeng Zhu, Zheng Qin, Wenkang Huang, Guan Liu, Chen Chen, Yuyin Wang, Yida Zhang, Xueyang Zhan, Yifei Zhan, Kun Jia, Peng Lang, Xianpeng Wang, Xingang Mei, Wenjun |
| author_facet | Ni, Chaojun Zhao, Guosheng Wang, Xiaofeng Zhu, Zheng Qin, Wenkang Huang, Guan Liu, Chen Chen, Yuyin Wang, Yida Zhang, Xueyang Zhan, Yifei Zhan, Kun Jia, Peng Lang, Xianpeng Wang, Xingang Mei, Wenjun |
| contents | Closed-loop simulation is crucial for end-to-end autonomous driving. Existing sensor simulation methods (e.g., NeRF and 3DGS) reconstruct driving scenes based on conditions that closely mirror training data distributions. However, these methods struggle with rendering novel trajectories, such as lane changes. Recent works have demonstrated that integrating world model knowledge alleviates these issues. Despite their efficiency, these approaches still encounter difficulties in the accurate representation of more complex maneuvers, with multi-lane shifts being a notable example. Therefore, we introduce ReconDreamer, which enhances driving scene reconstruction through incremental integration of world model knowledge. Specifically, DriveRestorer is proposed to mitigate artifacts via online restoration. This is complemented by a progressive data update strategy designed to ensure high-quality rendering for more complex maneuvers. To the best of our knowledge, ReconDreamer is the first method to effectively render in large maneuvers. Experimental results demonstrate that ReconDreamer outperforms Street Gaussians in the NTA-IoU, NTL-IoU, and FID, with relative improvements by 24.87%, 6.72%, and 29.97%. Furthermore, ReconDreamer surpasses DriveDreamer4D with PVG during large maneuver rendering, as verified by a relative improvement of 195.87% in the NTA-IoU metric and a comprehensive user study. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_19548 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | ReconDreamer: Crafting World Models for Driving Scene Reconstruction via Online Restoration Ni, Chaojun Zhao, Guosheng Wang, Xiaofeng Zhu, Zheng Qin, Wenkang Huang, Guan Liu, Chen Chen, Yuyin Wang, Yida Zhang, Xueyang Zhan, Yifei Zhan, Kun Jia, Peng Lang, Xianpeng Wang, Xingang Mei, Wenjun Computer Vision and Pattern Recognition Artificial Intelligence Machine Learning Robotics Closed-loop simulation is crucial for end-to-end autonomous driving. Existing sensor simulation methods (e.g., NeRF and 3DGS) reconstruct driving scenes based on conditions that closely mirror training data distributions. However, these methods struggle with rendering novel trajectories, such as lane changes. Recent works have demonstrated that integrating world model knowledge alleviates these issues. Despite their efficiency, these approaches still encounter difficulties in the accurate representation of more complex maneuvers, with multi-lane shifts being a notable example. Therefore, we introduce ReconDreamer, which enhances driving scene reconstruction through incremental integration of world model knowledge. Specifically, DriveRestorer is proposed to mitigate artifacts via online restoration. This is complemented by a progressive data update strategy designed to ensure high-quality rendering for more complex maneuvers. To the best of our knowledge, ReconDreamer is the first method to effectively render in large maneuvers. Experimental results demonstrate that ReconDreamer outperforms Street Gaussians in the NTA-IoU, NTL-IoU, and FID, with relative improvements by 24.87%, 6.72%, and 29.97%. Furthermore, ReconDreamer surpasses DriveDreamer4D with PVG during large maneuver rendering, as verified by a relative improvement of 195.87% in the NTA-IoU metric and a comprehensive user study. |
| title | ReconDreamer: Crafting World Models for Driving Scene Reconstruction via Online Restoration |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Machine Learning Robotics |
| url | https://arxiv.org/abs/2411.19548 |