OccScene: Semantic Occupancy-based Cross-task Mutual Learning for 3D Scene Generation
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866911115390222336 |
|---|---|
| author | Li, Bohan Jin, Xin Wang, Jianan Shi, Yukai Sun, Yasheng Wang, Xiaofeng Ma, Zhuang Xie, Baao Ma, Chao Yang, Xiaokang Zeng, Wenjun |
| author_facet | Li, Bohan Jin, Xin Wang, Jianan Shi, Yukai Sun, Yasheng Wang, Xiaofeng Ma, Zhuang Xie, Baao Ma, Chao Yang, Xiaokang Zeng, Wenjun |
| contents | Recent diffusion models have demonstrated remarkable performance in both 3D scene generation and perception tasks. Nevertheless, existing methods typically separate these two processes, acting as a data augmenter to generate synthetic data for downstream perception tasks. In this work, we propose OccScene, a novel mutual learning paradigm that integrates fine-grained 3D perception and high-quality generation in a unified framework, achieving a cross-task win-win effect. OccScene generates new and consistent 3D realistic scenes only depending on text prompts, guided with semantic occupancy in a joint-training diffusion framework. To align the occupancy with the diffusion latent, a Mamba-based Dual Alignment module is introduced to incorporate fine-grained semantics and geometry as perception priors. Within OccScene, the perception module can be effectively improved with customized and diverse generated scenes, while the perception priors in return enhance the generation performance for mutual benefits. Extensive experiments show that OccScene achieves realistic 3D scene generation in broad indoor and outdoor scenarios, while concurrently boosting the perception models to achieve substantial performance improvements in the 3D perception task of semantic occupancy prediction. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_11183 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | OccScene: Semantic Occupancy-based Cross-task Mutual Learning for 3D Scene Generation Li, Bohan Jin, Xin Wang, Jianan Shi, Yukai Sun, Yasheng Wang, Xiaofeng Ma, Zhuang Xie, Baao Ma, Chao Yang, Xiaokang Zeng, Wenjun Computer Vision and Pattern Recognition Recent diffusion models have demonstrated remarkable performance in both 3D scene generation and perception tasks. Nevertheless, existing methods typically separate these two processes, acting as a data augmenter to generate synthetic data for downstream perception tasks. In this work, we propose OccScene, a novel mutual learning paradigm that integrates fine-grained 3D perception and high-quality generation in a unified framework, achieving a cross-task win-win effect. OccScene generates new and consistent 3D realistic scenes only depending on text prompts, guided with semantic occupancy in a joint-training diffusion framework. To align the occupancy with the diffusion latent, a Mamba-based Dual Alignment module is introduced to incorporate fine-grained semantics and geometry as perception priors. Within OccScene, the perception module can be effectively improved with customized and diverse generated scenes, while the perception priors in return enhance the generation performance for mutual benefits. Extensive experiments show that OccScene achieves realistic 3D scene generation in broad indoor and outdoor scenarios, while concurrently boosting the perception models to achieve substantial performance improvements in the 3D perception task of semantic occupancy prediction. |
| title | OccScene: Semantic Occupancy-based Cross-task Mutual Learning for 3D Scene Generation |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2412.11183 |