HybridOcc: NeRF Enhanced Transformer-based Multi-Camera 3D Occupancy Prediction
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866917751873863680 |
|---|---|
| author | Zhao, Xiao Chen, Bo Sun, Mingyang Yang, Dingkang Wang, Youxing Zhang, Xukun Li, Mingcheng Kou, Dongliang Wei, Xiaoyi Zhang, Lihua |
| author_facet | Zhao, Xiao Chen, Bo Sun, Mingyang Yang, Dingkang Wang, Youxing Zhang, Xukun Li, Mingcheng Kou, Dongliang Wei, Xiaoyi Zhang, Lihua |
| contents | Vision-based 3D semantic scene completion (SSC) describes autonomous driving scenes through 3D volume representations. However, the occlusion of invisible voxels by scene surfaces poses challenges to current SSC methods in hallucinating refined 3D geometry. This paper proposes HybridOcc, a hybrid 3D volume query proposal method generated by Transformer framework and NeRF representation and refined in a coarse-to-fine SSC prediction framework. HybridOcc aggregates contextual features through the Transformer paradigm based on hybrid query proposals while combining it with NeRF representation to obtain depth supervision. The Transformer branch contains multiple scales and uses spatial cross-attention for 2D to 3D transformation. The newly designed NeRF branch implicitly infers scene occupancy through volume rendering, including visible and invisible voxels, and explicitly captures scene depth rather than generating RGB color. Furthermore, we present an innovative occupancy-aware ray sampling method to orient the SSC task instead of focusing on the scene surface, further improving the overall performance. Extensive experiments on nuScenes and SemanticKITTI datasets demonstrate the effectiveness of our HybridOcc on the SSC task. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_09104 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | HybridOcc: NeRF Enhanced Transformer-based Multi-Camera 3D Occupancy Prediction Zhao, Xiao Chen, Bo Sun, Mingyang Yang, Dingkang Wang, Youxing Zhang, Xukun Li, Mingcheng Kou, Dongliang Wei, Xiaoyi Zhang, Lihua Computer Vision and Pattern Recognition Vision-based 3D semantic scene completion (SSC) describes autonomous driving scenes through 3D volume representations. However, the occlusion of invisible voxels by scene surfaces poses challenges to current SSC methods in hallucinating refined 3D geometry. This paper proposes HybridOcc, a hybrid 3D volume query proposal method generated by Transformer framework and NeRF representation and refined in a coarse-to-fine SSC prediction framework. HybridOcc aggregates contextual features through the Transformer paradigm based on hybrid query proposals while combining it with NeRF representation to obtain depth supervision. The Transformer branch contains multiple scales and uses spatial cross-attention for 2D to 3D transformation. The newly designed NeRF branch implicitly infers scene occupancy through volume rendering, including visible and invisible voxels, and explicitly captures scene depth rather than generating RGB color. Furthermore, we present an innovative occupancy-aware ray sampling method to orient the SSC task instead of focusing on the scene surface, further improving the overall performance. Extensive experiments on nuScenes and SemanticKITTI datasets demonstrate the effectiveness of our HybridOcc on the SSC task. |
| title | HybridOcc: NeRF Enhanced Transformer-based Multi-Camera 3D Occupancy Prediction |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2408.09104 |