RenderOcc: Vision-Centric 3D Occupancy Prediction with 2D Rendering Supervision
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911788270878720 |
|---|---|
| author | Pan, Mingjie Liu, Jiaming Zhang, Renrui Huang, Peixiang Li, Xiaoqi Wang, Bing Xie, Hongwei Liu, Li Zhang, Shanghang |
| author_facet | Pan, Mingjie Liu, Jiaming Zhang, Renrui Huang, Peixiang Li, Xiaoqi Wang, Bing Xie, Hongwei Liu, Li Zhang, Shanghang |
| contents | 3D occupancy prediction holds significant promise in the fields of robot perception and autonomous driving, which quantifies 3D scenes into grid cells with semantic labels. Recent works mainly utilize complete occupancy labels in 3D voxel space for supervision. However, the expensive annotation process and sometimes ambiguous labels have severely constrained the usability and scalability of 3D occupancy models. To address this, we present RenderOcc, a novel paradigm for training 3D occupancy models only using 2D labels. Specifically, we extract a NeRF-style 3D volume representation from multi-view images, and employ volume rendering techniques to establish 2D renderings, thus enabling direct 3D supervision from 2D semantics and depth labels. Additionally, we introduce an Auxiliary Ray method to tackle the issue of sparse viewpoints in autonomous driving scenarios, which leverages sequential frames to construct comprehensive 2D rendering for each object. To our best knowledge, RenderOcc is the first attempt to train multi-view 3D occupancy models only using 2D labels, reducing the dependence on costly 3D occupancy annotations. Extensive experiments demonstrate that RenderOcc achieves comparable performance to models fully supervised with 3D labels, underscoring the significance of this approach in real-world applications. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2309_09502 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | RenderOcc: Vision-Centric 3D Occupancy Prediction with 2D Rendering Supervision Pan, Mingjie Liu, Jiaming Zhang, Renrui Huang, Peixiang Li, Xiaoqi Wang, Bing Xie, Hongwei Liu, Li Zhang, Shanghang Computer Vision and Pattern Recognition 3D occupancy prediction holds significant promise in the fields of robot perception and autonomous driving, which quantifies 3D scenes into grid cells with semantic labels. Recent works mainly utilize complete occupancy labels in 3D voxel space for supervision. However, the expensive annotation process and sometimes ambiguous labels have severely constrained the usability and scalability of 3D occupancy models. To address this, we present RenderOcc, a novel paradigm for training 3D occupancy models only using 2D labels. Specifically, we extract a NeRF-style 3D volume representation from multi-view images, and employ volume rendering techniques to establish 2D renderings, thus enabling direct 3D supervision from 2D semantics and depth labels. Additionally, we introduce an Auxiliary Ray method to tackle the issue of sparse viewpoints in autonomous driving scenarios, which leverages sequential frames to construct comprehensive 2D rendering for each object. To our best knowledge, RenderOcc is the first attempt to train multi-view 3D occupancy models only using 2D labels, reducing the dependence on costly 3D occupancy annotations. Extensive experiments demonstrate that RenderOcc achieves comparable performance to models fully supervised with 3D labels, underscoring the significance of this approach in real-world applications. |
| title | RenderOcc: Vision-Centric 3D Occupancy Prediction with 2D Rendering Supervision |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2309.09502 |