PonderV2: Pave the Way for 3D Foundation Model with A Universal Pre-training Paradigm
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912327112065024 |
|---|---|
| author | Zhu, Haoyi Yang, Honghui Wu, Xiaoyang Huang, Di Zhang, Sha He, Xianglong Zhao, Hengshuang Shen, Chunhua Qiao, Yu He, Tong Ouyang, Wanli |
| author_facet | Zhu, Haoyi Yang, Honghui Wu, Xiaoyang Huang, Di Zhang, Sha He, Xianglong Zhao, Hengshuang Shen, Chunhua Qiao, Yu He, Tong Ouyang, Wanli |
| contents | In contrast to numerous NLP and 2D vision foundational models, learning a 3D foundational model poses considerably greater challenges. This is primarily due to the inherent data variability and diversity of downstream tasks. In this paper, we introduce a novel universal 3D pre-training framework designed to facilitate the acquisition of efficient 3D representation, thereby establishing a pathway to 3D foundational models. Considering that informative 3D features should encode rich geometry and appearance cues that can be utilized to render realistic images, we propose to learn 3D representations by differentiable neural rendering. We train a 3D backbone with a devised volumetric neural renderer by comparing the rendered with the real images. Notably, our approach seamlessly integrates the learned 3D encoder into various downstream tasks. These tasks encompass not only high-level challenges such as 3D detection and segmentation but also low-level objectives like 3D reconstruction and image synthesis, spanning both indoor and outdoor scenarios. Besides, we also illustrate the capability of pre-training a 2D backbone using the proposed methodology, surpassing conventional pre-training methods by a large margin. For the first time, PonderV2 achieves state-of-the-art performance on 11 indoor and outdoor benchmarks, implying its effectiveness. Code and models are available at https://github.com/OpenGVLab/PonderV2. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2310_08586 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | PonderV2: Pave the Way for 3D Foundation Model with A Universal Pre-training Paradigm Zhu, Haoyi Yang, Honghui Wu, Xiaoyang Huang, Di Zhang, Sha He, Xianglong Zhao, Hengshuang Shen, Chunhua Qiao, Yu He, Tong Ouyang, Wanli Computer Vision and Pattern Recognition In contrast to numerous NLP and 2D vision foundational models, learning a 3D foundational model poses considerably greater challenges. This is primarily due to the inherent data variability and diversity of downstream tasks. In this paper, we introduce a novel universal 3D pre-training framework designed to facilitate the acquisition of efficient 3D representation, thereby establishing a pathway to 3D foundational models. Considering that informative 3D features should encode rich geometry and appearance cues that can be utilized to render realistic images, we propose to learn 3D representations by differentiable neural rendering. We train a 3D backbone with a devised volumetric neural renderer by comparing the rendered with the real images. Notably, our approach seamlessly integrates the learned 3D encoder into various downstream tasks. These tasks encompass not only high-level challenges such as 3D detection and segmentation but also low-level objectives like 3D reconstruction and image synthesis, spanning both indoor and outdoor scenarios. Besides, we also illustrate the capability of pre-training a 2D backbone using the proposed methodology, surpassing conventional pre-training methods by a large margin. For the first time, PonderV2 achieves state-of-the-art performance on 11 indoor and outdoor benchmarks, implying its effectiveness. Code and models are available at https://github.com/OpenGVLab/PonderV2. |
| title | PonderV2: Pave the Way for 3D Foundation Model with A Universal Pre-training Paradigm |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2310.08586 |