Lifting Unlabeled Internet-level Data for 3D Scene Understanding
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917432349687808 |
|---|---|
| author | Chen, Yixin Zhang, Yaowei Yu, Huangyue He, Junchao Wang, Yan Huang, Jiangyong Shen, Hongyu Ni, Junfeng Wang, Shaofei Jia, Baoxiong Zhu, Song-Chun Huang, Siyuan |
| author_facet | Chen, Yixin Zhang, Yaowei Yu, Huangyue He, Junchao Wang, Yan Huang, Jiangyong Shen, Hongyu Ni, Junfeng Wang, Shaofei Jia, Baoxiong Zhu, Song-Chun Huang, Siyuan |
| contents | Annotated 3D scene data is scarce and expensive to acquire, while abundant unlabeled videos are readily available on the internet. In this paper, we demonstrate that carefully designed data engines can leverage web-curated, unlabeled videos to automatically generate training data, to facilitate end-to-end models in 3D scene understanding alongside human-annotated datasets. We identify and analyze bottlenecks in automated data generation, revealing critical factors that determine the efficiency and effectiveness of learning from unlabeled data. To validate our approach across different perception granularities, we evaluate on three tasks spanning low-level perception, i.e., 3D object detection and instance segmentation, to high-evel reasoning, i.e., 3D spatial Visual Question Answering (VQA) and Vision-Lanugage Navigation (VLN). Models trained on our generated data demonstrate strong zero-shot performance and show further improvement after finetuning. This demonstrates the viability of leveraging readily available web data as a path toward more capable scene understanding systems. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_01907 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Lifting Unlabeled Internet-level Data for 3D Scene Understanding Chen, Yixin Zhang, Yaowei Yu, Huangyue He, Junchao Wang, Yan Huang, Jiangyong Shen, Hongyu Ni, Junfeng Wang, Shaofei Jia, Baoxiong Zhu, Song-Chun Huang, Siyuan Computer Vision and Pattern Recognition Artificial Intelligence Annotated 3D scene data is scarce and expensive to acquire, while abundant unlabeled videos are readily available on the internet. In this paper, we demonstrate that carefully designed data engines can leverage web-curated, unlabeled videos to automatically generate training data, to facilitate end-to-end models in 3D scene understanding alongside human-annotated datasets. We identify and analyze bottlenecks in automated data generation, revealing critical factors that determine the efficiency and effectiveness of learning from unlabeled data. To validate our approach across different perception granularities, we evaluate on three tasks spanning low-level perception, i.e., 3D object detection and instance segmentation, to high-evel reasoning, i.e., 3D spatial Visual Question Answering (VQA) and Vision-Lanugage Navigation (VLN). Models trained on our generated data demonstrate strong zero-shot performance and show further improvement after finetuning. This demonstrates the viability of leveraging readily available web data as a path toward more capable scene understanding systems. |
| title | Lifting Unlabeled Internet-level Data for 3D Scene Understanding |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2604.01907 |