Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2408.04034 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910864391536640 |
|---|---|
| author | Zhang, Zhuofan Zhu, Ziyu Li, Junhao Li, Pengxiang Wang, Tianxu Liu, Tengyu Ma, Xiaojian Chen, Yixin Jia, Baoxiong Huang, Siyuan Li, Qing |
| author_facet | Zhang, Zhuofan Zhu, Ziyu Li, Junhao Li, Pengxiang Wang, Tianxu Liu, Tengyu Ma, Xiaojian Chen, Yixin Jia, Baoxiong Huang, Siyuan Li, Qing |
| contents | Grounding natural language in 3D environments is a critical step toward achieving robust 3D vision-language alignment. Current datasets and models for 3D visual grounding predominantly focus on identifying and localizing objects from static, object-centric descriptions. These approaches do not adequately address the dynamic and sequential nature of task-oriented scenarios. In this work, we introduce a novel task: Task-oriented Sequential Grounding and Navigation in 3D Scenes, where models must interpret step-by-step instructions for daily activities by either localizing a sequence of target objects in indoor scenes or navigating toward them within a 3D simulator. To facilitate this task, we present SG3D, a large-scale dataset comprising 22,346 tasks with 112,236 steps across 4,895 real-world 3D scenes. The dataset is constructed by combining RGB-D scans from various 3D scene datasets with an automated task generation pipeline, followed by human verification for quality assurance. We benchmark contemporary methods on SG3D, revealing the significant challenges in understanding task-oriented context across multiple steps. Furthermore, we propose SG-LLM, a state-of-the-art approach leveraging a stepwise grounding paradigm to tackle the sequential grounding task. Our findings underscore the need for further research to advance the development of more capable and context-aware embodied agents. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_04034 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Task-oriented Sequential Grounding and Navigation in 3D Scenes Zhang, Zhuofan Zhu, Ziyu Li, Junhao Li, Pengxiang Wang, Tianxu Liu, Tengyu Ma, Xiaojian Chen, Yixin Jia, Baoxiong Huang, Siyuan Li, Qing Computer Vision and Pattern Recognition Grounding natural language in 3D environments is a critical step toward achieving robust 3D vision-language alignment. Current datasets and models for 3D visual grounding predominantly focus on identifying and localizing objects from static, object-centric descriptions. These approaches do not adequately address the dynamic and sequential nature of task-oriented scenarios. In this work, we introduce a novel task: Task-oriented Sequential Grounding and Navigation in 3D Scenes, where models must interpret step-by-step instructions for daily activities by either localizing a sequence of target objects in indoor scenes or navigating toward them within a 3D simulator. To facilitate this task, we present SG3D, a large-scale dataset comprising 22,346 tasks with 112,236 steps across 4,895 real-world 3D scenes. The dataset is constructed by combining RGB-D scans from various 3D scene datasets with an automated task generation pipeline, followed by human verification for quality assurance. We benchmark contemporary methods on SG3D, revealing the significant challenges in understanding task-oriented context across multiple steps. Furthermore, we propose SG-LLM, a state-of-the-art approach leveraging a stepwise grounding paradigm to tackle the sequential grounding task. Our findings underscore the need for further research to advance the development of more capable and context-aware embodied agents. |
| title | Task-oriented Sequential Grounding and Navigation in 3D Scenes |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2408.04034 |