Grounding 3D Object Affordance with Language Instructions, Visual Observations and Interactions
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910905256640512 |
|---|---|
| author | Zhu, He Kong, Quyu Xu, Kechun Xia, Xunlong Deng, Bing Ye, Jieping Xiong, Rong Wang, Yue |
| author_facet | Zhu, He Kong, Quyu Xu, Kechun Xia, Xunlong Deng, Bing Ye, Jieping Xiong, Rong Wang, Yue |
| contents | Grounding 3D object affordance is a task that locates objects in 3D space where they can be manipulated, which links perception and action for embodied intelligence. For example, for an intelligent robot, it is necessary to accurately ground the affordance of an object and grasp it according to human instructions. In this paper, we introduce a novel task that grounds 3D object affordance based on language instructions, visual observations and interactions, which is inspired by cognitive science. We collect an Affordance Grounding dataset with Points, Images and Language instructions (AGPIL) to support the proposed task. In the 3D physical world, due to observation orientation, object rotation, or spatial occlusion, we can only get a partial observation of the object. So this dataset includes affordance estimations of objects from full-view, partial-view, and rotation-view perspectives. To accomplish this task, we propose LMAffordance3D, the first multi-modal, language-guided 3D affordance grounding network, which applies a vision-language model to fuse 2D and 3D spatial features with semantic features. Comprehensive experiments on AGPIL demonstrate the effectiveness and superiority of our method on this task, even in unseen experimental settings. Our project is available at https://sites.google.com/view/lmaffordance3d. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_04744 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Grounding 3D Object Affordance with Language Instructions, Visual Observations and Interactions Zhu, He Kong, Quyu Xu, Kechun Xia, Xunlong Deng, Bing Ye, Jieping Xiong, Rong Wang, Yue Computer Vision and Pattern Recognition Artificial Intelligence Robotics Grounding 3D object affordance is a task that locates objects in 3D space where they can be manipulated, which links perception and action for embodied intelligence. For example, for an intelligent robot, it is necessary to accurately ground the affordance of an object and grasp it according to human instructions. In this paper, we introduce a novel task that grounds 3D object affordance based on language instructions, visual observations and interactions, which is inspired by cognitive science. We collect an Affordance Grounding dataset with Points, Images and Language instructions (AGPIL) to support the proposed task. In the 3D physical world, due to observation orientation, object rotation, or spatial occlusion, we can only get a partial observation of the object. So this dataset includes affordance estimations of objects from full-view, partial-view, and rotation-view perspectives. To accomplish this task, we propose LMAffordance3D, the first multi-modal, language-guided 3D affordance grounding network, which applies a vision-language model to fuse 2D and 3D spatial features with semantic features. Comprehensive experiments on AGPIL demonstrate the effectiveness and superiority of our method on this task, even in unseen experimental settings. Our project is available at https://sites.google.com/view/lmaffordance3d. |
| title | Grounding 3D Object Affordance with Language Instructions, Visual Observations and Interactions |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Robotics |
| url | https://arxiv.org/abs/2504.04744 |