PAVLM: Advancing Point Cloud based Affordance Understanding Via Vision-Language Model
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866909676289916928 |
|---|---|
| author | Liu, Shang-Ching Tran, Van Nhiem Chen, Wenkai Cheng, Wei-Lun Huang, Yen-Lin Liao, I-Bin Li, Yung-Hui Zhang, Jianwei |
| author_facet | Liu, Shang-Ching Tran, Van Nhiem Chen, Wenkai Cheng, Wei-Lun Huang, Yen-Lin Liao, I-Bin Li, Yung-Hui Zhang, Jianwei |
| contents | Affordance understanding, the task of identifying actionable regions on 3D objects, plays a vital role in allowing robotic systems to engage with and operate within the physical world. Although Visual Language Models (VLMs) have excelled in high-level reasoning and long-horizon planning for robotic manipulation, they still fall short in grasping the nuanced physical properties required for effective human-robot interaction. In this paper, we introduce PAVLM (Point cloud Affordance Vision-Language Model), an innovative framework that utilizes the extensive multimodal knowledge embedded in pre-trained language models to enhance 3D affordance understanding of point cloud. PAVLM integrates a geometric-guided propagation module with hidden embeddings from large language models (LLMs) to enrich visual semantics. On the language side, we prompt Llama-3.1 models to generate refined context-aware text, augmenting the instructional input with deeper semantic cues. Experimental results on the 3D-AffordanceNet benchmark demonstrate that PAVLM outperforms baseline methods for both full and partial point clouds, particularly excelling in its generalization to novel open-world affordance tasks of 3D objects. For more information, visit our project site: pavlm-source.github.io. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_11564 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | PAVLM: Advancing Point Cloud based Affordance Understanding Via Vision-Language Model Liu, Shang-Ching Tran, Van Nhiem Chen, Wenkai Cheng, Wei-Lun Huang, Yen-Lin Liao, I-Bin Li, Yung-Hui Zhang, Jianwei Robotics Computer Vision and Pattern Recognition Affordance understanding, the task of identifying actionable regions on 3D objects, plays a vital role in allowing robotic systems to engage with and operate within the physical world. Although Visual Language Models (VLMs) have excelled in high-level reasoning and long-horizon planning for robotic manipulation, they still fall short in grasping the nuanced physical properties required for effective human-robot interaction. In this paper, we introduce PAVLM (Point cloud Affordance Vision-Language Model), an innovative framework that utilizes the extensive multimodal knowledge embedded in pre-trained language models to enhance 3D affordance understanding of point cloud. PAVLM integrates a geometric-guided propagation module with hidden embeddings from large language models (LLMs) to enrich visual semantics. On the language side, we prompt Llama-3.1 models to generate refined context-aware text, augmenting the instructional input with deeper semantic cues. Experimental results on the 3D-AffordanceNet benchmark demonstrate that PAVLM outperforms baseline methods for both full and partial point clouds, particularly excelling in its generalization to novel open-world affordance tasks of 3D objects. For more information, visit our project site: pavlm-source.github.io. |
| title | PAVLM: Advancing Point Cloud based Affordance Understanding Via Vision-Language Model |
| topic | Robotics Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2410.11564 |