InstructPart: Task-Oriented Part Segmentation with Instruction Reasoning
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910969542737920 |
|---|---|
| author | Wan, Zifu Xie, Yaqi Zhang, Ce Lin, Zhiqiu Wang, Zihan Stepputtis, Simon Ramanan, Deva Sycara, Katia |
| author_facet | Wan, Zifu Xie, Yaqi Zhang, Ce Lin, Zhiqiu Wang, Zihan Stepputtis, Simon Ramanan, Deva Sycara, Katia |
| contents | Large multimodal foundation models, particularly in the domains of language and vision, have significantly advanced various tasks, including robotics, autonomous driving, information retrieval, and grounding. However, many of these models perceive objects as indivisible, overlooking the components that constitute them. Understanding these components and their associated affordances provides valuable insights into an object's functionality, which is fundamental for performing a wide range of tasks. In this work, we introduce a novel real-world benchmark, InstructPart, comprising hand-labeled part segmentation annotations and task-oriented instructions to evaluate the performance of current models in understanding and executing part-level tasks within everyday contexts. Through our experiments, we demonstrate that task-oriented part segmentation remains a challenging problem, even for state-of-the-art Vision-Language Models (VLMs). In addition to our benchmark, we introduce a simple baseline that achieves a twofold performance improvement through fine-tuning with our dataset. With our dataset and benchmark, we aim to facilitate research on task-oriented part segmentation and enhance the applicability of VLMs across various domains, including robotics, virtual reality, information retrieval, and other related fields. Project website: https://zifuwan.github.io/InstructPart/. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_18291 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | InstructPart: Task-Oriented Part Segmentation with Instruction Reasoning Wan, Zifu Xie, Yaqi Zhang, Ce Lin, Zhiqiu Wang, Zihan Stepputtis, Simon Ramanan, Deva Sycara, Katia Computer Vision and Pattern Recognition Computation and Language Robotics Large multimodal foundation models, particularly in the domains of language and vision, have significantly advanced various tasks, including robotics, autonomous driving, information retrieval, and grounding. However, many of these models perceive objects as indivisible, overlooking the components that constitute them. Understanding these components and their associated affordances provides valuable insights into an object's functionality, which is fundamental for performing a wide range of tasks. In this work, we introduce a novel real-world benchmark, InstructPart, comprising hand-labeled part segmentation annotations and task-oriented instructions to evaluate the performance of current models in understanding and executing part-level tasks within everyday contexts. Through our experiments, we demonstrate that task-oriented part segmentation remains a challenging problem, even for state-of-the-art Vision-Language Models (VLMs). In addition to our benchmark, we introduce a simple baseline that achieves a twofold performance improvement through fine-tuning with our dataset. With our dataset and benchmark, we aim to facilitate research on task-oriented part segmentation and enhance the applicability of VLMs across various domains, including robotics, virtual reality, information retrieval, and other related fields. Project website: https://zifuwan.github.io/InstructPart/. |
| title | InstructPart: Task-Oriented Part Segmentation with Instruction Reasoning |
| topic | Computer Vision and Pattern Recognition Computation and Language Robotics |
| url | https://arxiv.org/abs/2505.18291 |