Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2411.10446 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911599375155200 |
|---|---|
| author | Ekpo, Daniel Levy, Mara Suri, Saksham Huynh, Chuong Swaminathan, Archana Shrivastava, Abhinav |
| author_facet | Ekpo, Daniel Levy, Mara Suri, Saksham Huynh, Chuong Swaminathan, Archana Shrivastava, Abhinav |
| contents | Recent progress in vision-language models (VLMs) has opened new possibilities for robot task planning, but these models often produce incorrect action sequences. To address these limitations, we propose VeriGraph, a novel framework that integrates VLMs for robotic planning while verifying action feasibility. VeriGraph uses scene graphs as an intermediate representation to capture key objects and spatial relationships, enabling more reliable plan verification and refinement. The system generates a scene graph from input images and uses it to iteratively check and correct action sequences generated by an LLM-based task planner, ensuring constraints are respected and actions are executable. Our approach significantly enhances task completion rates across diverse manipulation scenarios, outperforming baseline methods by 58% on language-based tasks, 56% on tangram puzzle tasks, and 30% on image-based tasks. Qualitative results and code can be found at https://verigraph-agent.github.io. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_10446 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | VeriGraph: Scene Graphs for Execution Verifiable Robot Planning Ekpo, Daniel Levy, Mara Suri, Saksham Huynh, Chuong Swaminathan, Archana Shrivastava, Abhinav Robotics Artificial Intelligence Recent progress in vision-language models (VLMs) has opened new possibilities for robot task planning, but these models often produce incorrect action sequences. To address these limitations, we propose VeriGraph, a novel framework that integrates VLMs for robotic planning while verifying action feasibility. VeriGraph uses scene graphs as an intermediate representation to capture key objects and spatial relationships, enabling more reliable plan verification and refinement. The system generates a scene graph from input images and uses it to iteratively check and correct action sequences generated by an LLM-based task planner, ensuring constraints are respected and actions are executable. Our approach significantly enhances task completion rates across diverse manipulation scenarios, outperforming baseline methods by 58% on language-based tasks, 56% on tangram puzzle tasks, and 30% on image-based tasks. Qualitative results and code can be found at https://verigraph-agent.github.io. |
| title | VeriGraph: Scene Graphs for Execution Verifiable Robot Planning |
| topic | Robotics Artificial Intelligence |
| url | https://arxiv.org/abs/2411.10446 |