RoboEXP: Action-Conditioned Scene Graph via Interactive Exploration for Robotic Manipulation
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866910639813820416 |
|---|---|
| author | Jiang, Hanxiao Huang, Binghao Wu, Ruihai Li, Zhuoran Garg, Shubham Nayyeri, Hooshang Wang, Shenlong Li, Yunzhu |
| author_facet | Jiang, Hanxiao Huang, Binghao Wu, Ruihai Li, Zhuoran Garg, Shubham Nayyeri, Hooshang Wang, Shenlong Li, Yunzhu |
| contents | We introduce the novel task of interactive scene exploration, wherein robots autonomously explore environments and produce an action-conditioned scene graph (ACSG) that captures the structure of the underlying environment. The ACSG accounts for both low-level information (geometry and semantics) and high-level information (action-conditioned relationships between different entities) in the scene. To this end, we present the Robotic Exploration (RoboEXP) system, which incorporates the Large Multimodal Model (LMM) and an explicit memory design to enhance our system's capabilities. The robot reasons about what and how to explore an object, accumulating new information through the interaction process and incrementally constructing the ACSG. Leveraging the constructed ACSG, we illustrate the effectiveness and efficiency of our RoboEXP system in facilitating a wide range of real-world manipulation tasks involving rigid, articulated objects, nested objects, and deformable objects. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_15487 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | RoboEXP: Action-Conditioned Scene Graph via Interactive Exploration for Robotic Manipulation Jiang, Hanxiao Huang, Binghao Wu, Ruihai Li, Zhuoran Garg, Shubham Nayyeri, Hooshang Wang, Shenlong Li, Yunzhu Robotics Artificial Intelligence Computer Vision and Pattern Recognition Machine Learning We introduce the novel task of interactive scene exploration, wherein robots autonomously explore environments and produce an action-conditioned scene graph (ACSG) that captures the structure of the underlying environment. The ACSG accounts for both low-level information (geometry and semantics) and high-level information (action-conditioned relationships between different entities) in the scene. To this end, we present the Robotic Exploration (RoboEXP) system, which incorporates the Large Multimodal Model (LMM) and an explicit memory design to enhance our system's capabilities. The robot reasons about what and how to explore an object, accumulating new information through the interaction process and incrementally constructing the ACSG. Leveraging the constructed ACSG, we illustrate the effectiveness and efficiency of our RoboEXP system in facilitating a wide range of real-world manipulation tasks involving rigid, articulated objects, nested objects, and deformable objects. |
| title | RoboEXP: Action-Conditioned Scene Graph via Interactive Exploration for Robotic Manipulation |
| topic | Robotics Artificial Intelligence Computer Vision and Pattern Recognition Machine Learning |
| url | https://arxiv.org/abs/2402.15487 |