DynaMem: Online Dynamic Spatio-Semantic Memory for Open World Mobile Manipulation
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866909626754138112 |
|---|---|
| author | Liu, Peiqi Guo, Zhanqiu Warke, Mohit Chintala, Soumith Paxton, Chris Shafiullah, Nur Muhammad Mahi Pinto, Lerrel |
| author_facet | Liu, Peiqi Guo, Zhanqiu Warke, Mohit Chintala, Soumith Paxton, Chris Shafiullah, Nur Muhammad Mahi Pinto, Lerrel |
| contents | Significant progress has been made in open-vocabulary mobile manipulation, where the goal is for a robot to perform tasks in any environment given a natural language description. However, most current systems assume a static environment, which limits the system's applicability in real-world scenarios where environments frequently change due to human intervention or the robot's own actions. In this work, we present DynaMem, a new approach to open-world mobile manipulation that uses a dynamic spatio-semantic memory to represent a robot's environment. DynaMem constructs a 3D data structure to maintain a dynamic memory of point clouds, and answers open-vocabulary object localization queries using multimodal LLMs or open-vocabulary features generated by state-of-the-art vision-language models. Powered by DynaMem, our robots can explore novel environments, search for objects not found in memory, and continuously update the memory as objects move, appear, or disappear in the scene. We run extensive experiments on the Stretch SE3 robots in three real and nine offline scenes, and achieve an average pick-and-drop success rate of 70% on non-stationary objects, which is more than a 2x improvement over state-of-the-art static systems. Our code as well as our experiment and deployment videos are open sourced and can be found on our project website: https://dynamem.github.io/ |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_04999 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | DynaMem: Online Dynamic Spatio-Semantic Memory for Open World Mobile Manipulation Liu, Peiqi Guo, Zhanqiu Warke, Mohit Chintala, Soumith Paxton, Chris Shafiullah, Nur Muhammad Mahi Pinto, Lerrel Robotics Machine Learning Significant progress has been made in open-vocabulary mobile manipulation, where the goal is for a robot to perform tasks in any environment given a natural language description. However, most current systems assume a static environment, which limits the system's applicability in real-world scenarios where environments frequently change due to human intervention or the robot's own actions. In this work, we present DynaMem, a new approach to open-world mobile manipulation that uses a dynamic spatio-semantic memory to represent a robot's environment. DynaMem constructs a 3D data structure to maintain a dynamic memory of point clouds, and answers open-vocabulary object localization queries using multimodal LLMs or open-vocabulary features generated by state-of-the-art vision-language models. Powered by DynaMem, our robots can explore novel environments, search for objects not found in memory, and continuously update the memory as objects move, appear, or disappear in the scene. We run extensive experiments on the Stretch SE3 robots in three real and nine offline scenes, and achieve an average pick-and-drop success rate of 70% on non-stationary objects, which is more than a 2x improvement over state-of-the-art static systems. Our code as well as our experiment and deployment videos are open sourced and can be found on our project website: https://dynamem.github.io/ |
| title | DynaMem: Online Dynamic Spatio-Semantic Memory for Open World Mobile Manipulation |
| topic | Robotics Machine Learning |
| url | https://arxiv.org/abs/2411.04999 |