Embodied-RAG: General Non-parametric Embodied Memory for Retrieval and Generation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912195821961216 |
|---|---|
| author | Xie, Quanting Min, So Yeon Ji, Pengliang Yang, Yue Zhang, Tianyi Xu, Kedi Bajaj, Aarav Salakhutdinov, Ruslan Johnson-Roberson, Matthew Bisk, Yonatan |
| author_facet | Xie, Quanting Min, So Yeon Ji, Pengliang Yang, Yue Zhang, Tianyi Xu, Kedi Bajaj, Aarav Salakhutdinov, Ruslan Johnson-Roberson, Matthew Bisk, Yonatan |
| contents | There is no limit to how much a robot might explore and learn, but all of that knowledge needs to be searchable and actionable. Within language research, retrieval augmented generation (RAG) has become the workhorse of large-scale non-parametric knowledge; however, existing techniques do not directly transfer to the embodied domain, which is multimodal, where data is highly correlated, and perception requires abstraction. To address these challenges, we introduce Embodied-RAG, a framework that enhances the foundational model of an embodied agent with a non-parametric memory system capable of autonomously constructing hierarchical knowledge for both navigation and language generation. Embodied-RAG handles a full range of spatial and semantic resolutions across diverse environments and query types, whether for a specific object or a holistic description of ambiance. At its core, Embodied-RAG's memory is structured as a semantic forest, storing language descriptions at varying levels of detail. This hierarchical organization allows the system to efficiently generate context-sensitive outputs across different robotic platforms. We demonstrate that Embodied-RAG effectively bridges RAG to the robotics domain, successfully handling over 250 explanation and navigation queries across kilometer-level environments, highlighting its promise as a general-purpose non-parametric system for embodied agents. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_18313 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Embodied-RAG: General Non-parametric Embodied Memory for Retrieval and Generation Xie, Quanting Min, So Yeon Ji, Pengliang Yang, Yue Zhang, Tianyi Xu, Kedi Bajaj, Aarav Salakhutdinov, Ruslan Johnson-Roberson, Matthew Bisk, Yonatan Robotics Artificial Intelligence Machine Learning There is no limit to how much a robot might explore and learn, but all of that knowledge needs to be searchable and actionable. Within language research, retrieval augmented generation (RAG) has become the workhorse of large-scale non-parametric knowledge; however, existing techniques do not directly transfer to the embodied domain, which is multimodal, where data is highly correlated, and perception requires abstraction. To address these challenges, we introduce Embodied-RAG, a framework that enhances the foundational model of an embodied agent with a non-parametric memory system capable of autonomously constructing hierarchical knowledge for both navigation and language generation. Embodied-RAG handles a full range of spatial and semantic resolutions across diverse environments and query types, whether for a specific object or a holistic description of ambiance. At its core, Embodied-RAG's memory is structured as a semantic forest, storing language descriptions at varying levels of detail. This hierarchical organization allows the system to efficiently generate context-sensitive outputs across different robotic platforms. We demonstrate that Embodied-RAG effectively bridges RAG to the robotics domain, successfully handling over 250 explanation and navigation queries across kilometer-level environments, highlighting its promise as a general-purpose non-parametric system for embodied agents. |
| title | Embodied-RAG: General Non-parametric Embodied Memory for Retrieval and Generation |
| topic | Robotics Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2409.18313 |